Global Mapper v25.0

Compression ratio, file size, processing limits?

John Speargas
John Speargas Global Mapper User
edited November 2012 in Technical Support
I use Global Mapper frequently to load a large number of 3- and 4-band tiffs, sometimes reproject, and tile to tiff or compress to ECW. When compressing, there doesn't seem to be any reliable gauge to the file size I can expect, so sometimes several hours are wasted because the output is too large. Recently, for example, I compressed a set of tiffs to ecw at 2:1, and after running for some time, the resulting ecw was about 19 GB. The target was to be around 6, so I figured that compressing at 6:1 should then yield a file around 6.3 GB. The resultant file was in fact about 12 GB. I think 10:1 actually got pretty close to the target, though I don't recall the exact number and I don't have it in front of me. Is there any kind of insight you can offer on how we might better target a compression ratio based on the desired file size?

Sometimes the tiffs take quite a long time to load, and sometimes it stops responding altogether. I've taken to using map catalogs frequently so the rendering time is less of an issue, but I wonder about the processing demand of compression. Sometimes it seems like a compression is running along fine for hours or days only to have the estimated finish time start adding time more quickly than time actually passes.

This might not be our typical dataset, but as an example one area I've been working with requires me to load 6524 4-band tiffs (RGB and IR), 2000' by 2000' with a 3" pixel, each about 0.25 GB. When I load all of these I use a map catalog, and I set the raster display to ignore alpha (though sometimes on export I am prompted by several dialogue boxes per tile asking me about the fourth band of data). But is this pushing the limits of the software, loading 1.6 TB of imagery (and sometimes more on top of that)? Is there a max file size or max number of files or anything that I need to be aware of?

Thanks,
John

Comments

  • global_mapper
    global_mapper Administrator
    edited October 2012
    John,

    The target compression ratio is relative to the uncompressed size of the data involved. Your input TIFF files are almost certainly compressed already themselves, although likely with a lossless compressiong like LZW, Packbits, etc. so the compression isn't nearly as good as ECW. You could get a better guess by calculating the total number of pixels and multiplying by 3 to get the uncompressed size in bytes, then apply the target compression ratio to that. However even that could be wildly inaccurate as different data will compress differently. For example an image that is all one color might compress down to almost nothing, while one where the colors are completely random would not compress well at all.

    Map catalogs are definitely the way to go when working with large numbers of TIFF files like you are. The map catalog will automatically handle loading and unloading the individual TIFF files so in general you won't have very many loaded at a time. This will greatly speed up the access to the TIFF files. You can also get better performance if you are on the 64-bit version of Global Mapper as it can use much more memory. There really aren't any limits when using map catalogs, at least not any that you are even remotely approaching.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • wmiswm
    wmiswm Global Mapper User Trusted User
    edited October 2012
    I do not have much experience with the use of map catalog. However, there was one time that I tried to "search" while my files were opened by map catalog, it took endless hours for just to display the initial search sub-manuel. Is this normal?
  • global_mapper
    global_mapper Administrator
    edited October 2012
    Yes vector searching would not be recommended for map catalogs. Map Catalogs should be used when you have a lot of files to work with but only need a small subset of them at a time. The map catalog can then manage loading and unloaded them as needed. For example I keep map catalogs of data sets that cover the entire US at high resolution so I can easily make all of that available without having to load it all.

    A vector search is going to need everything to be loaded, so when you start the search every single file in the catalog is going to have to be loaded to get at the data. If you are going to do that you might as well just load all of the data directly.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • John Speargas
    John Speargas Global Mapper User
    edited November 2012
    I actually am getting worse performance since I upgraded to 14. For example, two days ago I ran an export from tiff (about 2.2 TB total dataset uncompressed) to ECW from GM 14 at 20:1. Yesterday morning, it said it was at 100% but not responding. I let it run all day, and as of 11 PM there was no change. I had to open the Task Manager and kill the program. Then, last night I started an export of a tiff dataset, about 15 GB uncompressed, to ECW at 1:1, gridded 2 by 2, and let it run overnight. This morning it showed 25% in the back window, meaning it was working on the first tile, but in the front window showed 0% and not responding. I killed it again, restarted the machine, and restarted the same export but using 3 by 3 gridding. After almost two hours, it was showing 11% in the back window, working on the first tile, but 0% and not responding in the front window. I killed it again, and restarted exactly the same process, same data, same machine, using GM 12 and it's humming along smoothly. What gives?
  • global_mapper
    global_mapper Administrator
    edited November 2012
    How is your 2.2 TB total data set loaded before you export? Specifically are you loading from a map catalog or directly loading the files?

    There shouldn't be any normal slow downs in v14 over v12, but maybe there is something in this particular case. It does sound like it is stuck at the end or something. Are you using the 32-bit or 64-bit version? Can you get the v14.0.3 release from the links below if you don't have it already?

    32-bit: http://www.globalmapper.com/downloads/global_mapper14_setup.exe
    64-bit: http://www.globalmapper.com/downloads/global_mapper14_setup_64bit.exe

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • John Speargas
    John Speargas Global Mapper User
    edited November 2012
    I just have 14.0, but I'll download 14.3. I am using the 64-bit version. When I load the data, it is in three map catalogs, each of which points to tiffs within a different folder.
  • John Speargas
    John Speargas Global Mapper User
    edited November 2012
    I clicked the link you provided for the 64 bit, ran it, and it's still showing 14.00.0010. Am I missing something?
  • global_mapper
    global_mapper Administrator
    edited November 2012
    That's not right, it should read 14.03.0010, or on the Help->About dialog in Global Mapper 14.0.3. Where are you seeing the 14.00.0010 at?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • John Speargas
    John Speargas Global Mapper User
    edited November 2012
    Control Panel > Programs > Programs and Features
  • global_mapper
    global_mapper Administrator
    edited November 2012
    Check the Help->About box in Global Mapper, not the Windows Control Panel as it probably has a version of the installer or something. The .exe itself is the version I would be interested in.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • John Speargas
    John Speargas Global Mapper User
    edited November 2012
    Okay, yes it is 14.0.3. It did the same thing again just now, where the front dialogue box indicates no problem and the rear one only reflects that it's on the first tile. I didn't let it run very long this time, but it appeared to be making no progress at all, whereas I just restarted it with 12 and it's running along fine. I hope we can get to the bottom of this.
  • global_mapper
    global_mapper Administrator
    edited November 2012
    If you export to a format other than ECW does it work ok? Do your TIFF files live on a local drive or a network drive? How many TIFF files are in your map catalog (can you provide the .gmc file)? If the problem happens exporting to another format then the issue is likely some performance issue reading from the TIFF files. Are you resampling at all? Is there anything loaded besides the map catalog?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com