Best raster file format for fast display

danielek

Hello,

I am using raster data of many different sources and frequently face the question whether the native or default format should better be converted for faster display in Global Mapper. Typically the formats are tiff, png, ecw, NASA img, jp2, png and jpg.

I guess the main question behind is whether compressed formats should be favoured because they save space in the hard drive, or uncompressed formats because Global Mapper will use more time to decompress than to open a large file. I guess the answer should also be modulated by the type of compression, such as jpeg or ecw/jp2 wavelet, and for tiff files, lzw or zip or none. The issue is especially critical for large files. For the same raster data, who should be favoured among a 500 Mb uncompressed tif, 50 Mb ecw, 100 Mb jpg...?

I have made several attempts to find the answer but it is a little tricky as the loading time may also depend on what Global Mapper already has in memory. It seems, however, that the loading time does not critically depend a lot on file format... in this case the smaller raster file size the better, is this correct?

samknight

Hi Danielek, at small sizes, you'll notice less practical difference in performance, but in general the wavelet compressed formats (JP2, ECW, MrSID) outperform all others and each of those will have similar performance at small or large sizes.  Another factor is the practical size of data they can handle.  A Tiff file with common packbit compression of about a gigabyte, will pack down to more like 2-10 megabytes depending on the chosen compression for the wavelet format, and the format itself can only handle up to about 2gb of data.  To your question about unpacking the data, the wavelet formats access the data differently than the more raw formats, so there isn't much of a hit there.  The analogy I use is a suitcase.  In formats like Tiff of JPG, it's like having a duffel bag full of clothes that were all just stuffed in there.  The wavelet formats are like a suitcase with many compartments and all the clothes are neatly folded and organized.  The stuff inside is the same, it's the efficiency of the storage itself that makes a difference.  In the duffel bag, you've got to dig through everything in the bag, even if you're only after your socks.  But in the neatly folded bag you know exactly which compartment to find your socks in, so you can get them out faster.

Another factor, if you are working purely in Global Mapper is that you can set up a Map Catalog that will allow you to load up individual tiles into a collection of tiles.  This allows you to set up zoom levels for tilesets of various resolution and coverage, and allow the application to only load particular images when they are needed, rather than loading up images covering larger areas all the time.  To extend my previous analogy, this would be like having a whole family of efficiently packed suitcases with names labeled on the outside so now you can find a particular person's socks.  Make sense?  I hope that helps.  

The short version of my answer is, I recommend JP2 as a go-to image format, and map catalogs of tilesets if you are working with coverages of big areas.

Regards,
Sam Knight
Director of Product Management
Blue Marble Geographics

danielek

many thanks for this detailed and quick answer... then jp2 for everyone... do you prefer jp2 for licensing issues or because it is better than ecw really?

samknight

I prefer JP2 over the others mainly because of compatibility, almost everyone supports read of JP2 natively in display engines now due to it being an open ISO standard, and the internal referencing in JP2 is more flexible than that found in ECW, this is a benefit when using custom coordinate systems which is something we happen to do a lot of here at Blue Marble.  Many state and federally supplied collections are in JP2 already so it's becoming more familiar to everyone.  That said, MrSID and ECW are much better supported for read now than years ago.  It's not like 10 years ago when you would have to find and install plugins into software to enable a particular format in a display package.  In terms of performance and superiority, there are many factors involved that make it very difficult to give a one size fits all answer; certain qualities of the image like transparency, the specific software it will be accessed through, etc.  There are scenarios where each of those three will outperform the others but generally they are all in the same ballpark.

Regards,
Sam Knight
Director of Product Management
Blue Marble Geographics

danielek

Sam,

JP2 seems to cause problems in Global Mapper 16 64 bit (windows) which I am using, and since I tend to use Er Mapper for image processing, I rather switched to ecw.

Certainly this format speeds up display in many circumstances, as you wrote. Great. However, I am gradually having what seems to be memory problems. I am gradually increasing the size of my raster database, which now is around 4 Gb. The largest files are ecw and geotiff (topography) mosaic tiles. I put all the ecw tiles in a catalog and the tiff (16 bit topography) in another. The symptoms are:

- Global Mapper has become slower and slower to start. Now it takes some 5 minutes with around 4 Gb for the whole workspace, and 16 Gb of RAM on my hard drive.

- Operations gradually become really slow with time in a Global Mapper session, and it seems that it just ends up with crushing If I insist too much.

- I need to switch from time to time from equirectangular projection (in which most my files are projected) to polar, and this takes also a lot of time. It actually seems that for reprojecting, Global Mapper read and reprojects all the files, even those which are not checked in the catalogs. I can see the succession of names of all my files on the screen, "reprojection of file A, reprojection of file B... they are all in the parade, which lasts around 5 minutes. This is for me useless, as the vast majority I don't need in polar projection; and also it does not appear to speed up the display. In polar projection, each box which I check or uncheck in the control center is followed by 1minute of processing and display time.

- Strangely enough I have a couple of images in png (with some transparent areas) which are less than 1-2 Mb in size and take 30 sec to display, whatever the projection. I have other png files (that also include transparent areas) which are 1 Mb and they display instantaneously.

I have read that there is no manual way to tune the Global Mapper memory, and that it handles it the most efficient way in every circumstances. Good. It means perhaps that there is an option somewhere which I missed, that would allow me not to have everyting processed all the time and keep the used memory to something reasonable...

thanks for your advice!

danielek

Me again... I forgot an information that may help identify the problem, or part of it. During workspace loading, Global Mapper takes a lot of time reprojecting my catalogues and some files which are not in a  catalogue, even though they are already in the projection I am using as defined in the configuration window (equirectangular/Mars MOLA sphere). There should be nothing to reproject there.

Regards,
 daniel