Export Web Format - Optimization

worldworld Global Mapper UserPosts: 10Trusted User
edited October 2012 in Technical Support
We want to export a really large imagery mosaic (R/G/B bands, 500GB) to tileset. How could we optimize this process (i.e. make it faster) in terms of the following two aspects:

1 - File format: Which imagery format is more ideal for the input file - GeoTIFF, tiled GeoTIFF, JPEG2000, BIL, BSB, etc.?

2 - Resampling: Is there a way to specify this as Nearest Neighbour always? I note that even for the same resolution between input file and the desired export tileset level, the tileset is always fuzzier. Maybe this is caused by the compression for the JPEG output.

THANKS!!!

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited September 2012
    There are a lot of variables that can affect which format is the best. In general GeoTIFF will read the fastest, however because it doesn't compress as well as JPEG2000 it might be faster to use that format for input as there is less data to read from disk and more processing, making it faster if your disk is slow.

    Which Global Mapper version are you using? There should be a resampling drop-down right on the export dialog for most formats in v13.2 and later. You will see additional "fuzziness" regardless if you export to a lossy format, like JPG. The JPG quality settings controls the amount of compression vs. quality. If you turn that up to near 100 for quality there shouldn't be any noticeable loss in quality.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • worldworld Global Mapper User Posts: 10Trusted User
    edited September 2012
    We are using V13.2 and talking about "Export Web Format". For the tile format, there is no option for resampling. I wonder if there is a hidden registry switch we can use to enable the option, say NN only.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited September 2012
    There isn't a special resampling option for web exports (typically you would want the automatic selection so the sub-sampled layers would look good). You can use the option in the Advanced section of the Generic tab of the Configuration dialog to disable automatic interpolation during export if you want to just keep whatever resampling you have set for each layer when exporting.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    Just to add to this discussion: if you are talking about this size of file (500 Gb), you will get a lot of png files and thus a lot of data in your web output. For selected projects I do, I optimize the png files with a special program called png squeeze (quantile optimization). It will reduce the file size of the individual png files without lozing any information. If you are interesed let me know and i can get you started with this. (a typical reduction in file size (not number of files, I don't export blanc tiles) is between 30 and 50% depending on the type of file you export.
    Stefan
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    And to add to the tilemap discussion: I am running a large file, partly obscured by a mask. I want to tile only the visible part, no empty tiles AND I select: fill to tile-bounds. Am I correct that this last setting will fill to the extend of the largest file in the project regardless what I have zoomed in to? I.O.W. If I only want to tile parts of a larger file using a mask and want the tiles to be filled to the tilebounds (of the visible part), you would have to clip the file and process individually or don't use the fill to tilebounds...
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    And to add to the tilemap discussion: I am running a large file, partly obscured by a mask. I want to tile only the visible part, no empty tiles AND I select: fill to tile-bounds. Am I correct that this last setting will fill to the extend of the largest file in the project regardless what I have zoomed in to? I.O.W. If I only want to tile parts of a larger file using a mask and want the tiles to be filled to the tilebounds (of the visible part), you would have to clip the file and process individually or don't use the fill to tilebounds...
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited October 2012
    The 'Fill to Tile Bounds' option means that each tile in the export, where the tile boundaries are predefined for different web mapping services, will be filled with data rather than the data being cropped at whatever export bounds that you specified when exporting. So basically your export bounds are used to determine which tiles need to be exported at each zoom level, then normally those tiles would be filled with background or marked transparent outside the export bounds. If you select to fill to the tile bounds then any loaded data will be used to fill them rather than the background.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    Hello Mike, Thanks for the awnser. So basically the fill to tilebounds setting ignores the view-boundaries...

    To add to this I would like to report a problem: I am running a big tile-project. Large map, high zoom level, many tiles. I run 3 to 4 instances of the same project on the same computer. There are two problems:

    1. I get errors because two (or more) instances want to write the same output tile. Because it's locked this can't be done. An error comes up reporting the same tile can't be written to.
    Is there a way to just tell GM to ignore this error and continue (maybe writing a log entry instead and just go on). And / or (better) give one of the instances priority and the next instance to go on with another tile.

    2. related: when tiling large and complex files (with overlays and vextors loaded etc) starting a number of instances is difficult. Especially if you want to use a tile-box (you can not make that 4 exact similar boxes). It's been discussed and requested before: is it possible to start more than one instance in the same project. This way the settings are exactly the same and memory management and instance coordination could be improved (see nr 1 on the error). I now have to load 3 to 4 times the data and this is memory intensive.

    NB I am running 13.2 and testing 14 and both have the same issues. I guess the tiling did not change between these versions, except I now get an XML (14) in stead of and HTM (13.2) both I can not load or use (I use my own standard web makeup (google and open layers) and only copy the view-box coordinates.
    Stefan
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited October 2012
    Stefan,

    For issue 1, do you have the option checked to Skip Existing Tiles in each instance. Checking that should prevent multiple instances from trying to write to the same tile at once (unless you get very unlucky with timing).

    For issue 2, we do have plans (although we keep putting them off) to make large gridded exports multi-threaded or at least spawn multiple processes to automatically handle speeding up those large exports. Hopefully sooner than later this will be automatic.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    Hi Mike,
    issue 1: when tiling large maps at hi level with more than 1 instance and of course skipping existing tiles checked by definition you get unlucky at some time ;-) we are talking 1.000.000 plus tiles, running for 8 - 12 hours.

    could you add an option to ignore this problem, so the instance does not stop and wait for my OK (the next morning...)

    Issue 2: I am one of those users trying to get this point on a higher priority with regular forum entries. It would be so nice (including CUDA / hardware acceleration) to have a very fast and reliable large map tiling operation...

    Anyway: solving issue 1 would be great because now the worker instance is stopped because of this and should be easy to do to add an ignore option...
    Stefan
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited October 2012
    Stefan,

    I can pretty easily make the 'Skip Existing Tiles' a bit smarter when you do get those unlucky timings. What format are you exporting to?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited October 2012
    Stefan,

    I went ahead and made the 'skip existing tiles' smarter so that there shouldn't be any possible of unlucky timing resulting in the error when running multiple instances. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • sjfsjf Global Mapper User Posts: 10
    edited October 2012
    Thanks Mike,
    I am testing the new build, so far so good... (no conflicts)
    I do get the message that the publisher of the software could not be verified when I start up the new GM build.
    Tracking progress is difficult also: I started 5 instances but reduced to 3 because all memory is used and the computer starts swapping to disk. I have still to benchmark the optimum setting for speed and reliability, based on the timestamps of the generated tiles.

    I use the WMS tiles (which we have improved a year ago or something) with output to PNG 8 bit with transparancy. I use the tiles in google maps and open layers.

    Could you also include the update in GM 13.2...
    greetings Stefan
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited October 2012
    Stefan,

    The code signing probably just failed for the local build, no worries about the verify error.

    For performance you do probably want less instances than you have processor cores, probably even 2 less. So if you have 8 cores run 6 instances. I suppose if you only have 4 then 3 would probably be ok.

    The change was actually a bit involved and leveraged some new stuff so back-porting to v13.2 wouldn't be easy. We would normally only bug port significant bugs to older versions anyway.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
Sign In or Register to comment.