Improve speed of reprojection

STHSTH Global Mapper UserPosts: 434Trusted User
edited July 2010 in Suggestion Box
Naturally it will take a long time to do the following operation:

Reproject from UTM32 to UTM33 (this is quite fast if you do not "force" the GSD to be the same but let GlobalMapper suggest it)

Then the long operation will be to cut the new tiles into another tilesystem. This is due to the following:

1. Background-color black have to be set to transparent to remove the "reprojection border"
2. Bicubic resampling to keep the quality
3. Resample to the correct GSD

Even with a fast computer and alot of RAM this takes alot of time. I notice that GlobalMapper does not use much RAM for this operation. If GlobalMapper was "smart" it would read all the files first into memory (provided there is enough space) and then do the operation in memory and then export the file to disk. How is GlobalMapper currenctly working doing the reprojection? It takes 15 minutes for 1 file to complete now, and I have 600 files... Even if I could run 10 instances simultaneously it would mean only 40 files an hour = 15 hours processingtime. And I am not sure if the disks can handle 10 instances reading and writing at once.

Any suggestions to improve the workflow is appreciated. I also tried to do all in one (ie no batch conversion of UTM32 to UTM33 first, but straight from the UTM32 tiles to the UTM33 tiles - but this seemed to take more time).

Comments

  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Ok, so something is definitely wrong. It takes 3 hours per file to create - so with all the files it estimates to be finished 10th of July.... Not sure what is really the issue though - and most importantly the solution...
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    To clarify even more, here are the data:

    Input: UTM32, WGS84: TIF (no GeoTIFF, uncompressed 24bit, 4 channels RGB data, 10 000 x 10 000 pixels, approx 400 MB each file) - total number of 700 files.

    Output: UTM33, WGS84: TIF (uncompressed 24bit, 4 channel RGB, 10 000 x 10 000 - however new tiling)

    I notice (if the ProcessExplorer is showing the correct values) that GlobalMapper seems to READ 2.6 GB (!!!) constantly while the progress bar moves forward. Memory usage is only 3.3 GB and not moving at all during the time.

    I have tried both the 32 and 64-bit version of GlobalMapper to see if there are any differences with no luck :(

    I have also tried with a much smaller export-size with the same result
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    That does seem excessive, is it possible for you to provide one of the data files that is taking so long to convert so that I can see what is taking so long?

    In terms of memory, for the most part a particular part of the file is only needed once, so there really isn't any benefit to read it all into memory up front rather than just reading each part when needed since it both cases each area of the file is read only once, and in the case where it is read on-demand much less system resources are required. The instance where this is not the case is if there is enough rotation/warping where the in-memory caching of each file as it is read becomes inefficient and each area ends up needing read multiple times.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Finished uploading files in 15 minutes to /UTM32_UTM33

    4 example files.

    When I use the this_workspace_OK.gmw it exports quickly to the UTM33 with export - raster.

    When I use the convert_script_from_directory.gmw to load the files (afterwards I have exported the workspace to convert_script_with_files.gmw) and try the same mapsheet (6114_488) then it takes forever to export. Not sure if it is the amount of files loaded that is the problem?

    Thanks for the quick look.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    I'll take a look when I get a chance (I'm actually in Vegas on vacation through Monday night).

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Thanks alot - untill then I will try to find another solution, however I am not sure if there is another quick-fix available with another software.
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Not sure if it matters, however I have noticed that GlobalMapper writes to the output file all the time (using a software called Disk Pulse - http://www.diskpulse.com/ you can see the files that are accessed in real-time). Perhaps this demands too many disc-accesses when working with large files like these. If a computer have alot of memory - perhaps it is better to completely write the new file in memory first, and when the file is complete write it to the disc. For this case the input files in UTM32 is typically 4*400 MB = 1.6 GB for one output-tile in UTM33 (400MB) so needed memory to store this is 2 GB. "Worst-case" scenario will perhaps be that some tiles have to load 6 files (2.4GB). Since the computer have 10 GB of RAM it should not be a problem to store these files in RAM.

    Starting to get very detailed: for a "clever" GlobalMapper with access to alot of RAM it will "reuse" the images or reprojections on the adjacent tile and not have to re-read these files. This approach is just a suggestion for future improvements. And for converting large files of data this can have a good impact. For my project one input-file typically will be read 4 times (it is used in 4 different output-tiles). If it instead could have been read 2 times that would save a read of 800 MB. Multiplying these by 700 files = 560 GB of reads that can be saved (I am assuming that GlobalMapper reads the complete file - its an uncompressed, untiled TIFF so its correct for this case, not sure if I first convert all the files to LZW-compressed and tiled TIFF-files if the reprojection to UTM33 will be faster

    I would convert the input-files to smaller output-files in UTM33 and then cut them to the correct tiling afterwards, however this does not seem to improve the speed.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    The actual writing of data to disk for the TIFF file is handled by the libtiff library (the de-facto standard library for TIFF support). I believe it is written in chunks rather than a bunch of tiny writes (which would be slower). The library has been around for a long time and is still actively developed, so I'm pretty sure it is optimized pretty well.

    Larger cache sizes in Global Mapper would help things out a bit, I'm actually guessing that when your data is reprojected from UTM32 to UTM33 that it rotates enough that the cache isn't large enough to store a single export row of tiles without having to flush. I'll confirm when I take a look at your data (probably tomorrow) and hopefully be able to do something about it, at least in the 64-bit version where more memory can be used.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Yes, I will need to use the 64-bit version so an update to that would be very welcome. I do see disc-activity every 2-3second so I guess it uses the flush on the disk.

    Thanks for improving the software, I will give you some numbers and processing-times after the test with the new software so you can use as "advertisement" for the software if you want. Its always nice to have: "GlobalMapper can transform from coordinate system X to Y within 12 hours....." and have some description on the test-dataset.
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    FYI I have now managed to load the same files in another software and is reprojecting the same files there. It takes much shorter time. It takes approximately 36 files per hour, while the GlobalMapper runs with approximately 12 an hour. The only difference I can see (to help improve GlobalMapper) is that it first calculates the "bounding box" if the images and use this to define the areas of which images it should use in each tile. The visual quality equal in the output-files and input-files (it is not possible to choose the resampling method used).

    Hope this is helpful feedback to improve GlobalMapper and working with large datasets.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    The issue does indeed seem to be that when reprojected enough that the image is rotated significantly that there is not enough cache allocated for each image to prevent having to re-read the same section of image repeatedly. When you load a bunch of TIFF files, the earlier ones that are loaded get larger caches and faster access, but when lots are loaded the later ones revert to smaller caches and slower access to save memory space.

    I did go ahead and increase how much memory can be cached for the files, especially for the 64-bit version, which may help when you have a lot loaded, but I would suggest rather than loading them individually to pull them all into a map catalog and export from that so that the files can be loaded and unloaded as needed for export, which will keep fewer in memory and allow more of them to use a larger cache size.

    I have placed a new build at http://www.globalmapper.com/global_mapper11.zip with the change for you to try. Simply download that file and extract the contents into your existing v11.xx installation folder to give it a try. If you are using the 64-bit version, there is a new build available at http://www.globalmapper.com/global_mapper11_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Thanks. I loaded the new 64-bit version and confirmed the 29th of June build date. I do not however see much difference in speed or less use of SWAP. It is still reading 2 GB from the swap and it estimates roughly 1 hour to finish one of the tiles (not the same example as I sent you). It is still using "only" approximately 2.3 GB when running two instances of GlobalMapper and when running two GlobalMapper at the same time the memory-usage does not increase.

    On some of the tiles there are even more than 4 images that have to be loaded to complete the tile.

    MapCatalogue is not possible due to working with 4-band RGBI-images.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    Are you using the 64-bit version? Are you doing a batch conversion or directly loading and then exporting cropped to an area or bounding box?

    I suspect the latter, what you could do to speed this up is to batch convert the individual tiles to the new projection (make sure to mark the background as transparent), then load those results and export from that to avoid the cache speed issues with having lots of very rotated images loaded.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Using hte 64-bit version. Loading all files similar to the workspace I sent you (only many more TIFFs), no workspace. Export raster - GeoTIFF - selecting Gridding to selected areas. No cropping selected.

    If I first batch reproject to UTM33 I will have to turn on "make black pixels transparent" when mosaicing together and I suspect this will also take alot of time based on previous experience.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    I think the best solution for this might be to allow setting band layout options for map catalogs that contain multi-band layers. I will take a look at adding this option and get back to you.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited June 2010
    I have completed updating map catalogs to allow modifying the band layout of layers within a map catalog. You will need to create a new map catalog from your data using the new build for this to work properly. I have placed a new build at http://www.globalmapper.com/global_mapper11.zip with the change for you to try. Simply download that file and extract the contents into your existing v11.xx installation folder to give it a try. If you are using the 64-bit version, there is a new build available at http://www.globalmapper.com/global_mapper11_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STHSTH Global Mapper User Posts: 434Trusted User
    edited June 2010
    Hi, I really appreciate you looking into this.

    I downloaded and installed the new version and ran the 64-bit version (Build Jun 30 2010 11:31:21).

    I created a new mapcatalogue with the same files and did the same as before. First file should take 1 hour to complete - let it run for 15 minutes and it seemed to not be changed. I noticed the disk read/write speed was very low (always less than 1MB/sec) and that the RAM used did NOT change at all before the export started and meanwhile it ran (using 2.7 GB before and 2.7 GB while running).
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited July 2010
    I made some additional improvements by detecting poor cache performance and automatically growing the cache for a layer when that happens. I just tried the 64-bit version on the files that you provided and export the tile in the middle that intersects all of the tiles and it exported a new 24-bit GeoTIFF in about 2 minutes. Hopefully you can get comparable performance.

    I have placed a new build at http://www.globalmapper.com/global_mapper11.zip with the change for you to try. Simply download that file and extract the contents into your existing v11.xx installation folder to give it a try. If you are using the 64-bit version, there is a new build available at http://www.globalmapper.com/global_mapper11_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
Sign In or Register to comment.