Global Mapper v25.0

Major Difference in Rendering Times

digitalaviator
digitalaviator Global Mapper UserTrusted User
edited February 2010 in Bug Report
Hi Mike,

I seem to have a huge difference in render times between exporting a gridded GeoTIFF by loading all data into GM11 (latest build) compared to doing it via a .GMS.

Doing it via the GM software and manually takes 30-60 mins compared to 3-10 mins if done via a .GMS

Should this be the case?

Kindest Regards,
Dean.

Comments

  • global_mapper
    global_mapper Administrator
    edited February 2010
    Dean,

    I would not expect any significant time difference between loading everything into GM via the user interface or using a script, assuming the exact same export was being done in both cases. What kinds of export options are you specifying? Do you have a large number of GeoTIFF files? If so you would get better performance by creating a map catalog (File->Create New Map Catalog) from all of the GeoTIFF files and then exporting from that.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    I noticed the same thing a while back using GM10...

    I do have a substantial number of B&W geotiffs at 1m resolution for all of California...

    I'll try the Create New Map Catalog option and see if that helps improve performance, at this point generating a gridded set of B&W tiled images for one county alone will take 11 days, compared to just 2-3 days for the ortho imagery which is coming from MrSID and then tiling out to geotiff and then doing color correction in photoshop then being loaded back into GM then tiling back out to the same format as the B&W water masks I'm trying to generate above...

    I would have expected minutes for the entire country B&W imagery to render...

    Anyway I'll test out your recommendation and see if that makes a difference...

    Regards,
    Dean.
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Mike,

    Negative on the speed of export using Map Catalog it's still extremely slow...

    I'll have to write a GMS for what I'm doing and benchmark the speeds as there are massively huge differences in rendering times...

    I'll run the tests and get back to you with the figures...

    Dean.
  • global_mapper
    global_mapper Administrator
    edited February 2010
    Dean,

    One other thought, if you are cropping to large complex polygons this will slow things down tremendously. If you are going to buffer the areas anyway, what you might do after that is use the simplify option in the Digitizer Tool to remove a large number of the vertices while maintaining the same basic shape, then crop to those polygons. This would speed up the export tremendously.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    I'm sure there's a bug here, doing a tile export from MrSid and tiling out to 10,000x10,000 takes seconds...

    Using the script for the 2nd phase where I load those 10,000x10,000 geotiff tiles back in then reproject and crop and export under GMS also takes only a short amount of time 1-5 mins for each gridded tile...

    If I do the same thing manually from within Global Mapper it takes 10x longer at least...

    As I said I'll get the benchmarking results over to you, as it shouldn't be taking 30-60 mins to export a Black and White 5849x5849 pixel tile from GeoTIFF to GeoTIFF using the same projection... Not when it's doing the exact same process via the GMS we've generated on 24bit LZW compressed files in 3-4 mins...

    As I've said there is a major time difference between tiling out exactly the same way manually within Global Mapper compared to doing the exact same process via GMS...

    Cheers,
    Dean.

    P.S. the polygon shouldn't be an issue here as the exact same polygon is being used for the crop, the speed of export is dependent upon the method i.e. manually choosing export via GM or doing it via GMS
  • global_mapper
    global_mapper Administrator
    edited February 2010
    Dean,

    Are you exporting tiles of 10,000 x 10,000 pixels in size, or gridding to 10,000 by 10,000 cells? I'm just wondering if you are generating a large number of exported tiles if the reporting of the progress is actually taking longer than the actual export. Since the script wouldn't report progress this would be a potential source of time difference. You could try minimizing the Global Mapper window to see if that improves the performance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    What I've got is about 150 indexed black and white lzw compressed tiles being loaded into global mapper each of about 5000x5000 pixels in dimension @ 1m per pixel post spacing.

    I load them all into Global Mapper and then load my county boundary and select it, then go to export to GeoTiff using same projection...

    Options are as follows:

    Export Bounds - Crop to Selected Polygon (it's a county boundary so isn't complicated)

    Gridding - Pixel Width 5849 x Pixel Height 5849, Separate Row/Column Numbers selected as numbers start at 1, step 1, prepend 0's separate rows/columns with underscore, overlap grid cells by 0.1 percent of cell size

    GeoTIFF Options - 8 bit grayscale palette, x axis 0.000008015317222595210000, y axis 0.000010687089630127000000, always generate square pixels, generate TFW and generate PRJ


    going that approach is 30 mins approx for just one 5849x5849pixel tile...

    now under the .gms approach with the 24 bit imagery I've been loading the same area with 10,000x10,000pixel 24 bit geotiffs, loading the polygon shp file with polygoncropuseall and exporting out to 24bit geotiff and it's taking at most 3 days to render...

    my calculations have the black and white maps taking at least 11 days for the same area of coverage under the same x/y pixel dimensions...

    note: the same slowness applied when I was doing the 24 bit exports from within GM as well...

    I'll have to run to town for a few hours to pick up a passport but I'll generate the .GMS tonight and will get back to you with the benchmarks...

    Cheers,
    Dean.

    EDIT: To effectively run this benchmark I'll have to spend a couple of days to give an accurate benchmark ;-)
  • global_mapper
    global_mapper Administrator
    edited February 2010
    Dean,

    Ah, I didn't realize that you were running different exports from the user interface than via script. I thought you were seeing vastly different performance with the exact same input. I'm guessing there is something about the black-and-white GeoTIFF files causing the slower performance, likely a poor caching performance issue or something.

    I have recently increased some of the cache sizes for GeoTIFF files so that may help the performance if that is the issue. Can you try getting the latest build to see if that helps? I have placed a new build at http://www.globalmapper.com/global_mapper11.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v11.xx installation folder to give it a try. If you are using the 64-bit version, there is a new build available at http://www.globalmapper.com/global_mapper11_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • STH
    STH Global Mapper User Trusted User
    edited February 2010
    I had a similar issue at the end of this thread (http://www.globalmapperforum.com/forums/raster-data/5027-change-tilesize-export-geotiff.html).

    I tried to "work around" the fact that tilesize did not work for batch reproject/convert and use a MapCatalogue with the TIF-images (approx 550 all 3 band RGB images) and crop to currently selected polygon (a relatively simple polygon) and then Export Raster, Gridding according to file-definition.

    I also noticed roughly 30 minutes for finishing one file, while when doing batch reproject/convert with the same data (as I can now due to tilesize is included in the batch reproject/convert) then the same files take roughly a couple of minutes. I havent tried creating a GMS with the same procedure yet. And I only tried it on the GMx64 version. If you want me to do some more tests to help please let me know.
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    Negative, same input and same output and same crop to polygon with exact same settings, one is done within the global script and is fast the other is done by manual export from Global Mapper and is 10-20 times slower...

    I've got my master script running right now and our software tool that builds all the GMS scripts has everything time stamped, so i'll be able to show the time differences with a direct comparison probably within 24-48 hours...

    I don't want to confuse things but this is how fast it is to load a mr sid source file for one county in southern california and tile out to 24 bit RGB 10,000x10,000 GeoTIFF tiles on stage 1 via the GMS without polygon cropping...
    2/24/2010 12:20:42 PM: ....fsd_v_ca_imp_01_01.tif processed.
    2/24/2010 12:22:16 PM: ....fsd_v_ca_imp_01_02.tif processed.
    2/24/2010 12:23:27 PM: ....fsd_v_ca_imp_01_03.tif processed.
    2/24/2010 12:24:55 PM: ....fsd_v_ca_imp_01_04.tif processed.
    2/24/2010 12:26:25 PM: ....fsd_v_ca_imp_01_05.tif processed.
    2/24/2010 12:28:06 PM: ....fsd_v_ca_imp_01_06.tif processed.
    2/24/2010 12:29:55 PM: ....fsd_v_ca_imp_01_07.tif processed.
    2/24/2010 12:31:47 PM: ....fsd_v_ca_imp_01_08.tif processed.
    2/24/2010 12:33:33 PM: ....fsd_v_ca_imp_01_09.tif processed.
    2/24/2010 12:35:35 PM: ....fsd_v_ca_imp_01_10.tif processed.
    2/24/2010 12:37:23 PM: ....fsd_v_ca_imp_01_11.tif processed.
    2/24/2010 12:39:11 PM: ....fsd_v_ca_imp_01_12.tif processed.
    2/24/2010 12:41:03 PM: ....fsd_v_ca_imp_01_13.tif processed.
    2/24/2010 12:42:57 PM: ....fsd_v_ca_imp_01_14.tif processed.
    2/24/2010 12:44:39 PM: ....fsd_v_ca_imp_01_15.tif processed.
    2/24/2010 12:45:35 PM: ....fsd_v_ca_imp_01_16.tif processed.
    2/24/2010 12:46:19 PM: ....fsd_v_ca_imp_02_01.tif processed.
    2/24/2010 12:48:11 PM: ....fsd_v_ca_imp_02_02.tif processed.
    2/24/2010 12:50:04 PM: ....fsd_v_ca_imp_02_03.tif processed.
    2/24/2010 12:51:48 PM: ....fsd_v_ca_imp_02_04.tif processed.
    2/24/2010 12:53:28 PM: ....fsd_v_ca_imp_02_05.tif processed.
    2/24/2010 12:55:06 PM: ....fsd_v_ca_imp_02_06.tif processed.
    2/24/2010 12:57:06 PM: ....fsd_v_ca_imp_02_07.tif processed.
    2/24/2010 12:59:06 PM: ....fsd_v_ca_imp_02_08.tif processed.
    2/24/2010 1:01:08 PM: ....fsd_v_ca_imp_02_09.tif processed.
    2/24/2010 1:03:23 PM: ....fsd_v_ca_imp_02_10.tif processed.
    2/24/2010 1:05:24 PM: ....fsd_v_ca_imp_02_11.tif processed.
    2/24/2010 1:07:25 PM: ....fsd_v_ca_imp_02_12.tif processed.
    2/24/2010 1:09:26 PM: ....fsd_v_ca_imp_02_13.tif processed.
    2/24/2010 1:11:31 PM: ....fsd_v_ca_imp_02_14.tif processed.
    2/24/2010 1:13:23 PM: ....fsd_v_ca_imp_02_15.tif processed.
    2/24/2010 1:14:24 PM: ....fsd_v_ca_imp_02_16.tif processed.
    2/24/2010 1:15:12 PM: ....fsd_v_ca_imp_03_01.tif processed.
    2/24/2010 1:16:55 PM: ....fsd_v_ca_imp_03_02.tif processed.
    2/24/2010 1:18:37 PM: ....fsd_v_ca_imp_03_03.tif processed.
    2/24/2010 1:20:36 PM: ....fsd_v_ca_imp_03_04.tif processed.
    2/24/2010 1:21:48 PM: ....fsd_v_ca_imp_03_05.tif processed.
    2/24/2010 1:23:18 PM: ....fsd_v_ca_imp_03_06.tif processed.
    2/24/2010 1:25:17 PM: ....fsd_v_ca_imp_03_07.tif processed.
    2/24/2010 1:27:20 PM: ....fsd_v_ca_imp_03_08.tif processed.
    2/24/2010 1:29:17 PM: ....fsd_v_ca_imp_03_09.tif processed.
    2/24/2010 1:31:32 PM: ....fsd_v_ca_imp_03_10.tif processed.
    2/24/2010 1:33:31 PM: ....fsd_v_ca_imp_03_11.tif processed.
    2/24/2010 1:35:32 PM: ....fsd_v_ca_imp_03_12.tif processed.
    2/24/2010 1:37:36 PM: ....fsd_v_ca_imp_03_13.tif processed.
    2/24/2010 1:39:37 PM: ....fsd_v_ca_imp_03_14.tif processed.
    2/24/2010 1:41:37 PM: ....fsd_v_ca_imp_03_15.tif processed.
    2/24/2010 1:42:41 PM: ....fsd_v_ca_imp_03_16.tif processed.
    
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    STH wrote: »
    I had a similar issue at the end of this thread (http://www.globalmapperforum.com/forums/raster-data/5027-change-tilesize-export-geotiff.html).

    I tried to "work around" the fact that tilesize did not work for batch reproject/convert and use a MapCatalogue with the TIF-images (approx 550 all 3 band RGB images) and crop to currently selected polygon (a relatively simple polygon) and then Export Raster, Gridding according to file-definition.

    I also noticed roughly 30 minutes for finishing one file, while when doing batch reproject/convert with the same data (as I can now due to tilesize is included in the batch reproject/convert) then the same files take roughly a couple of minutes. I havent tried creating a GMS with the same procedure yet. And I only tried it on the GMx64 version. If you want me to do some more tests to help please let me know.

    I believe this is a different issue altogether.
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    I've run my benchmark test and have found that the following .gms script renders 4 tiles per minute:
    GLOBAL_MAPPER_SCRIPT VERSION=1.00 ENABLE_PROGRESS=YES
    UNLOAD_ALL
    
    DEFINE_PROJ PROJ_NAME="GEO_WGS84"
    Projection     GEOGRAPHIC
    Datum          WGS84
    Zunits         NO
    Units          DD
    Xshift         0.000000
    Yshift         0.000000
    Parameters
    END_DEFINE_PROJ
    
    LOAD_PROJECTION PROJ_NAME="GEO_WGS84"
    
    SET_LOG_FILE FILENAME="E:\test\globalmapper.log"
    SET_BG_COLOR COLOR=RGB(255,255,255)
    
    IMPORT_DIR_TREE DIRECTORY="E:\INPUT_FOLDER" FILENAME_MASKS="*.TIF" TYPE="GEOTIFF" ANTI_ALIAS="NO"
    
    EXPORT_RASTER FILENAME="E:\test\Water\outputfilename.tif" TYPE=GEOTIFF SPATIAL_RES=0.000008015317222595210000,0.000010687089630127000000 FORCE_SQUARE_PIXELS=YES GEN_WORLD_FILE=YES GEN_PRJ_FILE=YES GRID_TYPE_PIXEL_SIZE="5849,5849" GRID_OVERLAP="0.1" GRID_NAMING=SEPARATE GRID_NAMING_COLS="NUM,01,,1" GRID_NAMING_ROWS="NUM,01,,1" GRID_NAMING_PREPEND_ZEROES=YES POLYGON_CROP_FILE="J:\NAIP\California\2009\Imperial\imperial county.shp" POLYGON_CROP_USE_ALL=YES USE_LZW=YES BG_TRANSPARENT=YES OVERWRITE_EXISTING=NO
    

    If I do the same process manually within global mapper (i.e. loading source tiles, reprojecting to WGS84, loading the .shp file, selecting the .shp file and exporting with the exact same settings as in the script above, i.e. from the export to raster menu) it renders 4 tiles in 2 hours, i.e. 30 mins per tile compared to 15 seconds per tile.

    I really believe there's something funky going on when doing this process manually within the global mapper program interface...

    Hope this helps target the problem...

    Kindest Regards,
    Dean.
  • global_mapper
    global_mapper Administrator
    edited February 2010
    Dean,

    In the GM user interface is the option to include vector data if displayed checked or unchecked when you start the export?

    Is it possible for you to provide the SHP file and say the first row of TIFF files so that I can try and reproduce your results? You can FTP the files to ftp.globalmapper.com with a username of 'upload' and a password of 'upload'.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • digitalaviator
    digitalaviator Global Mapper User Trusted User
    edited February 2010
    Hi Mike,

    Negative on include vector data if displayed.

    It's 12:30am here so I'll have to shoot over the upload in about 6-7 hours.

    I'll grab screenshots of the menu options as well...

    Cheers,
    Dean.