Global Mapper v25.0

Export to Web Format taking days to complete!

steve_n_ottawa
steve_n_ottawa Global Mapper UserTrusted User
edited January 2013 in Technical Support
This is what I'm trying to do: draw a 2km wide path from one end of a state to the other, crop the USGS topo pdf files along that path and then export the result to a MapBox tiles web format with only 1 zoom level.
To test the resultant size and computing time I did this: took a gpx file that goes from one end of Washington to the other. Drew a 2km wide buffer polygon from end to end. Then selected only two of the many geo pdf files that are along this path. I then cropped the topo maps based upon the selected 2km wide polygon. This all only took a few minutes. Now to export the resulting map, I export to web format....then choose the Map Box tiles....and then I also selected an option to crop to selected polygon (even though I've done this already and my resulting data set is what I want in the export).....and then press OK.
I did this 24 hours ago and the computer is STILL running. Something has to be wrong in what I'm doing or what GM is doing.
While experimenting with other formats, I tried Google Map tiles. Doing the same trail I got a warning saying that more than 1.5 million tiles would be created. It seems like GM is trying to tile the entire state not just my trail. The resulting data sets are huge. It just doesn't make sense to me...especially if I'm only doing one zoom level.
Anyone have any thoughts on this?
Thanks!
-Steve

Comments

  • global_mapper
    global_mapper Administrator
    edited January 2013
    Steve,

    Did you check the option to 'Skip Empty Tiles' when doing the export? If not it would build an entire rectangle grid covering the tile set. The 'Skip Empty Tiles' should give you just those tiles that intersect the crop area.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    So I tried the 'Skip Empty Tiles' check box. Same results....
    As an experiment, I exported just the track and the buffer of the track (IE no geo pdf files loaded and cropped).
    This worked perfectly....I got about 500 tiles when a zoom level of 1 was used. And maybe 1500-2000 tiles when zoom was about 5 level. Should I not expect the same number of tile if I crop a map to the buffer that I've created?
    Anything else I can try and do to figure out why I can't get this to work?
    -Steve
  • redbeard
    redbeard Global Mapper User Trusted User
    edited January 2013
    I've noticed GM can give a misleading # of tiles/time when using skip empty tiles as it's hard to figure that out when you dont have a standard box.

    Also - geopdfs take a long time to load - makes me wonder if GM is trying to load all of those pdfs to crop at once. Could try exporting to a geotif and use those instead of the pdfs.
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Which version are you using? If you check the option to 'Skip Empty Tiles' and the export is cropped to selected area features then any tiles that don't intersect any of the areas should automatically be removed from the list of tiles to export so the tile count itself will be lower so the progress bar should be close to correct. However there have been some issues with this fixed over the releases so make sure you have the latest build. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .

    For your real export, make sure to create a map catalog of the GeoPDF files and export from that. Then the PDF files will be loaded as needed rather than all at once, which is much slower. Or you could convert to GeoTIFF first which would tend to be faster, but I'd still create a map catalog of them.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • redbeard
    redbeard Global Mapper User Trusted User
    edited January 2013
    Good to know it was fixed. I havent tried it since upgrading to version 14.

    Also - I did notice a speed difference when I export to mbtiles format in the past. It was a fair bit slower (also noticed with exporting a vector to spatialite/sqlite format).

    I guess for the OP - what version he is using might have some weight on this.
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Is the export slower now or it was slower before and is faster now? I'm actually trying to track down a major slowdown users of the latest v14.1 build are seeing over v13 right now. I'm not sure what it is yet, but depending on what it is could be what you were seeing.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    I am using the latest build you posted above. I've since tried using tif as well....pretty much same issue. GM pretty much freezes up and while my CPU is at 100%.....I kill it after hours of waiting. If no maps are loaded then it takes only a few moments...but obviously that is useless as there is no map there.
    Can I provide some debug log info or anything? Or provide the gpx file and the two tiff or pdf files and let someone else try?
    I'm obviously happy to help as I'm getting no where and it's been days of trying at this point....but there has been a good learning curve to this too.
    -Steve
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Steve,

    If you can provide that sample data that does this I can take a look. There have been massive underlying changes lately to support the new Lidar architecture and it's quite possible I have broken something. If I can reproduce it I should be able to fix it pretty quickly.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • global_mapper
    global_mapper Administrator
    edited January 2013
    I got your data and took a look. The issues was that the initial tile set was created over the entire bounding box of the buffer area. Then each of those rectangles was checked to see if it intersected the buffer area, and that was taking a VERY long time as there were millions. I have added a check before that if you are skipping empty tiles to discard any tiles that don't intersect any of the data being exported. Then what is left after that is checked against the crop area. When I do this the export starts very quickly and is only going to do a few hundred tiles.

    I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    Mike, do I need to do something other than what I've outlined above now?
    Or can I just:
    1) load the gpx
    2) select the trail that is created by the gpx
    3) right click on the selected trail, then create a buffer (I've tried 2km and also 100m)
    4) then while the buffer is still selected I load in the two pdf or tif files
    5) crop the collar on the maps
    6) crop maps based upon selected polygon
    7) Export to Web Format....MapBox in this case.
    8) change Export Bounds to "Crop Selected Area Features...everything else is default
    This still looks like it's going to take hours.
    Did I miss something in the process? I must have if you got it working quickly!
    -Steve
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Steve,

    I forgot to mention, make sure to uncheck the option to 'Save Vector Data if Displayed' when doing the web export, otherwise your trail and buffer will also be exported and because the bounds of that layer covers the whole thing, the new speedup would have no effect!

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    Working on it now. In the mean time could you please educate me a bit on why when I create the buffer, if I create a wide buffer (say 2km) I end up creating 3500 Areas....but if I reduce the size the buffer, then the number of Areas goes way up...I would have expected it to go down....no?
    -Steve
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    Hmmm I don't see anything that looks like "Save Vector Data if Displayed". At least not in the b011013 build and in the Mbox or Google Tiles export screens. Is it hiding someplace else?
    -Steve
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Steve,

    The message about the area count is because the line you are buffering is actually simplified to remove insignificant vertices before creating and combining the buffer areas, so you actually end up working on less vertices with a larger buffer as the simplification threshold is a function of the buffer distance. So a 2 km buffer might simplify to 20m, whereas a 2m buffer would simplify to 0.2m, so a lot less vertices would be discarded. I think I'll remove the count from the progress bar as it can be confusing.

    The MBTiles export should have a checkbox just under the Transparency section on the 'Tiling Options' tab labeled 'Save Vector Data if Displayed'. That has always been there.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • steve_n_ottawa
    steve_n_ottawa Global Mapper User Trusted User
    edited January 2013
    Ah....yes I see it now....but it's greyed out.
    Maybe because I have a demo license?
    -Steve
  • global_mapper
    global_mapper Administrator
    edited January 2013
    Steve,

    Ah yes if you have a demo license it is indeed always grayed because the big "DEMO" text pasted on the web export in demo mode uses vector data and you can't turn it off. So I guess the speedup won't work for you with a demo license, but it should be fine with a regular license.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • George Whitworth
    George Whitworth Global Mapper User
    edited January 2013
    relating to this post, I wish to publish a GM GIS with approx 20 vector and raster layers online - conveniently, but not necessarily, within Wordpress . . . . . also peg (my) photos in fashion of panoramio . . . . any advice re served basemap and GM friendly publishing utility . . . . and re best file format (eg KML) . . . . . thanks, George
  • global_mapper
    global_mapper Administrator
    edited January 2013
    George,

    For online publishing with layers KML files are probably your best bet as you can use those in the Google Maps interface. Another option would be one of the File->Export->Export Web Format options to create pre-tiled multi-zoom data sets for easy publishing. You would lose the layering though with a single export. You could maybe export just your raster layers to Google Maps tiles, then do a vector KML export to keep the vector layers separate and render those on top.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/