Export to Web Format taking days to complete!
steve_n_ottawa
Global Mapper UserTrusted User
This is what I'm trying to do: draw a 2km wide path from one end of a state to the other, crop the USGS topo pdf files along that path and then export the result to a MapBox tiles web format with only 1 zoom level.
To test the resultant size and computing time I did this: took a gpx file that goes from one end of Washington to the other. Drew a 2km wide buffer polygon from end to end. Then selected only two of the many geo pdf files that are along this path. I then cropped the topo maps based upon the selected 2km wide polygon. This all only took a few minutes. Now to export the resulting map, I export to web format....then choose the Map Box tiles....and then I also selected an option to crop to selected polygon (even though I've done this already and my resulting data set is what I want in the export).....and then press OK.
I did this 24 hours ago and the computer is STILL running. Something has to be wrong in what I'm doing or what GM is doing.
While experimenting with other formats, I tried Google Map tiles. Doing the same trail I got a warning saying that more than 1.5 million tiles would be created. It seems like GM is trying to tile the entire state not just my trail. The resulting data sets are huge. It just doesn't make sense to me...especially if I'm only doing one zoom level.
Anyone have any thoughts on this?
Thanks!
-Steve
To test the resultant size and computing time I did this: took a gpx file that goes from one end of Washington to the other. Drew a 2km wide buffer polygon from end to end. Then selected only two of the many geo pdf files that are along this path. I then cropped the topo maps based upon the selected 2km wide polygon. This all only took a few minutes. Now to export the resulting map, I export to web format....then choose the Map Box tiles....and then I also selected an option to crop to selected polygon (even though I've done this already and my resulting data set is what I want in the export).....and then press OK.
I did this 24 hours ago and the computer is STILL running. Something has to be wrong in what I'm doing or what GM is doing.
While experimenting with other formats, I tried Google Map tiles. Doing the same trail I got a warning saying that more than 1.5 million tiles would be created. It seems like GM is trying to tile the entire state not just my trail. The resulting data sets are huge. It just doesn't make sense to me...especially if I'm only doing one zoom level.
Anyone have any thoughts on this?
Thanks!
-Steve
Comments
-
Steve,
Did you check the option to 'Skip Empty Tiles' when doing the export? If not it would build an entire rectangle grid covering the tile set. The 'Skip Empty Tiles' should give you just those tiles that intersect the crop area.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
So I tried the 'Skip Empty Tiles' check box. Same results....
As an experiment, I exported just the track and the buffer of the track (IE no geo pdf files loaded and cropped).
This worked perfectly....I got about 500 tiles when a zoom level of 1 was used. And maybe 1500-2000 tiles when zoom was about 5 level. Should I not expect the same number of tile if I crop a map to the buffer that I've created?
Anything else I can try and do to figure out why I can't get this to work?
-Steve -
I've noticed GM can give a misleading # of tiles/time when using skip empty tiles as it's hard to figure that out when you dont have a standard box.
Also - geopdfs take a long time to load - makes me wonder if GM is trying to load all of those pdfs to crop at once. Could try exporting to a geotif and use those instead of the pdfs. -
Which version are you using? If you check the option to 'Skip Empty Tiles' and the export is cropped to selected area features then any tiles that don't intersect any of the areas should automatically be removed from the list of tiles to export so the tile count itself will be lower so the progress bar should be close to correct. However there have been some issues with this fixed over the releases so make sure you have the latest build. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .
For your real export, make sure to create a map catalog of the GeoPDF files and export from that. Then the PDF files will be loaded as needed rather than all at once, which is much slower. Or you could convert to GeoTIFF first which would tend to be faster, but I'd still create a map catalog of them.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
Good to know it was fixed. I havent tried it since upgrading to version 14.
Also - I did notice a speed difference when I export to mbtiles format in the past. It was a fair bit slower (also noticed with exporting a vector to spatialite/sqlite format).
I guess for the OP - what version he is using might have some weight on this. -
Is the export slower now or it was slower before and is faster now? I'm actually trying to track down a major slowdown users of the latest v14.1 build are seeing over v13 right now. I'm not sure what it is yet, but depending on what it is could be what you were seeing.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
I am using the latest build you posted above. I've since tried using tif as well....pretty much same issue. GM pretty much freezes up and while my CPU is at 100%.....I kill it after hours of waiting. If no maps are loaded then it takes only a few moments...but obviously that is useless as there is no map there.
Can I provide some debug log info or anything? Or provide the gpx file and the two tiff or pdf files and let someone else try?
I'm obviously happy to help as I'm getting no where and it's been days of trying at this point....but there has been a good learning curve to this too.
-Steve -
Steve,
If you can provide that sample data that does this I can take a look. There have been massive underlying changes lately to support the new Lidar architecture and it's quite possible I have broken something. If I can reproduce it I should be able to fix it pretty quickly.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
I got your data and took a look. The issues was that the initial tile set was created over the entire bounding box of the buffer area. Then each of those rectangles was checked to see if it intersected the buffer area, and that was taking a VERY long time as there were millions. I have added a check before that if you are skipping empty tiles to discard any tiles that don't intersect any of the data being exported. Then what is left after that is checked against the crop area. When I do this the export starts very quickly and is only going to do a few hundred tiles.
I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
Mike, do I need to do something other than what I've outlined above now?
Or can I just:
1) load the gpx
2) select the trail that is created by the gpx
3) right click on the selected trail, then create a buffer (I've tried 2km and also 100m)
4) then while the buffer is still selected I load in the two pdf or tif files
5) crop the collar on the maps
6) crop maps based upon selected polygon
7) Export to Web Format....MapBox in this case.
8) change Export Bounds to "Crop Selected Area Features...everything else is default
This still looks like it's going to take hours.
Did I miss something in the process? I must have if you got it working quickly!
-Steve -
Steve,
I forgot to mention, make sure to uncheck the option to 'Save Vector Data if Displayed' when doing the web export, otherwise your trail and buffer will also be exported and because the bounds of that layer covers the whole thing, the new speedup would have no effect!
Let me know if I can be of further assistance.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
Working on it now. In the mean time could you please educate me a bit on why when I create the buffer, if I create a wide buffer (say 2km) I end up creating 3500 Areas....but if I reduce the size the buffer, then the number of Areas goes way up...I would have expected it to go down....no?
-Steve -
Hmmm I don't see anything that looks like "Save Vector Data if Displayed". At least not in the b011013 build and in the Mbox or Google Tiles export screens. Is it hiding someplace else?
-Steve -
Steve,
The message about the area count is because the line you are buffering is actually simplified to remove insignificant vertices before creating and combining the buffer areas, so you actually end up working on less vertices with a larger buffer as the simplification threshold is a function of the buffer distance. So a 2 km buffer might simplify to 20m, whereas a 2m buffer would simplify to 0.2m, so a lot less vertices would be discarded. I think I'll remove the count from the progress bar as it can be confusing.
The MBTiles export should have a checkbox just under the Transparency section on the 'Tiling Options' tab labeled 'Save Vector Data if Displayed'. That has always been there.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
Ah....yes I see it now....but it's greyed out.
Maybe because I have a demo license?
-Steve -
Steve,
Ah yes if you have a demo license it is indeed always grayed because the big "DEMO" text pasted on the web export in demo mode uses vector data and you can't turn it off. So I guess the speedup won't work for you with a demo license, but it should be fine with a regular license.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
relating to this post, I wish to publish a GM GIS with approx 20 vector and raster layers online - conveniently, but not necessarily, within Wordpress . . . . . also peg (my) photos in fashion of panoramio . . . . any advice re served basemap and GM friendly publishing utility . . . . and re best file format (eg KML) . . . . . thanks, George
-
George,
For online publishing with layers KML files are probably your best bet as you can use those in the Google Maps interface. Another option would be one of the File->Export->Export Web Format options to create pre-tiled multi-zoom data sets for easy publishing. You would lose the layering though with a single export. You could maybe export just your raster layers to Google Maps tiles, then do a vector KML export to keep the vector layers separate and render those on top.
Thanks,
Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.bluemarblegeo.com/
Categories
- 12.7K All Categories
- 5.6K Features Discussion
- 342 Downloading Imagery
- 1.3K Elevation Data
- 380 Georeferencing Imagery Discussion
- 628 GM Script Language
- 53 User Scripts
- 113 GPS Features
- 414 Projection Questions
- 819 Raster Data
- 1.3K Vector Data
- 6.6K Support
- 177 Announcement and News
- 908 Bug Report
- 558 SDK
- 1.2K Suggestion Box
- 3.7K Technical Support
- 562 Other Discussion
- 129 GIS Data Sources
- 27 Global Mapper Showcase
- 233 How I use Global Mapper
- 107 Global Mapper Forum Website