Global Mapper v25.0

Managing a large set of Google Maps tiles

Boyd
Boyd Global Mapper UserTrusted User
edited January 2013 in Raster Data
I'm working an a fair sized project that involves creating map tiles from LIDAR and vector data using File > Export Web Format. You can see an alpha version of the website here: http://boydsmaps.com/#lidar,39.809974,-74.609699,15

See the image below where each of the cells is a USGS 24k quad. The yellow ones have already been uploaded and the others will be added over a period of time. I have chosen points in the corners of my dataset and use those as export bounds so that the getTileURL() code is the same and I don't have to change the Javascript each time I add a new quad. This works very well.

But my problem is that I seem to need to regenerate the entire dataset whenever I add a new quad. I tried just adding one new quad using the option to skip blank tiles and the result was a blank white border around the edges of the new tile that covered up adjacent tiles.

Am I doing something wrong here? Is it possible to selectively add tiles to an existing dataset without affecting the ones that I previously created? I live out in the sticks and a 2.5M DSL connection is the most bandwith available. My project currently consists of about 20,000 tiles and 350MB so it would be nice to find a way to do this. :)



project.png

Comments

  • global_mapper
    global_mapper Administrator
    edited January 2013
    Is the border that you are seeing just when zoomed out? The web format will have multiple zoom levels of resolution, so the lower zoom resolution tiles will likely be much larger than your updated area, so if the other data isn't also loaded you will be replacing an earlier filled in large tile with just data in the area of the new data. Unfortunately there isn't really a way around that, you need to have all data loaded that covers the largest tile (i.e. lowest zoom level) that the web export generates otherwise there won't be anything to fill it in.

    Now what you might do to emulate this if you already have the existing data online is load your web site data as a custom online source, then load your new updated tiles on top so they replace the pre-existing data. Do the web export and set the bounds just to that of your new tile, and also check the option to 'ADVANCED: Fill to Tile Bounds'. This way each tile will be filled all the way out to its bounds, which will fill from the loaded background source, which should be the same as what you had before.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • Boyd
    Boyd Global Mapper User Trusted User
    edited January 2013
    Thanks Mike, but that doesn't really sound like the solution I was hoping for. No big deal however, I will just continue to upload the entire tileset when I do an update. I have been loading them to an alternate directory and then just changing one line of code on the site to point to the new tiles when the update is complete.

    One other question if I might... are there any practical limits to the size of the source Globalmapper raster image that can be exported to google map tiles? I am running the 64 bit version. My Globalmapper files have a background DEM raster image that is overlaid with several vector layers for things like roads and hydro. If I convert these all to raster format (like GeoTIFF) as I go, would it be faster to export the map tiles? When I load all the existing Globalmapper files now, it results in a lot of layers.

    Thanks again for your help.
  • global_mapper
    global_mapper Administrator
    edited January 2013
    You won't need to do the whole data set if you have everything available and just limit to the bounds of the new tiles. Then you can overwrite the existing tile set with the new tree of tiles as the names should be the same.

    Global Mapper doesn't limit you so much on the Google Maps tile export, but if you start doing large areas at high resolution you start getting into millions of tiles which cause problems with upload and the file system. I think most systems that server huge amounts of data only pre-tile the lower resolutions and dynamically generate the most detailed layers to avoid having so many files.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • redbeard
    redbeard Global Mapper User Trusted User
    edited January 2013
    I think what Mike said would work. If you export the new area to the tile bounds, then only upload that new set overwriting the duplicate tiles -

    Also - http://wiki.openstreetmap.org/wiki/Tile_Disk_Usage gives you an idea of the prerender tiles
  • Boyd
    Boyd Global Mapper User Trusted User
    edited January 2013
    if you already have the existing data online is load your web site data as a custom online source, then load your new updated tiles on top so they replace the pre-existing data. Do the web export and set the bounds just to that of your new tile, and also check the option to 'ADVANCED: Fill to Tile Bounds'. This way each tile will be filled all the way out to its bounds, which will fill from the loaded background source, which should be the same as what you had before.

    OK, I'm still trying to wrap my brain around your earlier suggestion... sorry, I am new to web based mapping. I'm using code based on the automatically generated GM getTileUrl() and getNormalizedCoord() functions to actually fetch the tiles on the website. Wouldn't I need to run a WMS server on my site in order to use a custom online source?

    And why would that be preferable to loading the tiles locally from my PC which already contains all the tiles from the existing site? Further showing my ignorance, what exactly does the "fill to tile bounds" option do? How is "tile bounds" defined? And while I'm at it, what is the difference with exporting a basemap instead of an overlay? Does that mean that no Google map tiles will be rendered in the browser?

    That makes sense regarding issues with huge numbers of files. But doesn't sound like it will be a big problem for me, I am only using zoom levels 13-16 which appears to generate about 1000 tiles for a 24k USGS quad. So I should be fine with the amount of coverage I'm contemplating.

    Thanks again guys, I'm learning as I go. :)
  • global_mapper
    global_mapper Administrator
    edited January 2013
    There is a PDF at http://www.microimages.com/documentation/TechGuides/76googleMapsStruc.pdf that might explain it better. Basically the Google Maps tiles are just image tiles in a predefined projection, zoom level, and row/column naming scheme, so you don't need a full server like WMS to manage them. Any software (like Global Mapper or Google Maps in the web browser) that knows how the tiles are named can quickly access them. Since the names are predefined, you can update just part of them without affecting the others.

    The 'fill to tile bounds' means that even though you specify some bounds for an export (i.e. the quad boundary), the Google Maps export will generate all of the tiles needed to cover that bounding box at the zoom levels that you specify. If you went all the way to the top one of those tiles would cover the entire world! If you check 'fill to bounds' then the parts of the tile outside of your specified export bounds will be filled with loaded data rather than the background color or transparent pixels (for PNG). If you don't check that you will just get the background, so you will obliterate whatever was there when you update the existing tiles (which have the same filename and folder structure).

    You can add new Google Maps tiled sources on the File->Download Online Imagery/Data dialog in Global Mapper and provide the appropriate URL for the map tiles so Global Mapper can get at them. These can even be local files if you setup the URL for a local file source rather than something out on a server.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • Boyd
    Boyd Global Mapper User Trusted User
    edited January 2013
    Thanks again - I didn't realize I could use a local file path to fetch tiles, that is pretty cool! And the PDF should be useful as well. I appreciate your taking the time to explain all of this!
  • Boyd
    Boyd Global Mapper User Trusted User

    Funny, here I am nearly ten years later and stumbled on this old thread while searching for something else. 😀 I just had to chuckle about my "big set of 20,000 map tiles", as I just now published a new map with about 22 million tiles!

    Anyway, the best solution for my problem of "a blank white border around the edges of the new tile that covers up adjacent tiles" is a new(er) Globalmapper web export option to "only export fully-covered tiles". The problems I had back then were the result of a map coverage area that did not align with the tile boundaries, which causes partially covered tiles and blank areas.

    This export option suppresses those tiles, however it is necessary to create overlapping sections of the larger map to make it work properly. My workflow now is to export a lower resolution tileset of the entire map, such as levels 4 through 15 in one export. Then I have separate overlapping sections of the higher resolution tilesets for level 16 and 17. I export these to a temporary directory, zip them, then upload to my server and unzip, adding new tiles to the existing main tileset.

    I stopped using Google's API, it was too restrictive and I wanted to create my own user interface. So I'm an open source API called maptalks. I built my own web app from scratch that works on all platforms and has lots of features. Here's my map of the US Mid Atlantic region made from 1-meter LIDAR DEM, rendered with Globalmapper and exported as .png tiles up to level 17. This is Washington, DC with a screenshot from the mobile version of the web app.

    boydsmaps.com/#14.00/38.907254/-77.036540/midatlidar/0.00/0.00


    That map has about 11 million tiles which is about 350gb on the server. But I also have a three-dimensional version of the same map which consists of another 11 million tiles of elevation data for a total of 22 million tiles and 615 gb. 32-bit floating point uncompressed TIFF DEM files were converted to Mapbox RGB DEM with their rasterio command-line program. I then exported this as .png map tiles with Globalmapper. Here's an example of the 3d map showing Danbury, CT and a screenshot from my desktop web app.

    boydsmaps.com/#16.00/41.393639/-73.451397/mbx3dmidatl/137.40/74.80


    I have another pretty large 3d map that covers most of North America, created from ASTER GDEM data, but only at level 12 resolution. I used landcover data from LANDSAT and the ESA with my own pseudo-natural pallete for sort of a "painted" effect. It is "only" about 4 million tiles / 80 gb. For fun, I created an atmosphere with the Mapbox API. Here's a view of Mount Washington, NH

    boydsmaps.com/#12.72/44.035397/-71.681892/mbx3dnalc/48.60/76.40


    My web app is free with no advertising and no registration. Anyway, that's how I spent the last ten years - how about everyone else? 🤣 I'm still using Globalmapper and still managing large sets of tiles!

  • hennie
    hennie Global Mapper User Trusted User

    Love the web app!

    Well done 👍️

  • Boyd
    Boyd Global Mapper User Trusted User

    Thanks! 😀