Determine whether a bounds is covered with terrains

James Cox

Hi again,

I hope this isn't a dumb question. I don't really know much about GeoTIFFs/Global Mapper (as you might have guessed. )

My app takes in an extent from the user, then loads GeoTIFFs from a folder that overlap the extent. It layers them according to a simple number on each GeoTIFF, then exports the result based on the user-supplied extent and saves that cropped version to a new file. Works great.

I want to speed up the app by not loading any GeoTIFF which is already "covered" by other data. I notice that the GeoTIFF format has a mask. Is there any way to use this information in the GM SDK to discard those layers which are completely covered by layers above?

Thanks,
James

global_mapper

James,

You wouldn't really be able to use the mask for that. If you are using Global Mapper or the Global Mapper SDK to do this then the load time for the GeoTIFF is almost certainly almost nothing anyway as GeoTIFF files load very fast. They aren't read into memory but instead pulled from disk on-demand, so if you never have to access a lower GeoTIFF but just load it, only the header metadata will be loaded.

You could also create a map catalog of your GeoTIFF files, then just load that. The map catalog will automatically handle loading and unloaded layers as needed for display and export.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

James Cox

Hi Mike,

Thanks very much for the reply. I understand that loading the files into the GM SDK is very fast, that matches up with my testing.
The problem is that the export slows down the more layers that are loaded. That is why I wanted to try and avoid loading them, or possibly unload them myself manually, before doing any calls to GM_ExportElevation().

global_mapper

This is likely due to how caching of GeoTIFF files works as if you have lots of them loaded memory-mapped file access can no longer be used and slower access has to be done. If you use a map catalog and export from that then your issue will likely be solved as the map catalog will make sure only a few GeoTIFF files are loaded at a time. What format are you exporting to?

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

James Cox

Hi Mike,

I'm typically layering a bunch of high resolution elevations on top of some lower res background tiles, and then exporting the result to a single elevation GeoTIFF.

I'll describe the test I've done:
-test A is a very basic case, as if the system was brand new. There are 7 small (25 km^2) high res (1 m/px) GeoTIFFs, 220MB total on disk. There is one large (18180 km^2) low res (100 m/px) background tile that is 7MB.
I load those 8 GeoTIFFs into GM. I export a section of this based on the user input. It takes 4 seconds.

-test B simulates what the image folder will be like after a while, as newer data gets produced. Instead of a single background tile there are now 11. Technically of those 11 only 1 is required but of course GM doesn't know that.
I load those 18 GeoTIFFs into GM as before, and export a section of the map as before. It takes 10 seconds now.

In actuality I expect the performance degradation to be even worse, because there won't just be newer versions of the low res background tiles, but of course newer versions of the high res data as well. It would be great to avoid ballooning RAM requirements by loading multiple sets of the high res data.

Does it sound like the map catalog will fix this issue? If so I have no problem going in that direction.

Thanks again,
James

James Cox

I'm just looking at the Map Catalog functions in the SDK now, and it all looks pretty straightforward except how do I control the ordering of the files in the catalog?

Thanks,
James

global_mapper

James,

I think the map catalog could help if it was already built up so everything was in a single map catalog, with the lowest resolution stuff first, then the higher resolution stuff on top. Then when the export went along for each sample it would ask the map catalog for the elevation at each location. It would only have to load and query the lower resolution background maps if there were samples where something didn't exist in the top-most map layers.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

global_mapper

James,

The order is the order that the layers were added. There are some functions in the GM user interface to sort maps in a map catalog based on the resolution. This could potentially be added to the SDK as well if you weren't building these dynamically so you could control the order easily.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

James Cox

Thanks for everything so far. I have written some code to take the list of elevations and add them in the correct order to a map catalog. The map catalog file is saved on disk successfully and then I call into GM_LoadLayer() with the full path of the catalog file on disk.

Unfortunately that then gives me a GM_Error_LoadError.

I can upload the .cat file if that will help but it appears to be well formed XML with all 18 of the elevations described.

James Cox

Scratch that! Changing the extension to .gmc fixed it.

James Cox

OK, I've just repeated tests A and B and unfortunately I don't think the map catalog is optimising anything.

Unfortunately the same effect is observed - as more layers are added the export becomes slower.

global_mapper

Are there areas where the lower resolution layers are being needed? Is it the export call itself that is taking the time or are the loads a factor?

Is it possible for you to convert the GeoTIFF files to Global Mapper Grid files and use those instead? Global Mapper Grid files have the benefit of being much smaller on disk and even faster to load than GeoTIFF files. They also have internal pyramid layers and are tiled which can make some access methods faster.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

James Cox

The lower res background tiles are needed to fill the rest of the area provided by the user if it is not completed covered by the high res elevations.

In my tests it is the call to GM_ExportElevation that increases roughly linearly with the number of layers. Creating the catalog or loading multiple files is almost instantaneous in comparison.

My aim with all this is to avoid this increase no matter how many layers may be involved (within reason). Ideally the user should not have to clean up the repository of elevations at all or at least not very often.

I can definitely try the conversion idea. I'll convert a bunch of data into GMGs and report back any difference.

global_mapper

Yes definitely see if the GMG format helps (if nothing else they will take less disk space). What format are you exporting to? Some of the exports are smart enough to pare down the list of layers during the export for each row to reduce the search space at each pixel when you have a lot of layers that cover different areas of the export bounds. If you are exporting to a format that doesn't yet have those smarts that could be part of the problem as lots of bounding box checks end up being done for each export pixel just to skip layers which couldn't possibly have a value for a particular pixel (but might for other parts of the export).

Basically it looks through the list of layers at each location in reverse (i.e. top down in draw order) for the first one with a valid sample, then uses it. So if you pass in 100 layers, many of which are small and don't cover much of the export, each sample location might have to discard a bunch of those via bounding box check before finding some to actually check at a location.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com

James Cox

We're exporting to GeoTIFF and now I think this is the root cause of the whole thing.

I changed the export to be GMG instead and now the export takes roughly the same time whether there are 8 layers or 18. The speed either way is even faster than before as well ( 2 seconds rather than 4.)

I'll have to see if we can use the GMG export because it definitely seems like the way to go.
Even if we end up having to use GeoTIFF, a GMG export followed by a GMG -> GeoTIFF conversion should suffice.

Thanks again,
James

global_mapper

James,

Oh my goodness, I checked the GeoTIFF elevation export and found that while it had the filter on the overlay list for each row, it was repeating the same filter for each pixel in the row! This completely destroyed the performance enhancement. I've fixed that now so the GeoTIFF elevation export should stay fast regardless of how many input layers. I have placed a new SDK build with this change at http://www.globalmapper.com/GlobalMapperSDK_latest_beta.zip for you to try.

Let me know if I can be of further assistance.

Thanks,

Mike
Global Mapper Guru
gmsupport@bluemarblegeo.com
http://www.globalmapper.com