GeoTalks 2025 User Conference

Downloading WMS/WMTS/TMS

falqn
falqn Global Mapper UserTrusted User
edited October 2012 in Downloading Imagery
When I'm loading the data from WMS/WMTS service, which, for example, covers whole Europe in scale 1:10 000 000, but on a scale 1:500 000 covered is only one state (for the rest of area the data does not exist) I have a problem: GM is trying to download the data from the the area where there is no data and is asking huge amount of requests but answer is always "404 not found". It takes him a veeeeeeeery long time and downloading becomes impossible. What to do in this situation?

I think that GM is trying to download this missing tiles even if I crop the WMS/WMTS layer to selected polygon (when adding the WMS/WMTS source "current screen bounds" option is not an option because often the source is not a rectangle)

Example of this kind of service is French geoportal: http://gpp3-wxs.ign.fr/tyujsdxmzox31ituc2uw0qwl/geoportail/wmts and WMTS&STYLE=normal. For small scale it covers Europe and Africa, but for small scale it has data only for France and neighbourhood.
The same problem is with google satellite view, on seas they do not have hi-res photos, but GM is asking for them (answer is 404 not found) and downloading became very very slow.

Best regards
Wojtek from Poland

Comments

  • global_mapper
    global_mapper Administrator
    edited October 2012
    Wojtek,

    Is the problem due to the tiles being retried multiple times? I might be able to smarten that up a bit as it currently retries tiles that fail a couple of times to work better with flaky servers, but I think in the case of a 404 error I could probably just abort after the first failure. There will have to be at least one download attempt for the tile though as there isn't any way to know which tiles aren't there without trying.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • global_mapper
    global_mapper Administrator
    edited October 2012
    Wojtek,

    I tried this service and did go ahead and update the download to no longer retry (with the associated short pause) when 404 (and other 4xx errors) are encountered. This seems to dramatically speed things up when zoomed in. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • falqn
    falqn Global Mapper User Trusted User
    edited October 2012
    Mike,
    Hmm, I'm not sure that now is better. Downloading from this kind of sources is still almost impossible.
    Please look at these GM reguests - they are the same, GM is still asking 2 times about one tile... These requests were made by GM one after the other

    a) http://gpp3-wxs.ign.fr/tyujsdxmzox31ituc2uw0qwl/geoportail/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&FORMAT=image/jpeg&LAYER=CARTE_PYR-JPG_WLD_WM_20121005&TILEMATRIXSET=PM&TILEMATRIX=21&TILEROW=733186&TILECOL=1021041&STYLE=normal


    b) http://gpp3-wxs.ign.fr/tyujsdxmzox31ituc2uw0qwl/geoportail/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&FORMAT=image/jpeg&LAYER=CARTE_PYR-JPG_WLD_WM_20121005&TILEMATRIXSET=PM&TILEMATRIX=21&TILEROW=733186&TILECOL=1021041&STYLE=normal

    But I probably know what is the main problem: GM is searching tiles in next zoom levels. Data which I need is in 14 zoom level, I left GM at night to download data. GM works fine until it encounters an 404 error - then GM is trying to find tiles in next zoom levels. In the morning I found GM requesting 21 zoom level! This generate a realy hudge amount of unnecessary requests. Mayby you should dramatically reduce number of searched zoom in levels when 404 errors are encountered.
    I think good idea is to search in zoom out levels because in most of cases the data should be there.

    Best regards
    Wojtek from Poland
    GM v14.0.2
  • global_mapper
    global_mapper Administrator
    edited October 2012
    Wojtek,

    Ah indeed the export will continue to search for more detailed layers when it can't find anything in the closest zoom level. I have vastly improved this so when zoomed in at all it will check just a couple of higher zoom levels, then a few lower, then dig further higher, then even lower, until it finds something. This should change your export to look at most in 2 zoom levels further down before finding something further up. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • falqn
    falqn Global Mapper User Trusted User
    edited October 2012
    Mike,
    Mayby is a little bit better, but exporting is still impossible. As I wrote earlier, I need data in 14 zoom level (36 meters/piksel). Everything is fine when GM is downloading 14 level, when 404 error encounters the problem is starting. I see now that GM is digging in 15, 16, 17, 18, 19, 20, 21 level - there is no data in this levels (in 30 minutes GM make 10000 unnecessary reguests and 99% of time takes him to dig in 18, 19, 20 and 21 level). Digging in so high zoom levels does not make any sense, usually greater data cover is in less detailed levels, not in more detailed.
    Mayby GM should dig (after 404 encounters) ONLY 1 or maximum 2 higher levels and then should start finding tiles in lower levels: 13, 12, 11 - where data is usually really present. Or maybe extra option like "maximum dig level" should be present to limit the unnecessary requests.

    Best regards
    Wojtek from Poland
  • global_mapper
    global_mapper Administrator
    edited October 2012
    Wojtek,

    Ah I didn't realize you were trying to export from such low resolution. I had only adjusted the behavior when the closest zoom level was less than 10m in resolution, so nothing had changed for yours! I have changed this now to dig up and down when at or below 100m resolution. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    The behavior was designed for the large number of services where the data only exists when zoomed in but there aren't any overview layers. For example many sources will only have any valid data when zoomed in beyond say 1:50000 scale, so you have to dig down quite far to find data when zoomed out. Of course your source is just the opposite. With the new change it will still dig a little bit down, but it should just be 2 levels and not 10 so the difference should be only 6 or so extra tile checks rather than thousands!

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • falqn
    falqn Global Mapper User Trusted User
    edited October 2012
    Mike,
    New build is working fine. Thanks!
    Best regards
    Wojtek from Poland

Categories

In this Discussion