Global Mapper v25.0

Strange Download Rate Behavior When Transparent/Black/Empty Pixels Exist.

When a multi-threaded download is proceeding and contains areas/tiles with invisible pixels i.e. if downloading imagery to fit the shape of a county, the tiles that have even a small percentage of invisible/black/transparent pixels, ie, those that are partially cropped by the shape - the download time for that particular tile takes a long time from 15 minutes to up to 6 hours. Tiles that are downloading at the same time as part of the multi-threaded download that contain no invisible pixels and are a complete square of data always complete in around 6 to 10 minutes. It seems that the more black/empty pixels in the tile, the longer it takes. That doesn't make sense since the amount of data being downloaded is much less for those tiles and I am not sure that there is that much processing overhead to cause a 1 Mb partial tile to take 6 hours while a 200 Mb full square tile right next to it can take just 6 to 10 minutes. I see this behavior all of the time. I have solved the problem by simply creating a "rectangle" shape with all pixels filled with data and by doing that ALL tiles come down at the same speed and time - 6 to 10 minutes.

What also occurs in this situation of the partially cropped tiles that memory use goes through the roof - I have seen it as high as 7.8 Gb at which point a lot of instability starts occurring with the program. When I am downloading the "rectangular" shape, memory use rarely exceeds 300 Mb.

So there is something not right there that is causing the extreme memory use and much slower download times for small partially empty tiles. Would be great if everything no matter how the shape comes down at the same speed.

FYI I was downloading a country with the shape of the border (not too intricate) last night and it took 12 hours for just 40, 10K x 10K tiles. I aborted it and created my rectangle coverage meaning there was also a lot of unneeded imagery outside of that country and the same 40 tiles were downloaded in 90 minutes. I can reproduce the same behavior every time.

Not sure if you could call this a bug or an undesirable behavior.

Answers

  • tjhb
    tjhb Global Mapper User Trusted User
    Can you also say what you are downloading and where from?
  • bmg_bob
    bmg_bob Global Mapper Programmer
    Answer ✓
    Hello,

    It sounds like you are using the "Download Within Currently Selected Polygon(s)" option on your download.  Is this correct?

    I expect that the performance problem is related to the cropping -- any tile that intersects the bounding rectangle of your data has to be downloaded in full, then cropped.  When you download based on a rectangle, no cropping is required, which is why it is much faster.  We have an open bug related to multi-threaded exports where cropping is performed.  I suspect that this may ultimately be the same issue.  I have added item #18481 to our task list so we can look at this.

    In the meantime, you can work around this using the following steps:
    1. Download the full rectangle data
    2. Export it to a local raster file data set
    3. Import the raster and crop it as needed
    Cheers,

    Bob


  • Yes, I was downloading NAIP imagery and World Imagery. And yes in a way you were correct with your assumptions that I had download within currently selected polygon. You will see that from previous posts I have had stability issues and this seemed to be related to high memory use issues when downloading via this method. So the difference here now is that I was cropping to the polygon when displaying the imagery but then when I export I turned off all other layers and only exported all loaded data which was the image but with the shape of the polygon. So I guess it is the same end result and that is exporting data that is or was cropped to a polygon.

    But yes I think definitely a bug there since the extra time is WAY too disproportionate compared to the fact that it has to do some cropping. What I do notice is that it ultimately is the less image in the export tile the longer that tile takes. It's more a case of this than number of polygon points.

    Yes I did also work out that the best performance and stability and least memory use happens when you export a plain old rectangle. :-) So I am doing as you suggested.

    Thanks.

  • Mykle
    Mykle Global Mapper User Trusted User
    Global Mapper downloads data at the resolution appropriate for screen display.  So when you export, Global Mapper will probably RE-DOWNLOAD the data at the maximum available resolution unless you specify otherwise. 

    Recommend exporting to the desired resolution, without any cropping, other than to specified bounds whether to screen extents or specific coordinates.  Some people export downloaded data without specifying extents and wonder why the process does not complete (downloading the whole world).   

    Then, load your local exported file and any vector cropping layers, and export (again) from your local data file with cropping as desired.  Finally, reload your cropped export file and continue. 

    The lesson here is, whenever possible, export downloaded data as soon as possible to a local file, whether imagery or elevation.  Then work with the local file(s) and you won't have any internet and/or server delays, and you will be able to adjust your zoom level and/or extents without internet delays.  Many users have difficulty understanding what is going on here, but once you puzzle it out it is simple.

  • Yes thanks I did work out that this is the best way to use online imagery. Download it simply as a rectangular selection and then doing all of my vector work with local data placed in a map catalog. Generating rasters is much faster too when dealing with local data.
  • I can confirm that this issue seems to have been corrected and the problem does no longer exist.