Panning or zooming TIFF images with overviews (pyramids) stalls GM

tjhbtjhb Global Mapper UserPosts: 454Trusted User
edited December 2012 in Bug Report
Mike,

I'm having trouble viewing TIFF images with embedded overviews in Global Mapper 14.0.3 (64-bit, latest installer build, 25 October 2012 13:40:43).

The TIFFs are ordinary TIFFs, each tile 16384 pixels square, originally written by Photoshop. Overviews were added using GDAL (gdaladdo), at 1/2, 1/4, 1/8, 1/16, 1/32 and 1/64 resolution, which brings us down to 256 pixels square for the smallest overview. Worldfiles were written pro forma from GM and added to the source folder. The projection is New Zealand Transverse Mercator. There is no overlap between tiles.

Viewing is generally *lightning* fast (comparable to ECW if not faster). Brilliant.

However, if many tiles are loaded, then under some circumstances a panning or zooming operation appears to stall Global Mapper.

For example (doing this now): I load a set of 115 tiles, and zoom to their native resolution, which is 4m per pixel.

I seem to be able to pan around at this zoom level indefinitely, no problems.

But if I now hit the Page Up key to zoom out, I get a wait cursor, with GM showing 3 boxes on the progress bar (on a 1600 x 900 pixel monitor, in case it varies). This lasts for some minutes (so far, as I write, about 10 minutes, still at 3 boxes progress).

I think GM may eventually come back to life. It did after one test, though perhaps not this one (forcibly closed; Windows says App Hang B1).

With only 64 of the tiles open, it is more difficult to make GM stall. Now I have stalled it by zooming in with the scroll wheel, after panning and zooming for a minute or so. Again 3 boxes on the progress bar.

With only 12 of the tiles open, I can't make the display stall by any amount of panning and zooming.

Loading all 115 tiles *without* overviews (the images as written by Photoshop), I can't make GM stall by panning and zooming (though of course this is a much slower operation).

It would be difficult to supply the full dataset showing the issue because of size (75GB). Unless you visit us in New Zealand!

I haven't yet tested with a fake dataset. E.g. 115 tiles at the same resolution but all blank, which it would be easy to zip up and send.

If there's anything else I can test, or more details you need, please let me know.

Tim Baigent

System details:
Windows 7 64-bit SP1
16 GB RAM
Core i7-2600
NVIDIA GTX 560 Ti (drivers 306.97)
OS, applications and Windows swap file on first SSD
Source images and TEMP folder both on second SSD

GM options:
All source images: No resampling (nearest neighbour)
Automatically anti-alias raster layers when loaded: unchecked
Never automatically contrast adjust images on load: checked
Disable automatic interpolation of zoomed-out display: unchecked

Comments

  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    The last line of my message above made me think (slightly) harder.

    If I change that setting to
    Disable automatic interpolation of zoomed-out display: checked
    then I don't seem to be able to make Global Mapper stall at all, with 115 tiles loaded (images with overviews).

    Could there be a conflict between code applying resampling at resolutions lower than native resolution (if the option is enabled), and code utilizing image overviews (if available)?

    Tim
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    Unfortunately I'm not able to go to New Zealand this season as I had hoped.

    Are you loading your files directly or using a map catalog? TIFF files can suffer a large degradation of performance when loading large numbers of them directly, but if you put them in a map catalog (File->Create New Map Catalog) and work from that only a small number of them will actually be loaded at any one time and you can see extremely fast performance regardless of how many you have loaded.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    That's a pity, but we will keep a warm welcome for you whenever you can visit. New Zealand isn't going anywhere. (In geological time the western side of the country, on the Australian plate, is grinding up past and over the eastern side, on the Pacific plate, but depending on your exact plans we can ignore that.)

    And thanks. Yes, I am currently loading the tiles directly. I will try a catalog as well.

    However, I don't think that could explain the marked difference between tiles with overviews (which can stall GM) and tiles without overviews (which do not).

    Nor the equally marked difference, given images with overviews, between using automatic interpolation of zoomed-out display, and disabling it.

    This is not a degradation of performance but an apparent hang.

    Perhaps there is a deadlock or race condition between the code that implements resampling for zoomed-out display, and the code utilizing image overviews for display where they are available; or between the code that updates the display and the code harvesting mouse and keyboard events.

    I'm not a proper programmer and am totalling guessing (for which I half-apologize) but perhaps testing on a debug build might show something useful?

    For now we can ensure that the automatic interpolation of zoomed-out display is disabled, whenever we are using images with embedded overviews. This seems to be safe. (And to repeat, we can also try a catalog.)

    Tim
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    I just tried using a map catalog on this set of images with embedded overviews, which was very interesting.

    Judging by display and navigation speed, and also image quality, the map catalog interface does not seem to utilize image overviews at all. (I don't think it implements interpolation of zoomed-out display either? Not sure.)

    So display is quite slow, compared to navigating tiles (with embedded overviews) loaded directly, but there is no hanging problem.

    Tim
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    Can you provide one (or several if possible) of the images with overviews? You can FTP them to ftp.globalmapper.com using a username of 'upload@globalmapper.com' and a password of 'upload' and I can pull them down from there.

    If I can reproduce the stall in a debugger I can see exactly what is happening. I can also try and determine why the overview images wouldn't be used from a map catalog. I wouldn't expect that at all.

    The interpolation setting also shouldn't make a difference except for a slightly slower render but with slightly better quality when zoomed out a bit. It is the same as having the bilinear resampling method turned on for the layer at some zoom levels.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    Thanks very much for being prepared to have a closer look at this.

    I tried to get one of our real tiles packaged up but they don't compress at all well (about 500 MB each fully compressed). Anyway I really needed to get you something close to a full tile set, since the apparent problem doesn't occur with a small number of tiles.

    So I hope this is a better idea.

    I've made a single RGB test image (with no aesthetic goodness in it whatseover; it's just a cycle of R, G, B and white pixels) matched to the resolution and extent of one of the original images (4 mpx, 16384 x 16384).

    Overviews have been added using gdaladdo, as before.

    I have also included the .prj and .tfw metadata files for all 115 tiles in the dataset.

    This is all under 4 MB compressed.

    Using NTFS hardlinks, the test data can be made to appear as a set of 115 tiles (all identical), as a stand-in for the original dataset.

    You can make the hardlinks by opening a Windows command prompt with administrative privileges, and in the target directory executing:
    for %I in (*.tfw) do fsutil hardlink create %~nI.tif Test.tif

    Loading this test dataset in Global Mapper does show the difference in behaviour, according to whether automatic interpolation is enabled or disabled for zoomed-out display. GM will occasionally stall during panning and zooming if it is enabled—though not indefinitely, as for the real dataset, and further along in the progress bar—and it will never stall if the option is disabled.

    (The quickest way to see the problem is probably to zoom to 1:1, then pan around, zoom out and in until you see an obvious delay, depending on the interpolation option.)

    I hope this will be enough for you to test. If not I can try doing the same with one real texture tile (bandwidth and patience permitting) to cause an (apparently) indefinite lockup.

    Thanks again Mike,

    Tim

    GM test data.zip
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    Thanks for the file, I was able to track down a few separate issues and get them fixed. There weren't really any lockups per-se, but some slower performance issues. I also discovered why the overview layers weren't being used from map catalogs, so if you rebuild your map catalog with a new build then you should get the full speed and use of overviews. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Thanks Mike. It's not working so well yet though.

    (1) With the new build, I don't think I can see a performance improvement when viewing the test dataset. With real data we still see apparent hangs (long enough to give up on). We've tested on a couple of different machines.

    In fact I think the new build is not nearly as fast as the October 25 installer build. And disabling automatic interpolation no longer seems to help.

    (2) We are now also seeing image corruption (something new). The exact pattern of corruption varies (for example, on panning). Examples below.

    There is no corruption at 1:1, only at lower resolutions (zoomed out).

    Example 1.pngExample 2.pngExample 3.pngExample 4.png

    I'll provide some real data for you to test with, if that's alright.

    Tim
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    That last image was followed by
    Problem signature:
    Problem Event Name: APPCRASH
    Application Name: global_mapper14.exe
    Application Version: 14.10.10.0
    Application Timestamp: 50cf7d05
    Fault Module Name: StackHash_87ab
    Fault Module Version: 6.1.7601.17725
    Fault Module Timestamp: 4ec4aa8e
    Exception Code: c0000374
    Exception Offset: 00000000000c40f2
    OS Version: 6.1.7601.2.1.0.256.48
    Locale ID: 5129
    Additional Information 1: 87ab
    Additional Information 2: 87ab4298633f5f61abe759e3c35aea54
    Additional Information 3: 6635
    Additional Information 4: 66355d87398cb87e9bb07cf25d180db1
    ...

    when I tried to Unload All.

    And from another instance of GM, when I tried to close the application:
    Problem signature:
    Problem Event Name: APPCRASH
    Application Name: global_mapper14.exe
    Application Version: 14.10.10.0
    Application Timestamp: 50cf7d05
    Fault Module Name: StackHash_afa8
    Fault Module Version: 6.1.7601.17725
    Fault Module Timestamp: 4ec4aa8e
    Exception Code: c0000374
    Exception Offset: 00000000000c40f2
    OS Version: 6.1.7601.2.1.0.256.48
    Locale ID: 5129
    Additional Information 1: afa8
    Additional Information 2: afa826699db1e61b1d696983b816d214
    Additional Information 3: 8fb5
    Additional Information 4: 8fb58cf14a218eb3a5d81b3985abf7fe
    ...
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    Well I messed that right up. My new supposedly faster cache code had a bug that defeated the cache and also caused the crash on unload. I just missed it with the sample data since it is harder to obviously corrupt that data. I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Try creating the map catalog of the files after getting the new build and see if the problems are hopefully fixed.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    [Sorry: I posted this without refreshing the thread, and missed your reply above. I'll read it and test again. Ignore all this for now!]

    Mike,

    I have uploaded a tile of real data* to your ftp server: GM test data 3.zip. (Please ignore GM test data 2.zip--explained below.)

    It's packaged as before, with metadata for all tiles, so you can make it appear as 115 separate copies by using hardlinks. (It's tile QE, in case you want to load it alone with matching metadata.)

    *Actually not quite real data. I reduced the colour information by converting to indexed colour then back to RGB. This takes the compressed package down to a manageable size (410 MB).

    (In GM test data 2.zip, the image was left in indexed colour mode, which compressed even better, but unfortunately or fortunately the stalling problem did not appear.)

    Even this third set isn't ideal, since it doesn't provoke the seemingly indefinite stalling we are seeing with a full set of original data. (Damn it!)

    But you can usually provoke long delays by doing this:
    Zoom to full extent
    Zoom to spacing (pixel size): 4 (native resolution)
    Pan left or right (say, a screenful)
    Page Up (to half resolution)
    If necessary, Page Down then Page Up (and repeat)

    If automatic interpolation is disabled, then the Page Up operation is instantaneous. If it is enabled, then it can take many seconds (though not always). I think (having tested this a few times now!) that the delay may occur only when the zoom level is changed to half the native resolution (by zooming in or out), or to something lower than native but higher than a quarter. With the original data, that same range is where seemingly indefinite delays can occur.

    I now think these statements hold for both the 25 October build and the new 17 December build, but the display corruption issue makes it harder to be sure, since the garbling on screen seems to have secondary effects (possibly via memory corruption?). This build always crashes on exit, after viewing a corrupted display.

    Hope you can see what's going on here Mike!

    Tim

    [See note above.]
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    No corruption! Very cool.

    But rendering still seems really slow in that first range, just between native resolution and half, iff automatic interpolation is enabled. To repeat: not every time, but often.

    I hope you can see it with the upload GM test data 3.zip. ("Expanded" into hardlinks.)

    Exciting to see CGAL libraries in there. Are you adding more GIS functions for 14.1? Like Voronoi? Ha--just answered my own question, by looking at the Analysis menu. Congratulations. Many people will be very happy!
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    More notes, a summary.

    (1) Map catalogs now use image overviews brilliantly. Perfect combination: intelligent rendering, intelligent data access, intelligent storage. Extreme speed.

    (2) With data loaded directly, I'm still getting (not hangs but) sometimes very long delays using tiles with overviews, panning or zooming at or about half-resolution, unless automatic interpolation is disabled. I've just re-tested with a full set of our original texture images, and it's the case.

    It's easy to disable automatic interpolation, and in layman's terms that's a fix.

    I have two strong feelings: (a) that this is something you'd want to resolve in due course, and (b) you have higher development priorities.

    How about putting this aside for now, and we send you a full set of our New Zealand textures on hard drive, as an example of what users create using your work. When you receive it you could try to find out what's causing the apparent delay, in specific circumstances, with complete data.

    Tim
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    The in-memory caching gets smaller and smaller as you load more TIFF files to avoid taking up too much memory. At some point it was reducing to just 2 cached tiles, so if you were resampling and right at a tile border where you might need 4 to resample, the tiles would keep getting re-read all along that seam. I have fixed this so now the cache size never goes below 4 tiles, which should eliminate this even when directly loading a huge number of TIFF files (each overview layer counts as a completely new file, so you have a lot more than just the number of TIFF files you load). I have placed a new build at http://www.globalmapper.com/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.globalmapper.com/global_mapper14_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Mike,

    You're going to hate me. Thanks very much for all this work, and the excellent explanation. I feel like a right PIA.

    The 18 December build hangs with the full dataset (apparently a real hang now, in the tens of minutes anyway), at two boxes on the progress bar, unless automatic resampling is disabled. Repeatably.

    I tried making screengrab videos of the process with and without the option enabled; not really very informative, and I haven't found a free tool that is stable.

    There is no hang using the data I uploaded as GM test data 3.zip.

    A "fake" dataset comprised of one original tile (not passed through indexed colour), hardlinked into a mosaic, does not hang either.

    Zooming out from native to half resolution is noticeably slow in these cases (significantly more in the second case than in the first), but it never seems to skive off into a hang.

    The possibility arises that there is something about one particular tile (or more than one) that is choking GM. Somehow malformed, or unusual. I will do some testing on different subsets of the whole dataset and let you know.

    Is there anything further I can do to help you help? I am pretty new to Visual Studio; I have 2010 Professional and 2008 Standard (so no Intellitrace). Would I be able to do something useful at this end, armed with a debug build and symbols? It may not be your policy to let these things out of the building, but if there's something I can do please let me know.

    Or is it a case of waiting until we can get you the whole dataset on hard drive?

    Really sorry about this Mike. All I can say is that we highly recommend you, to all our customers, far and wide!

    Tim
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    It would be quite an effort to get something that could provide useful information in a debug. I really need to have it reproducible on my end for this type of problem.

    Is the new hang when loading them all directly or using a map catalog to get at them?

    If you pull up Task Manager is Global Mapper hard at work on one of the cores?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    It would be quite an effort to get something that could provide useful information in a debug. I really need to have it reproducible on my end for this type of problem.
    Understood. Plus you might have to explain quite a lot to an amateur.
    Is the new hang when loading them all directly or using a map catalog to get at them?

    I thought: only when loaded directly.

    Actually: also in a map catalog, iff I set it up to always display all tiles (by setting a display threshold 4096 metres per pixel). In this case, a hang that looks exactly the same as for all tiles loaded directly.

    Also:
    Loading half the tiles directly -> no hang.
    The other half directly -> no hang.
    Just the two central tiles (the tiles that occupy the screen when you go from full extent to native resolution) -> no hang.
    Loading 87 of 115 tiles directly -> no hang.
    Loading 93 of 115 -> hang.
    If you pull up Task Manager is Global Mapper hard at work on one of the cores?
    Yes, saturated, with 7 threads running, and private working set steadily increasing at a rate of about 20KB every 2s. (Only about 150 MB total [corrected] as I write this; this is with 93 tiles loaded.)

    Aha! I'm glad for my slow editing. Not a hang. GM came back to life after 13 minutes. Repeating the procedure (returning to native resolution, then Page Up to zoom out to half) has started the stall/delay again, with the same thing happening in Task Manager, except this time I notice the number of threads varies between 5 and 7.

    I just started a separate instance with all 115 tiles loaded directly. Also apparently stalled/delayed. The number of threads was initially 17, then 10, and is now varying between 9 and 11.

    Tim
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    (The 93-tile instance above returned to life for a second time after 13 minutes. And the 115-tile instance also returned after 13 minutes.)
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Ok hopefully last one. I was able to duplicate an extreme slowdown when zoomed in by loading the earlier hard-link data multiple times. Basically it's the same issue as before that I fixed with the tile cache, but the most detailed layer actually uses a strip cache and that cache was supposed to automatically grow when it was experiencing poor performance, but that wasn't working when there was so much data loaded that only a single strip was loaded at a time for each file, which slowed the render way down at file boundaries. I've got that fixed now so the cache will still grow in those extreme conditions which should fix it. If you had a machine with limited memory this could cause problems, but really a map catalog fixes it all the time. I have a much better long-term solution, but deferred that for now.

    I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    Yahoo! Mike I am immensely grateful for your extraordinary persistence.

    This works perfectly. Not a moment's hesitation, anywhere.

    I wish you could see the faces of people the first time they see this amount of data loaded all at once... at once! Zooming in and out and panning as if the data were somehow lighter than air.

    I'll be showing this dataset to a couple of largish organizations in the next few days who don't currently use Global Mapper. One of many selling points. A total no brainer.

    The tile versus strip thing is interesting. Nice bit of detective work. I think Photoshop will only write strip format.

    Thanks many times over. I hope you have a delicious meal tonight. If not, never fear, I will have a delicious meal a bit later on, and a good glass of wine, on your behalf. I am considerate like that.

    Tim
  • tjhbtjhb Global Mapper User Posts: 454Trusted User
    edited December 2012
    I think Photoshop will only write strip format.

    FWIW I got authoritative confirmation on the Photoshop forum.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited December 2012
    Tim,

    That is great, I'm glad I was able to get this working for you in all cases!

    The strip format is definitely the most common of the organizations for TIFF, but tile isn't too rare. They each have their advantages and disadvantages, but with overviews the disadvantages of tiles go away and they can be much faster. In any case Global Mapper should work fine with both types.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-map
Sign In or Register to comment.