Global Mapper v25.0

GM16 Hanging on LOAD_PROJECTION

dcoggin
dcoggin Global Mapper UserTrusted User
edited October 2014 in Technical Support
Hello Mike,

Just tried opening a workspace that works fine with 15.2. When opening, GM16 (64 bit) hung up on the LOAD PROJECTION command. I let it run about 3-5 minutes before cancelling. I'm not sure if there's a limitation on the size of the project with the beta demos. You're welcome to the workspace if you need it, but there are lots of files that go with it.

Thanks,
David

Comments

  • bmg_bob
    bmg_bob Global Mapper Programmer
    edited September 2014
    Hi David,

    I think it would be useful for us to have the workspace file. You can attach it here as a .zip file, or send it to geohelp@bluemarblegeo.com with a reference to the subject line of your post as the subject of the e-mail. Also, including a link back to this thread would be helpful.

    Cheers,

    Bob
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Hi Bob,

    The problem seems related to loading large map catalogs. In the problematic workspaces there are large MrSid map catalogs. If I remove those from the workspace, the workspace loads fine. GM16 loads the map catalog by itself just fine -- very quickly. If I draw a line before loading or load a shapefile and then load the map catalog, things slow way down. I am uploading a sample shapefile and map catalog to the ftp link you sent earlier referencing this thread. When I load the shapefile, then load the map catalog in GM15.2, it takes 12 seconds. The same operation in GM16 takes just over 2 minutes.

    The problem I was having with the workspaces is not a slow load but just flat hanging up -- it never loads, or I've never waited long enough. I'm hoping this is the same problem.

    The map catalog I'm uploading is around 12 GB. Thanks for looking into it. Also, I'm seeing the same behavior with the new version 16 SDK, so if you fix it I'd appreciate a new SDK build also.

    David
  • bmg_bob
    bmg_bob Global Mapper Programmer
    edited October 2014
    Hi David,

    I got your data. Thank you for uploading it. I am using Global Mapper 16.0 (b101614) 64-bit (this is a recent daily build).

    Using the UI, I loaded the shapefile, then the map catalog. The catalog loaded in roughly 15 seconds. I saved the data in a workspace, unloaded all data, then loaded the new workspace. The workspace load took about 20 seconds. I loaded the same workspace in 15.2.9, and got a very similar time. In both cases, after the catalog is loaded, I only see bounding rectangles for the SID file tiles. The actual tile images don't get displayed until I zoom in quite a bit.

    In the Overlay Control Center, I selected the catalog and clicked Options... On the resulting Modify Map Catalog, under When to Display Maps, my setting is "Map Box is % of Display Size", between 10% and 0%. Are you using something different? I wonder if Global Mapper is trying to load all of the SID files for some reason, and that is slowing down your load.

    Cheers,

    Bob
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Hi Bob,

    Thanks for looking at it. I tried creating a new map catalog from the same MrSid aerials and am still seeing the same thing -- almost 2 minutes for GM16 to load; less than 10 seconds for GM15.2 to load. I am using the 101414, 64-bit build. I am using the same map catalog view settings as you. I'll try a newer build if one's available.

    David
  • bmg_bob
    bmg_bob Global Mapper Programmer
    edited October 2014
    Hi David,

    This is perplexing, to say the least. We did make parts of the map catalog load process multithreaded in GM 16, so you could try turning that off. You can do that by creating a DWORD key named 'MaxThreadCount' in HKEY_CURRENT_USER\Software\Global Mapper. Give it a value of 1. This should turn off most threading. If this gets rid of the LOAD_PROJECTION hang you mentioned, then that at least points to where the problem is (although it is not a very good solution overall.)

    Cheers,

    Bob
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Bob,

    Hi Bob,

    Well that did it. With the DWORD set to 1, GM16 loaded the map catalog after the shapefile in about 5 seconds. I also tried MaxThreadCount 4, and it took 30 seconds. I switched back to 0 and it's still about 2 minutes. As I said earlier, if I don't load the shapefile first (load the map catalog as the first item), it loads almost instantaneously. Watching the performance tab of Windows Task Manager confirms that several cores (saw 4) are working during the load when the DWORD is set to 0 or 4. There are 8 "cores" shown -- I have an Intel i7. Hope this gives you some ideas.

    David
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Bob,

    Also, setting the DWORD to 1 allows GM16 to load the workspace mentioned in the original post.

    David
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Bob,

    In case it helps -- I just tested this on my laptop with a slower i7 and it takes 48 seconds to load with the normal "0" value for the DWORD and about 8 seconds with the DWORD=1.

    David
  • bmg_bob
    bmg_bob Global Mapper Programmer
    edited October 2014
    Hi David,

    Given the difference in load time based on the order in which the data is loaded, it sounds like Global Mapper is running into a bottleneck or deadlock when it has to reproject the map catalog data. When you load the shapefile first, Global Mapper has to reproject the map catalog data. When you load the map catalog first, Global Mapper has to reproject the shapefile data, which is smaller than the map catalog.

    Something similar could be happening in the workspace mentioned in the original post. The LOAD_PROJECTION command is using a projection based on the NAD83_FLORIDA_HPGN datum, and the Collier_2012 map catalog uses datum NAD83. That workspace also uses a MonroeEverglades_2012 map catalog. Though I don't know the projection used for the Monroe map catalog, it wouldn't be terribly surprising if it were the same as the Collier map catalog.

    It looks like we have some more investigating to do.

    Cheers,

    Bob
  • bmg_mike
    bmg_mike Global Mapper Guru Moderator, Trusted User
    edited October 2014
    David/Bob,

    I've finally got the data to download and will investigate. It loads right up for me even when reprojected, but it should certainly never be slower when multiple threads are allowed. I will debug in and see what might be happening.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • bmg_mike
    bmg_mike Global Mapper Guru Moderator, Trusted User
    edited October 2014
    David,

    I haven't been able to reproduce the slowness either or find anything in debug. The main activity is calculating the new projected bounding box for each map in the map catalog. For the size of those maps that calculation ends up doing about 256 reproject operations per map. It's really not that many.

    I did add a progress dialog when the reprojection is happening if it takes more than 2 seconds, so you can see progress of the bounding box calculation for each map to see if it is hanging on a particular one. I also now require a much larger number of reprojections to be needed before threading it as it appears as though the thread overhead was taking longer than the actual calculations, so it was slower to break up the operation. This is much smarter now.

    I have placed new builds with the latest changes at the links below:

    32-bit: http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper16.zip
    64-bit: http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper16_64bit.zip

    Simply download the .zip file for the build that you are using and extract over your existing installation files. The likely locations to extract to are as follows:

    32-bit: C:\Program Files (x86)\GlobalMapper16
    64-bit: C:\Program Files\GlobalMapper16_64bit

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Mike,

    Thanks – the new build works fine. The reprojection progress bar advanced smoothly during the load. It’s a mystery why I saw the problem on two different machines and neither you nor Bob saw it. I’m just not sure what will happen if I have a map catalog that hits the number for threading.

    Could I please get a new build for the SDK with the same changes? I was seeing the problem there also.

    Thanks very much,
    David
  • bmg_mike
    bmg_mike Global Mapper Guru Moderator, Trusted User
    edited October 2014
    David,

    I have placed a new SDK build with the latest changes at http://www.globalmapper.com/GlobalMapperSDK_v16_latest_beta.zip . This should use the same projection updates. I wonder if maybe it was just really slow on your machine for some reason and without the progress bar it looked like a lockup. If there was a timing thing causing a lockup I wouldn't expect you to consistently hit it.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited October 2014
    Mike,

    Thanks for the new build. I finally determined that it was not a lockup for the workspace mentioned in the original post. However, it did take over 10 minutes to load. I saw the same results on two different machines. It was consistent for the many times I attempted. When I changed the DWORD to one thread for the prior build it loaded normally just as it does with the new build you provided. I assume it's not hitting your new, increased threading criteria. If I have a map catalog that does hit the threading criteria, I expect to see the slowdown again. I haven't used GM16 much due to the problems, but I'll be interested to see how it handles other threaded tasks on my machines.

    Thanks very much for investigating and providing a new build.

    David
  • bmg_mike
    bmg_mike Global Mapper Guru Moderator, Trusted User
    edited October 2014
    David,

    That is good news. The issues isn't that the threading is always slower, but that it was being split in to threads much too soon before and the extra overhead of creating the thread was more time than the 16 or so reprojections that were ending up being done in each thread. Now each thread will have at least 256 reprojections to do. This is where the overhead of the thread creation is overcome by doing the calculation in multiple threads at once. So you still get a big benefit on very large coverage reprojections where it may need over a million reprojections to develop the reprojection mesh.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation