Global Mapper v26.0

Changed multithreaded behavior for GM_GetPathProfileLOSEx()?

2»

Comments

  • global_mapper
    global_mapper Administrator
    edited April 2013
    Is it the child thread that is crashing or the main one?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    The error I just described seems to be occuring in the main thread, in terms of throwing the invalid_arg exception.

    For what it's worth (and I'm trying to reproduce it to be sure), I just went to my debugger settings and turned on the option to break on just about any kind of exception that occurs. On one of the runs through this route, I got a debugger exception of trying to free 0xCCCCCCCC (uninitialized/invalid ptr) from my child thread. The call stack showed a few levels of GM dll calls, but I couldnt see where the call stack started so I don't fully trust it. However, when I told the debugger to 'pass the exception to the calling program' and continue, I got a GM pop-up I've never seen before saying 'Unknown error closing layer', followed by a normal crash.
  • global_mapper
    global_mapper Administrator
    edited April 2013
    Would you be able to use a Debug build of the SDK with a PDB file with the function names if I sent it? It is built in Visual C++ 2008 and would be the 32-bit version. If you can use that then you could see the function names in the call stack when it crashes which could help me tremendously.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Yeah that should work well enough...we are also using VS 2008 and building 32-bit.
  • global_mapper
    global_mapper Administrator
    edited April 2013
    I have placed a new SDK build at http://www.globalmapper.com/GlobalMapperSDK_v14_latest_beta_debug.zip that includes a bin_debug folder with the Debug libraries that you need as well as the GlobalMapperInterface.pdb. You should be able to use that and get the function names in the call stack.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Hi Mike,

    I ran with the debug SDK a few times to hopefully give you a picture of what is happening. The first two runs I had all exceptions turned on, then the third run I set them back to their default state.

    The first run I got a deque assert error, with the following call stack:
    2_Call_Stack_First_Run.PNG

    The second run created a very similar assert with a similar call stack:
    3_Call_Stack_Second_Run.PNG

    The 3rd run, with default exception handling was a bit different. First I got the following GM popup dialog:
    4_Popup_Third_Run.PNG

    Then I think it ran ok for another minute or so, followed by a slightly different kind of deque assert, and the following call stack:
    6_Call_Stack_Third_Run.PNG
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Hi Mike,

    I was suspicious of how quickly I'm seeing crashes with logging turned on, so I turned it back off and did another test run with your debug SDK. It ran much longer (over half an hour) before crashing, and the call stack looks different. It started with a series of the 'unknown error closing layer' messages one right after the other, followed by access violation with this call stack.

    I wonder if the error messages are actually triggered from the child thread that handles loading/unloading tiles? This error call stack is from my main thread:
  • global_mapper
    global_mapper Administrator
    edited April 2013
    Ah it seems that the new error log stuff itself wasn't thread safe! I've fixed that so hopefully now we can get back to tracking down the original problem! I've updated the SDK build at the same URL as before with the fix. Give this about 10 minutes to upload if you get this right away.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • global_mapper
    global_mapper Administrator
    edited April 2013
    When you get the call stack in the path profile thread can you also do a copy of the call stack in your child threads? I need to see where those are as well.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    global_mapper wrote: »
    When you get the call stack in the path profile thread can you also do a copy of the call stack in your child threads? I need to see where those are as well.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/


    Haha ya sorry, after I killed the run last time I realized I hadn't captured the call stack from the child thread. In this test app I have purposefully tried to keep just the main thread and the single child thread in order to simplify the situation.

    I left it running overnight (hadn't noticed your new logging build) and came back to a similar crash this morning. Here is the set of call stacks:

    Main Thread:
    Main_Thread.PNG

    Child Thread:
    child_thread.PNG

    I can download and give the new thread-safe logging build a try next if you'd like.
  • global_mapper
    global_mapper Administrator
    edited April 2013
    I took a look at those call stacks and found one possible place that if the timing was just wrong a problem could occur. I've hopefully fixed that and placed a new build at the same location with just the Debug updated. If it still has the crash can you provide the dumps again and also absolutely confirm that the layer being closed is not the one being passed into the GM_GetPathProfileLOS function? I can tell the layer handle being closed in the child thread, but can't see what's in the list passed in for the other thread.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Hi Mike,

    Next time I see it crash on a layer close I will verify that the layer points arent the same, although I had checked that in the past and it wasn't the case.

    I ran with the new debug build, and this time when I crashed I was loading a tile on the sub thread while doing my getpathprofile on the main thread. I think the crash was the main thread/GetPathProfile call. Here are the call stacks:

    Main Thread:
    main_thread.PNG

    Child Thread:
    child_thread.PNG

    That "SendEventToListener' call definitely seems to be appearing at the end of the stack quite often for what it's worth.
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Here is a subsequent run/crash where the child thread was closing a layer. I can verify that at the time of the crash I was passing different layer pointers into GM_CloseLayer and GM_GetPathProfileLOSEx.

    Here are the stacks again:

    Main Thread:
    main_thread.PNG

    Child Thread:
    child_thread.PNG
  • global_mapper
    global_mapper Administrator
    edited April 2013
    You are correct that the SendEventToListener function does seem to be the culprit. That is a central place for dispatching events to "listeners" and had some pretty complex thread protection mechanisms to handle one thread sending messages to all listeners while another thread was adding or removing new listeners. I worked through that and found a couple of potential issues and was able to simplify the code a bit while hopefully correcting the problems. It seems to work in my tests and walking through it in my mind, but since I don't have any thing that hits it like your sample application I can't be 100% sure it's ok.

    I have updated the .zip again with new builds for you to try. Both the Debug and Release libraries are updated, so if everything now works in Debug you can try Release.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Ok that makes sense. I've taken the latest debug build but I think I must have run into some basic 'gotcha'. I crash on the first attempt to load a tile with a null pointer access violation. I assume this is just some little thing that got missed after your code refactor. It appears its attempting to access a NULL std::list ptr when it throws the exception.

    Here is the call stack for the tile loading thread:
    child_thread.PNG

    The main thread isn't calling into GM at the time of this error.
  • global_mapper
    global_mapper Administrator
    edited April 2013
    Oh this is strange, I responded to this with a fix a couple of days ago but apparently my reply didn't post. I was actually waiting on you to provide new feedback! There is a new .zip at http://www.globalmapper.com/GlobalMapperSDK_v14_latest_beta_debug.zip which should get around this issue which was just a bad build.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • jkenneally
    jkenneally Global Mapper User Trusted User
    edited April 2013
    Just a quick update, but I ran my test app against the debug binaries all afternoon with no crash yesterday. I left the release build running overnight and it was still chugging along this morning.

    We are now in the process of migrating this version of the SDK into a new build of our product for a round of real world soak testing, but its looking good so far!
  • global_mapper
    global_mapper Administrator
    edited April 2013
    That's fantastic! I'll check the changes in so they will be in the next official SDK release. Thanks for your help tracking this down, it's probably been affecting random SDK users with multi-threaded applications for years, but just never consistently enough to track down!

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/

Categories

In this Discussion