Global Mapper v25.0

Slow FindNearestFeatures performance with large number of vector features

dcoggin
dcoggin Global Mapper UserTrusted User
edited January 2012 in SDK
Mike,

I have an application that uses the SDK to build bathymetric models where I load large numbers of point features. The performance during digitizing lines which sometimes connect these points is very poor (very slow) when I have a lot of point features. For example, digitizing with 1.7 million point features loaded makes the program very difficult to use -- it's no problem with only a few features loaded. Drawing lines with the same number of features loaded and snapping enabled in Global Mapper 13 is still very fast -- the number of features seems to have little effect on performance.

During the OnMouseMove function in my application when digitizing, I call GM_FindNearestFeatures as part of the code to check for a snap point. A code profiler says this is using almost all the time when I have a large number of features loaded (when there's a relatively small number of features drawing takes most of the time). When profiling the Debug build of the application, the average time per call to GM_FindNearestFeatures is almost 0.4 seconds making the application painfully slow. In practice the Debug build performs about the same as the Release build for this type of activity.

I have tried calling the function with both the current geographic and pixel screen bounds and with NULL for those arguments. With the 1.7 million point example I gave above, the points are contained in 17 vector layers, each with no other features but the points. Even when I try it with only one vector layer with 223,000 features, the performance is slow though much improved over the 1.7 million. I have been calling the function with NULL for the layer list argument, but I could add some code to compare the screen and layer bounding rectangles to make this smarter. However, from the fact it's still slow with only one layer, it's not going to be anywhere near as fast as Global Mapper.

I would appreciate any ideas and help in speeding up the performance. As always, thanks for your great support.

David

Comments

  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    I took a look and think I've got this fixed. Basically a fetch of the features to consider was being re-done with each call to GM_FindNearestFeatures, which could be slow with lots of features. I updated the SDK to cache the on-screen features, so until you change your draw list or need a different bounds it will just use the cached lists and should be much faster. I have placed a new SDK build with this change at http://www.globalmapper.com/GlobalMapperSDK_latest_beta.zip for you to try.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    I'm using the new version and it does speed it up, but I'm not getting any returned found features after the initial one when digitizing a line. For example, if I digitize a line to points, the first vertex snaps to a point, but the others after that do not. Even if I pan while digitizing, I don't get any returned found features.

    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    Can you show me how you are calling this? I have tried in the sample app with the existing handling of the right-click and it was working fine and not re-fetching the list as expected.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    Thanks for looking at it again. Then the problem is probably on my end, I just didn't see anything. Here's my call:

    theErr = GM_FindNearestFeatures
    (
    &thePixelSearchPt, // Search location in pixel coordinates
    m_pDoc->m_arrSelectableLayers, // Search all selectable layers
    m_pDoc->m_uNumSelectableLayers, // Num selectable layers
    GM_FindFlags_FindAll, // Search for closest point, line, or area
    &m_recViewGround, // Search world rectangle
    &m_recGMViewPixel, // Search pixel view rectangle
    theNearestFeatures, // Returned list of nearest features
    uMaxFoundFeatures, // Max number of nearest features to find
    puFoundFeatureCnt, // Number of nearest features returned in theNearestFeatures
    a_uMaxDistPixels // IN: Max distance in pixels to search
    );
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Woops, didn't mean to post that yet. More information. I've checked:

    thePixelSearchPt - converted to ground coordinates it's confirmed to be close to the known feature (I've checked with lines and points)

    The layerlist, number of layers, world rectangle, and pixel rectangle are unchanged in this application for a long time and they've been working fine. uMaxFoundFeatures is set to 1 and a_uMaxDistPixels is set to 10. This call is unchanged from before and was working fine. I'm not saying it's not on my end, I'll just need to do more looking to check it out. Thanks again for your help.

    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    If you pass in NULL for the list of layers and 0 for the layer count and NULL for the world and pixel rectangles is it any different? This would basically use all layers and the last drawn bounds.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    Looked through my changes and did find a problem if the cached feature bounding box matched but not the layer list that could be causing your issue. I am uploading a new build now to http://www.globalmapper.com/GlobalMapperSDK_latest_beta.zip for you to try. Upload is slow so give it about 15 minutes before you download to see if that helps.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    Thanks for looking at it again. I was getting some very strange behaviour with the last version -- styles changing, etc. This new version has none of those problems, but it doesn't seem any faster for finding features either. With the same large dataset I'm still getting very slow performance with calls to GM_FindNearestFeatures averaging almost 0.4 seconds. This occurs even when I'm zoomed in with few features on the screen (and they're all point features). It occurs whether I pass in my list of layers (which I need to do for my application) and rectangles or assign them all as NULL.

    Thanks for your help,
    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    Hmmm, are you making any calls in between that might use a different layer list than what you are passing in, like any kind of other draw or something? The bounds that you are passing in are just for the screen right and not the entire bounds of the loaded data? I'm just trying to think of some way that the cache of last fetched features wouldn't be used and the whole fetch done again. I suppose I could add some debug logging in there so we could see if the cache is being used or not.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    Yes, the FindNearestFeatures and Draw calls have different layer lists. The way I have it setup is that the current digitized line or area is on a separate TempFeaturesLayer, so that one is excluded from the list when finding features. Also, layers that aren't selectable are excluded. Is this what you were asking? There are Draw calls between the FindNearestFeatures calls.

    I don't know if you'd want to do this, but would it be possible to make it user controllable when to refetch onscreen features?

    The bounds are just for the screen and I've also tried with NULL for both rectangles with no change.

    David
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    Actually what's happening on the redraws while I'm digitizing is that only the new line or area being drawn is included in the DrawToOffscreenBuffer call. The rest of the data is BitBlt'd and TransparentBlt'd to display it. So what's being drawn during digitizing is usually the only layer I don't want in the FindNearestFeatures call.

    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    Ah ok this explains it then. What I will do is make a separate cache for each layer list/bounds combination, that way you can intermix the calls and it will still be fast.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike, that sounds great.

    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    I've got a new build at http://www.globalmapper.com/GlobalMapperSDK_latest_beta.zip which should keep multiple layer-specific feature lists. This gets cleared for each layer set when you load or close a layer or change the global projection or reposition a feature. And I just realized that you might be doing the latter all the time so I may have to make that a bit smarter too, darn.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    I can't extract the files from that archive correctly. I'm getting a CRC error -- whatever that is. If I open the folder to look at the contents, I only see the 32 bit dll and the .dat and .csv files.

    David
  • global_mapper
    global_mapper Administrator
    edited January 2012
    David,

    Looks like there was an upload problem. I went ahead and fixed the likely issue if you move a feature on some layer so the cache for unrelated layer lists aren't cleared. There is a new build at http://www.globalmapper.com/GlobalMapperSDK_latest_beta.zip for you to try.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • dcoggin
    dcoggin Global Mapper User Trusted User
    edited January 2012
    Mike,

    Thanks so much -- it works great! It took me a while to figure out because I was still testing with NULL for the layer lists which would of course required a new search at each MouseMove since I was changing my digitized line layer. Once I went back to passing in the correct layer list it worked like a champ.

    It's much faster now. It was to the point of being almost unusable with very big datasets. Thanks again for the great support!

    David