Global Mapper v25.0

GM_LoadLayerListEx speed for directories with large amount of files

nickgoodliff
nickgoodliff Global Mapper UserTrusted User
edited May 2014 in SDK
Hi Mike

Is there anyway to stop the bottleneck in GM_LoadLayerListEx when the directory the file is in has a large amount of files (in my case > 320,000)? In my app, it is taking over a minute to load a 2MB tiff file. If the tiff files are in a directory with just a few files, then it loads in seconds.

I'm guessing your method must do some "FileExists" type of method. Could this be slowing it down? Is there anyway we can disable this check?

Cheers
Nick

Comments

  • global_mapper
    global_mapper Administrator
    edited May 2014
    Nick,

    I have added a new load flags, GM_LoadFlags_AssumeFileExists, that you can pass in to GM_LoadLayerListEx to disable the check for the existence of the file before loading. I have placed a new SDK build with the change at http://www.globalmapper.com/GlobalMapperSDK_v15_latest_beta.zip for you to try.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    Hi Mike

    Doesn't seem to have any effect unfortunately

    I do a check for the files myself in my c# .NET app using:
    var file = Directory.EnumerateFiles(folder, fileName, SearchOption.TopDirectoryOnly).FirstOrDefault();if (!String.IsNullOrEmpty(file))
    {
    path = file;
    }
    

    And it comes back instantly so not sure where the bottleneck is in the SDK.

    I'm calling the SDK with this command:
    GlobalMapperDLL.GM_LoadLayerListEx(path, out theLayerList, out theNumLayers, GlobalMapperDLL.GM_LoadFlags_t32.GM_LoadFlags_UseDefaultProj | GlobalMapperDLL.GM_LoadFlags_t32.GM_LoadFlags_AssumeFileExists, "");
    

    And it is taking 55 seconds to load each file from a directory container 320k, however running the exact same code when the there are only a few files in the directory takes < 1s.

    Any other ideas?

    Cheers
    Nick
  • global_mapper
    global_mapper Administrator
    edited May 2014
    Nick,

    Do you just have a single file loaded or are you trying to load all of those TIFF files at once? If you want to load them all you should create a map catalog from them as that will be tremendously more efficient than loading them all at once.

    I just created a folder with 320,000 TIFF files in it but could select and load the files from it very quickly. Do you know what file system your drive with the files is using? Mine is a NTFS drive. I have read that FAT32 drives have a lot more issues with large file counts in a single folder. Also having a bunch of similarly named files can slow things down if short (8.3) filename support is enabled on your system.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    Hi Mike

    At any time I will be loading a subset of them, depending on what area has been selected by a user. For my current test I am loading 4 files from the directory, they are named (derived from British Grid reference):

    ST5445.tif
    ST5545.tif
    ST5446.tif
    ST5545.tif

    There is also .tfw file (ST5445.tfw etc) for each TIFF. Is it a search for the .tfw file that could take time?

    The drive is actually a Network Drive, that is formatted as NTFS. It's just strange that when I use Directory.EnumerateFiles() to search for the file in my app, it returns instantly and the performance of read/write to the drive is nearly equivalent to a local drive.

    If necessary I'll get the files moved in a heirarchy of directories: /ST/5/ST5445.tif etc.

    Nick
  • global_mapper
    global_mapper Administrator
    edited May 2014
    Nick,

    It's possible the TFW file search is slowing down, especially if it also has to search for a projection file. I have added a bunch of debug logging to pin down the time as I can't get it to be slow on my machine even with the TFW files. I have placed a new SDK build at http://www.globalmapper.com/GlobalMapperSDK_v15_latest_beta.zip with the updated logging. To enable the logging, make the following calls at your app startup:

    GM_SetMiscOptionEx( GM_MiscOpt_SetLogLevel, GM_Log_Debug, NULL );
    GM_SetMiscOptionEx( GM_MiscOpt_SetLogFilename, "c:\\path_to_log_file\\log_filename.txt", NULL );

    Then just send me the resulting log so I can see where the time is.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    Thanks Mike - I've just spent the last 24 hours moving the files into sub directories and this has fixed the issue to a large degree. There is now only 1000 .tif files per directory (2000 files altogether including the .tfw) and this the output from the log:

    [05/09/14 14:29:29] Calling XTIFFOpenEx on W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif
    [05/09/14 14:29:29] Found TIFF world file W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tfw
    [05/09/14 14:29:29] Read of TIFF world file OK
    [05/09/14 14:29:29] Used world file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif
    [05/09/14 14:29:29] Search for .TAB file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.2 s
    [05/09/14 14:29:29] Search for .prj file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:29] Search for .map file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .map file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .TAB file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .txt file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .gm_xform file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .pal file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .clr file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .smp file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .act file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .tbl file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:30] Search for .lut file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s
    [05/09/14 14:29:31] Search for .met file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\5\ST5445.tif took 0.1 s

    Some strange duplication of searches for .tab and .map files?

    I'll move some files back into 1 big folder and report back later... it would be good to be able to set a flag so that no search is made for anything but a specified info file like .tfw?

    Nick
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    I can also confirm that this only happens on the Network Drive, when I run this on a local drive, all the searches take 0.0s to run - which is probably why you are struggling to replicate it!

    Nick
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    Ok, with just over 10k files in a directory we're up to 3 seconds per tiff:

    [05/09/14 15:45:19] Calling XTIFFOpenEx on W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif
    [05/09/14 15:45:19] Found TIFF world file W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tfw
    [05/09/14 15:45:19] Read of TIFF world file OK
    [05/09/14 15:45:19] Used world file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif
    [05/09/14 15:45:19] Search for .TAB file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:20] Search for .prj file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:20] Search for .map file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:20] Search for .map file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\445.tif took 0.1 s
    [05/09/14 15:45:20] Search for .TAB file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:20] Search for .txt file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:21] Search for .gm_xform file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
    [05/09/14 15:45:21] Search for .pal file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.3 s
    [05/09/14 15:45:21] Search for .clr file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.3 s
    [05/09/14 15:45:21] Search for .smp file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.2 s
    [05/09/14 15:45:22] Search for .act file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.3 s
    [05/09/14 15:45:22] Search for .tbl file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.2 s
    [05/09/14 15:45:22] Search for .lut file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.3 s
    [05/09/14 15:45:22] Search for .met file for W:\Mapshop_Full_Res_Tiles\NDVI\Tiles\ST\ST5445.tif took 0.1 s
  • global_mapper
    global_mapper Administrator
    edited May 2014
    Nick,

    Ah it could be much slower on network drives. The real issue is that there doesn't look to be a projection file to provide the projection once the TFW is found and there must not be a projection in the TIF file, so a whole host of different metadata files are checked for to try and automatically figure out the projection without having to ask the user. Usually all of those searches are fast, but on a network drive things could take a lot longer. Splitting into folders works better (as you've noted) as would adding a PRJ file or if possible having GeoTIFFs with all of that embedded right in the TIF.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • nickgoodliff
    nickgoodliff Global Mapper User Trusted User
    edited May 2014
    All the data is in British Grid OSGB36 projection so I just load a projection file at the start of the app.

    I do pass in GM_LoadFlags_t32.GM_LoadFlags_UseDefaultProj to my LoadLayerEx method so thought this might stop the checking for a projection file?

    Anyway, I'm more than happy to keep the directories at less than 5k per directory for the network drive - makes it a lot easier with Windows Explorer too as that was taking a couple of minutes to display the files in the large directories!

    Cheers
    Nick
  • global_mapper
    global_mapper Administrator
    edited May 2014
    Nick,

    The GM_LoadFlags_UseDefaultProj flag means that if no projection could be found from any other means then the default projection will be used rather than asking the user. If you want to disable the extra file checks you can use GM_LoadRectifiedLayer and actually pass in the GM_Projection_t to use for the projection. Just pass in NULL for the control points. This will have the projection set up front so it won't need to be looked for.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation