DIR_LOOP and subfolders

VincentVincent Global Mapper UserPosts: 28Trusted User
edited March 2013 in GM Script Language
Hi,

I want to mosaic a large number of aerial vertical images according to a certain directory structure. GM script and DIR_LOOP function seem to be a good solution for that but I didn't manage to write the right script (very slow, so I suppose that the loop is a bit bigger than expected...).
Here is my directory structure :

2011-Survey (Main directory)
AREA-1 (Sub-folder for the geographic area 1)
Line-A (Sub-folder for the flight line A, this sub-folder contains the images ECW + EWW files)
about 100 XXXX.ecw and XXXX.eww files. Each individual picture is georeferenced
Line-B
Line-C....
AREA-2
Line-J
Line-K.... etc

I want to go through each AREA folder. For AREA-1, I want to produce an ECW mosaic for each individual line (and not a full mosaic for all the AREA-1 using all the lines in the subfolders).
My idea is to :
- open the AREA-1 folder
- then to open the Line-A sub-folder
load all the ecw images that are contained in the Line-A sub-folder (using IMPORT_DIR_TREE command)
- then to export all the images loaded in a single ecw image using the EXPORT_RASTER to create a mosaic
- then to unload these images and move to the next line
- then to open the Line-B sub-folder and create a mosaic for this line, etc...
Once all the individual lines in the AREA-1 are done, I want to move to the next AREA...

Here is the script I used (test for just one AREA, I skip the initial "main loop") :
GLOBAL_MAPPER_SCRIPT VERSION=1.00
UNLOAD_ALL
SET_BG_COLOR COLOR=RGB(255,255,255)

DIR_LOOP_START DIRECTORY="F:\2011-Survey\AREA-1" FILENAME_MASKS=*.ecw RECURSE_DIR=YES

UNLOAD_ALL

IMPORT_DIR_TREE DIRECTORY="%DIR%" \
FILENAME_MASKS="*.ECW" TRANSPARENT_COLOR=RGB(0,0,0) TRANSPARENT_COLOR_DIST=5

EXPORT_RASTER FILENAME="F:\2011-Survey\Mosaic-ECW-line\AREA-1\Mosa-line_%PARENT_DIR%_L93.ecw" TYPE=ECW BG_TRANSPARENT=YES

DIR_LOOP_END

It works but it takes hours, so I am afraid that somethinks is wrong. When I do manually on GM some lines mosaic, it takes only few minutes.
Thanks for your help
Regards

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    The problem is that if you have 100 ECW files in a single folder, your inner script is going to run separately for each one of those ECW files, so by the end of that loop you will have loaded those 100 files 100 times. Is there some type of file that exists just once inside each of the sub-folders that you could loop on in the outer DIR_LOOP_START command so that each sub-folder is only visited once?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    Unfortunatly, in each sub-folders I have only images (.ecw and .eww files), no specific file I can stop the loop on it.
    The IMPORT_DIR_TREE command, important only the files in the specified directory or also the files in the directory sub-folders ?
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    The IMPORT_DIR_TREE can do everything in an entire tree, including sub-folders. What if you use FILENAME_MASKS=".." to match just on the folder "file", then you can use the path to it? I'm actually not 100% sure it will work. If not I can add something to make something like this work.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    I checked and the ".." didn't work before, but I have gone ahead and made it work. I have also added an EDIT_MAP_CATALOG script command for the upcoming v14 release. I have placed a new build at http://www.globalmapper.com/global_mapper13.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v13.xx installation folder to give it a try. If you are using the 64-bit v13 version there is a new build at http://www.globalmapper.com/global_mapper13_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    Hi Mike,

    Thanks for this update. I downloaded the new release and tested it, I tried the DIR_LOOP with the mask on "..", but it doesn't really work, it scans also all the files in the different sub-folders (I want to scan only one sub-folder at each loop).
    I made a test with few lines :
    Main directory "Test" containing the sub-folders for each flight line
    Sub-directories :
    • 100.0.1
    • 101.0.1 each sub-folder contains individual images acquired during each flight line (test made with only 4 lines)
    • 102.0.1
    • 102.1.1
    I want to open the Test directory tree, and create one mosaic for each line (so for each sub-folder), and write the mosaic files in a new subfolder "Test\Mosaic". I want to use for naming the mosaics, the name of the flight line (so the name of the subfolder) :
    > Test\Mosaic\100.0.1_mosa.ecw
    > Test\Mosaic\101.0.1_mosa.ecw
    > Test\Mosaic\102.0.1_mosa.ecw
    > Test\Mosaic\102.1.1_mosa.ecw

    Here is the script I used for that :
    GLOBAL_MAPPER_SCRIPT VERSION=1.00
    UNLOAD_ALL
    SET_BG_COLOR COLOR=RGB(255,255,255)

    DIR_LOOP_START DIRECTORY="D:\Test\" FILENAME_MASKS=".." RECURSE_DIR=NO

    IMPORT_DIR_TREE DIRECTORY="%DIR%" \
    FILENAME_MASKS="*.ECW" TRANSPARENT_COLOR=RGB(0,0,0) \
    TRANSPARENT_COLOR_DIST=5

    EXPORT_RASTER FILENAME="D:\Test\Mosa\%FNAME_WO_EXT%_mosa.ecw" \
    TYPE=ECW BG_TRANSPARENT=YES

    UNLOAD_ALL

    DIR_LOOP_END

    In fact it opens directly all the subfolders and create only one mosaic with all the images in the subfolders. I made other tests, but didn't manage to get it works. I can't find what is wrong with my script (but something is for sure ...).
    Help welcome !
    I can send you the images and folder structure I used for my test (only 3 Mb).
    Thank you
    Regards

    Vincent
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vincent,

    As you have your outer loop now you would just get 'D:\test' as the loop value as you aren't recursing. I think what you actually want is something like 'D:\test\100*\.." for your loop value for the first loop to see if that gives you a separate pass for each sub-folder. Either that or make the outer loop recurse sub-folders but not the IMPORT_DIR_TREE command.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    Mike,

    Not sure to understand what you suggest, should I use a "full tree" in the loop start command or in the mask ?
    I understand that you suggest 2 methods and not to use the IMPORT_DIR_TREE command (in both solutions or only for one). Can you detail the 2 solutions ?
    I continue my tests but I am "a bit' lost...
    Thanks
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    I run out of idea, and did not manage to get it works...
    Can you detail how the loop works and how the "\.." is understood by the script ?
    Can I send you some data samples for you to test and help me to make the script work ?
    Thanks
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    The ".." is the name of an actual file in each folder (goes back to DOS days). So when you seach on ".." as the filename mask you get the loop run once for each folder. So when you specify "D:\test" for the directory you only get the file "d:\test\..", so just the folder "d:\test". What you want is each folder under that run, so try the following:

    DIR_LOOP_START DIRECTORY="D:\TEST\100*" FILENAME_MASK=".."

    This should get the loop contents run for each directory starting with d:\test\100.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    Thanks Mike for the explanation.
    I did again some tests (I tried on 2 different computers with GM v13 and the new release, I copied the file as indicated before) but still not successfull...
    My file directory structure is :
    Main directory "Test" containing the sub-folders for each flight line
    Sub-directories :
    • 100.0.1
    • 101.0.1 each sub-folder contains individual images acquired during each flight line (test made with only 4 lines)
    • 102.0.1
    • 102.1.1

    With this command :
    DIR_LOOP_START DIRECTORY="D:\TEST\10*" FILENAME_MASKS=".." RECURSE_DIR=NO
    The loop should run at the sub-directory level.
    After I used the IMPORT_DIR_TREE command, I tried all the arguments availabe :
    "%DIR%" "%FNAME_W_DIR%" "%FNAME%" "%FNAME_WO_EXT%" "%PARENT_DIR%"
    But none is working as I want. These special character sequences work fine at a file level but not at a folder level.

    I tried also the IMPORT command, with and without nesting loops. But didn't manage to get it work, out of idea for the moment.
    Not certain that the LOOP command works to manage a certain directory tree. It works fine to open all the files in all the subfolders, but didn't find how to stop at each sub-folder level as you tried to explain to me (thanks for your patience !).
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Sorry I forgot that wildcards aren't supported in the directory mask directly. Instead try the following:

    DIR_LOOP_START DIRECTORY="D:\TEST\" FILENAME_MASKS="10*\.." RECURSE_DIR=YES

    Make sure to recurse since you want this going for each sub-folder.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VincentVincent Global Mapper User Posts: 28Trusted User
    edited August 2012
    Thanks a lot Mike, it works perfectly with this syntax.
    Perfect !
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    This should be really useful!! Good topic.
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    Hello All,

    This is my version of code:

    DIR_LOOP_START DIRECTORY="C:\GMFiles\" FILENAME_MASKS="tiff*\.." RECURSE_DIR=YES
    IMPORT_DIR_TREE DIRECTORY="%DIR%" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\TILEINDEX\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    UNLOAD_ALL
    DIR_LOOP_END

    It works like a charm, I just cannot get it to export a shapefile for each sub-folder for which it generates layer bounds. In the format above, it keep overwriting the export name: "Exporting vector data to file C:\TILEINDEX\..shp..." and in the end I only have the shape for the last folder in the loop.

    I would like to have a different shapefile for each sub folder, or all of them merged in one.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited March 2013
    Try building the parent folder of the input into your output name, like the following:

    EXPORT_VECTOR FILENAME="C:\TILEINDEX\%PARENT_DIR%\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    Try building the parent folder of the input into your output name, like the following:

    EXPORT_VECTOR FILENAME="C:\TILEINDEX\%PARENT_DIR%\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/


    I tried your advice and everything is ok now.

    Is there a solution to gain time when working with thousands of files?

    The idea is that in each of my 37 folders I have more that 400 tiff files. It takes hours for the process to complete for all. Is there a way to somehow "index" the files in a way? I think that the huge amount of time it takes to finish the task is due to the fact that it runs an "import" command for all the files in each folder (which is logical).

    I know that gdal with gdaltileindex only 'reads' the folder without actually loading the features in contrast to GMS. I did not work until now with "Map Catalog"..would this be a key to decrease time?
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited March 2013
    If you were exporting the contents of the TIFF files I would say use EDIT_MAP_CATALOG to create a map catalog, then export from that, but for just getting the bounds of each TIFF file that wouldn't really help much. How long does it take to run this to get a Shapefile with the bounds for every file? I really wouldn't expect that much for that operation, at least for most TIFF file types. Are your TIFF files just imagery or do they have terrain data?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    I was thinking at the same thing at first, that it would not be a time-related problem...

    It took about 30 mins for 6 folders and I have a lot of them, the number of files will increase.

    The file types are Imagery. tiff+prj for each file.

    As I saw in the script results the:

    IMPORT_DIR_TREE takes 3min44sec
    GENERATE_LAYER_BOUNDS takes 0min17sec
    UNLOAD_ALL takes 0min34sec to complete.

    This was the result generated for a folder with 1.538 tif files, size of folder is 900 MB (the tif files are small); If you sum up the time multiplied by 37 it takes a while. Also, often, I get the "Not responding" message but it clears and GM still does the job when it does the loop for the next folder. It can only be disturbing if you want to see the log result as it passes through each folder. It gets stuck and you cannot do that.

    If you glance at the results, they are quite good but if you compare it with GDAL, you might think that there is room for enhancements here.

    By comparison, the gdaltileindex in 10 minutes generates the shapefiles for all folders. I guess this occurs because one actually imports(loads) the data -GMS, while the other -GDAL- only reads the data.

    I would be great if a 'read_dir_tree' could be added in the future, and also, maybe a timestamp with the time of running for each process. It would be easy to see where your script is running slow.

    The rest of the script does a great job with the processes I used to do manually, in the Interface! Great job with that!

    I am just trying to see if I can boost all of it when it comes to time.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited March 2013
    The load of each TIFF file will get a bit slower as you get more of them loaded as the access switches from memory-mapped to slower full read access once your memory address space fills up.

    You could instead create a separate directory loop that separately does an IMPORT for each TIFF file rather than using IMPORT_DIR_TREE. In the inner loop use IMPORT to load the TIFF, GENERATE_LAYER_BOUNDS to create a bounding box layer for the file, then UNLOAD_LAYER to unload the TIFF layer. When the loop completes you will have a separate bounding box layer for each TIFF that was loaded. Then the normal EXPORT_VECTOR will export from those. This approach will result in just a single TIFF being loaded at a time.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    Ok, As I understand, I should use IMPORT and drop IMPROT_DIR_TREE. Use UNLOAD_LAYER and drop UNLOAD_ALL.
    I will try that and come back tomorrow with a result.

    Thanks Mike
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited March 2013
    Yes, but keep in mind that this is inside another directory loop inside your outer one, like the following:

    DIR_LOOP_START DIRECTORY="C:\GMFiles\" FILENAME_MASKS="tiff*\.." RECURSE_DIR=YES
    DIR_LOOP_START DIRECTORY="%DIR%" FILENAME_MASKS="*.tif" RECURSE_DIR=YES
    IMPORT FILENAME="%FNAME_W_PATH%"
    GENERATE_LAYER_BOUNDS
    UNLOAD_LAYER FILENAME="%FNAME_W_PATH%"
    DIR_LOOP_END

    EXPORT_VECTOR FILENAME="C:\TILEINDEX\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    UNLOAD_ALL
    DIR_LOOP_END

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    I think you were referring to %FNAME_W_DIR% and not %FNAME_W_PATH%.

    It runs faster at first, for sure, but the
    UNLOAD_LAYER FILENAME="%FNAME_W_DIR%" is not considered (it does not unload the layer) and so, it loads all of them again.
    DIR_LOOP_START DIRECTORY="C:\GMFiles\" FILENAME_MASKS="tiff*\.." RECURSE_DIR=YES        
           DIR_LOOP_START DIRECTORY="%DIR%" FILENAME_MASKS="*.tif" RECURSE_DIR=YES
                IMPORT FILENAME="%FNAME_W_DIR%"
                GENERATE_LAYER_BOUNDS LAYER_DESC="Tiles"
                UNLOAD_LAYER FILENAME=[COLOR=#000000]"%FNAME_W_DIR%"[/COLOR]
            DIR_LOOP_END
        EXPORT_VECTOR FILENAME="C:\TILEINDEX\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
        UNLOAD_ALL
    DIR_LOOP_END
    --------------------------------------------------------------------------------------
    
    Removed all loaded overlays.
    Starting loop over all files in directory...
    Starting loop over all files in directory...
    Importing file C:\GMFiles\tiff_one\test1(1).tif...
    Generating layer bounds areas from loaded data...
    Importing file C:\GMFiles\tiff_one\test1(10).tif...
    Generating layer bounds areas from loaded data...
    Importing file C:\GMFiles\tiff_one\test1(100).tif...
    Generating layer bounds areas from loaded data...
    Importing file C:\GMFiles\tiff_one\test1(101).tif...
    

    It loads all of the files as you can see below and it generates bounds for each tif file and then also for the entire tiff files loaded. Naturally, I only need those of the tif(s).
    attach.JPG

    If I replace with UNLOAD_LAYER FILENAME="C:\GMFiles\tiff_one\test1(102).tif" it works for that file only so I guess it cannot retrieve the right path from "%FNAME_W_DIR%" in the loop for the UNLOAD_LAYER command.

    If I use UNLOAD_ALL, it will remove my generated bound as well and if I place the EXPORT before the UNLOAD_ALL command, it does the job, but it exports a shapefile for each tif file (it generates ~ 4,800 files/folder).

    Anyhow, I think if the UNLOAD_LAYER FILENAME issue is corrected, surely it is faster than the previous try.

    Thanks,
    Robert
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    My apologies, I was misguided by the fact that the unload_layer command is not printed in the script result log and/or dialog but it indeed unloads the file(s).

    The extra bounding boxes are still created with the attribute FILENAME empty so I guess I can delete them based on this but unfortunately, after a very speedy start, it slows down a lot and the time consumed overall is longer than before for some reason.

    The result for the first folder containing 282 tiff files takes about 6mins40sec and it generates 39,903 areas (39,621 with FILENAME -> empty). To avoid this I may have to place an export_vector command after the generate one, and then unload all. But again, this will cause a high number of distinct shapefiles in the export folder, each containing one area.

    I think I would better try to somehow avoid the adding/deleting of tif files in my folders for which I already ran the generate_bounds command, so I would only need to run it for the last added folder. I'll see how that works out;

    Nevertheless, thanks for the help, I hope other users could gain something from this topic.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited March 2013
    Ah yes I can see that the GENERATE_LAYER_BOUNDS would add the bounds of both the TIFF file and the existing layer bounds layer, which is not what you want. I have updated GENERATE_LAYER_BOUNDS so you can now add a FILENAME attribute to specify to just add bounds for a particular layer. So your command could now look like:

    GENERATE_LAYER_BOUNDS LAYER_DESC="Tiles" FILENAME="%FNAME_W_DIR%"

    Then only the bounds for each TIFF file will be added. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v14.xx installation folder to give it a try. If you are using the 64-bit v14 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper14_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertRRobertR GlobalMapper Fan! Posts: 285Trusted User
    edited March 2013
    Quick again in reply and in producing great enhancements.

    Thanks again! and keep up the development of GMScripting. :) I think this could be the main asset of GM by which it will boost its functionality. I already made a suggestion.
Sign In or Register to comment.