Global Mapper v25.0

Building a shapefile as a raster tileindex in GMS (gdaltileindex from GDAL)

RobertR
RobertR GlobalMapper Fan!Trusted User
edited March 2013 in GM Script Language
I would like to use the GM Scripting Language to build a shapefile as a raster tileindex;


Currently, I am using gdaltileindex to do this, and it runs like this in Linux:

synopsis: gdaltindex [-tileindex field_name] [-write_absolute_path] [-skip_different_projection] [-t_srs target_srs] index_file [gdal_file]*


example: gdaltindex /me/maps/data/tiles.shp /me/maps/data/final/*.tif;


Is it possible to do this using GMS? I would like to transfer the entire project in the GM script..I managed to do all except this one. I am kinda new with GM Scripting so bear with me a bit.


Many thanks for the support!

Comments

  • global_mapper
    global_mapper Administrator
    edited March 2013
    If you use the GENERATE_LAYER_BOUNDS (http://www.bluemarblegeo.com/knowledgebase/global-mapper/ScriptReference.html#cmd_generate_layer_bounds) script command you can create bounding areas for any loaded raster layers, then use EXPORT_VECTOR to export that out to a file.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Hello Mike, that's almost good, but what if I have 30.000 raster files and I want to create the tile index for all? In batch, by using gdal, I do not need to load all of them, it would surely cause problems to load such an amount of raster files.
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Update. I think that loading so many tiff files will be an issue and I encounter the same problem. I would like to keep only one or two columns in the resulted shapefile, the attribute that contains the full path adn maybe the EPSG code.

    ATTR_TO_DELETE is not working for me, it does not delete the column(s). I was thinking of something like EDIT_VECTOR FILENAME="xxxx" KEEP_COLUMN="LOCATION" would be awesome but I dunno if we have this option somewhere, under another name that I am missing. I skimmed every post in the Scripting thread and did not find something similar.

    Here is my code with a few comments with questions on enhancements and/or fixes:

    GLOBAL_MAPPER_SCRIPT VERSION=1.00


    / We import the TIFF files from default directory
    / I want to check if all the tif files have the correct EPSG code, if not, reproject it.
    /(here I have two folders "tiff_one" and "tiff_two" but there can be up to 100 folders eventually)
    /would it be ok to unload the folders at some point? it takes a while for these two, imagine for all 100
    IMPORT_DIR_TREE DIRECTORY="C:\GMFiles" FILENAME_MASKS="*.tif"


    /Generate shapefile with path towards location
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMFiles\layer_bounds.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    UNLOAD_ALL


    /Import resulted shapefile to delete attribute columns which are not needed
    IMPORT FILENAME="C:\GMFiles\layer_bounds.shp"


    /How could I insert a whole list of attribute columns?
    /It would be easier to add the one that we need to keep in this case
    EDIT_VECTOR FILENAME="C:\GMFiles\layer_bounds.shp" ATTR_TO_DELETE="ELEVATION"


    /Export the result which should contain a clean shapefile
    EXPORT_VECTOR FILENAME="C:\GMFiles\layer_bounds_no_elev.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"

    Result Log

    Importing directory C:\GMFiles...
    Generating layer bounds areas from loaded data...
    Exporting vector data to file C:\GMFiles\layer_bounds.shp...
    Removed all loaded overlays.
    Importing file C:\GMFiles\layer_bounds.shp...
    Editing vector features...
    WARNING: Empty COMPARE_STR list, all features will be matched on.
    Updated 0 features.
    Exporting vector data to file C:\GMFiles\layer_bounds_no_elev.shp...
    Script processing COMPLETED.


    In the end, I still have all the columns (a number of 40) and I need to keep only one or two.

    Can you recommend a way to do this? It seems to be almost what I need, except for the questions raised in the comments. Anyway, GM Scripting is a great way to enhance tasks.

    Thanks for the support
  • global_mapper
    global_mapper Administrator
    edited March 2013
    You don't actually need to export your intermediate results to a Shapefile and reload them. Just add LAYER_DESC="Bounds" or something like that to GENERATE_LAYER_BOUNDS and then you can use FILENAME="Bounds" to refer to that layer later on.

    I don't think the layer bounds areas would have an ELEVATION attribute to delete so that ATTR_TO_DELETE wouldn't do anything. You can provide multiple on the same line though. So something like:

    EDIT_VECTOR FILENAME="Bounds" ATTR_TO_DELETE="UPPER LEFT X" ATTR_TO_DELETE="UPPER LEFT Y" \
    ATTR_TO_DELETE="LOWER RIGHT X"

    Just add all of the attributes you want to get rid of. There isn't currently a way to specify just those attributes to keep.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Ok, I will try again, but somehow, I have it:

    attribue_table.JPG

    Would it help to load all the tif s from a folder, create tiles, then unload_all and proceed to the next one?
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Ok.

    I found the problem. When I click to edit a feature, I get the actual names of the columns. When I click on the info tool, I have those that you see in the attach. Different display of column names.
  • global_mapper
    global_mapper Administrator
    edited March 2013
    Ah, when you export to a Shapefile as an intermediary the field names are cropped to 10 characters (a format limitation). So if you are exporting to Shapefile first and reloading you would need to delete the modified attribute names. I would just not export to Shapefile and then you don't have to worry about the names changing.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Ok, I understand. Thanks for clarifying that.

    I would have a final question regarding my script. I am repeating a command for 37 folders located in a parent folder (like in the first attach).

    snip1.JPG
    My code:
    [SIZE=2]GLOBAL_MAPPER_SCRIPT VERSION=1.00
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA[B][COLOR=#b22222]\week1\[/COLOR][/B]" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS[B][COLOR=#b22222]\shapefile1.shp[/COLOR][/B]" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA[B][COLOR=#008000]\week2\[/COLOR][/B]" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS[B][COLOR=#008000]\shapefile2.shp[/COLOR][/B]" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA[B][COLOR=#0000ff]\week3\[/COLOR][/B]" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS[B][COLOR=#0000ff]\shapefile3.shp[/COLOR][/B]" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA\week4\" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\shapefile4.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA\week5\" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\shapefile5.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL
    
    
    IMPORT_DIR_TREE DIRECTORY="C:\GMScript\ALL_DATA\week6\" FILENAME_MASKS="*.tif"
    GENERATE_LAYER_BOUNDS
    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\shapefile6.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    
    
    UNLOAD_ALL[/SIZE]
    

    The only thing that changes from one set to another is the folder that is read/imported (week1,2,3...37) and its corresponding shapefile output (shapefile1,2,3...shp)

    I used this method to avoid loading so many .tif files at once. I tried to look over the DIR_LOOP command but as I saw in the example ( http://www.globalmapper.com/helpv14/ScriptReference.html#sample_create_contours , I just cannot make it work in my case.

    Is there a solution to maybe reduce the code length so I will not have to insert 37 paragraphs and another extra paragraph of code every time I add a new folder with data?

    In batch I used a "for": for I in {1..37}; do gdaltindex [...]

    Basically what I want to do is to "read" the folders and for each, to export a shapefile with layer_bounds. A single shapefile output would also work but how can I do that without reading/importing the .tif files all at once? (i have thousands of them);
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    This is what I tried and failed. Reason: It keeps running the generate and export command for the same folder on and on (week10), it does not go to the next one (week29 for eg).
    GLOBAL_MAPPER_SCRIPT VERSION=1.00
    UNLOAD_ALL
    
    DIR_LOOP_START DIRECTORY="C:\GMScript\ALL_DATA\" FILENAME_MASKS="*.tif" RECURSE_DIR=YES
        IMPORT_DIR_TREE DIRECTORY="%DIR%" FILENAME_MASKS="*.tif"
        GENERATE_LAYER_BOUNDS
        EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
        UNLOAD_ALL
    DIR_LOOP_END
    UNLOAD_ALL
    

    Another try which I made was by following your example in a previous topic ("DIR_LOOP_START DIRECTORY="D:\TEST\100*" FILENAME_MASK="..") but again, The result was negative, see bottom of code example;
    GLOBAL_MAPPER_SCRIPT VERSION=1.00
    UNLOAD_ALL
    
    DIR_LOOP_START DIRECTORY="C:\GMScript\ALL_DATA\week*\" FILENAME_MASKS="*.tif" RECURSE_DIR=YES
        IMPORT_DIR_TREE DIRECTORY="%DIR%" FILENAME_MASKS="*.tif"
        GENERATE_LAYER_BOUNDS
        EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
        UNLOAD_ALL
    DIR_LOOP_END
    UNLOAD_ALL
    
    --------------------------------------------------------
    Removed all loaded overlays.
    Starting loop over all files in directory...
    WARNING: No files matching the specified mask were found under the specified directory. The loop will be skipped.
    Loop over files in directory completed.
    Removed all loaded overlays.
    Script processing COMPLETED.
    
  • global_mapper
    global_mapper Administrator
    edited March 2013
    There is a special feature allowing you to hit each folder in a directory tree just once. If you use the following DIR_LOOP_START you should get what you want:

    DIR_LOOP_START DIRECTORY="C:\GMScript\ALL_DATA\" FILENAME_MASKS=".." RECURSE_DIR=YES

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Sorry, to ask again, but isn't that exactly what I used in the Code sample? What seems to be incorrect in my code which causes the repetition of the same folder over and over again?


    DIR_LOOP_START DIRECTORY="C:\GMScript\ALL_DATA\" FILENAME_MASKS="*.tif" RECURSE_DIR=YES

    IMPORT_DIR_TREE DIRECTORY="%DIR%" FILENAME_MASKS="*.tif"

    GENERATE_LAYER_BOUNDS

    EXPORT_VECTOR FILENAME="C:\GMScript\LAYER_BOUNDS\%FNAME_WO_EXT%.shp" TYPE="SHAPEFILE" SHAPE_TYPE="AREAS" GEN_PRJ_FILE="YES"
    UNLOAD_ALL
    DIR_LOOP_END

    UNLOAD_ALL
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    I tried with IMPORT FILENAME="%FNAME_W_DIR%" FILENAME_MASKS="*.tif" instead of IMPORT_DIR_TREE DIRECTORY="%DIR%" FILENAME_MASKS="*.tif" and it works.

    The setback is that it exports a tile for each TIFF file which is too much as I have thousands of TIFFs.

    I think that dir_loop_start does not work properly with
    IMPORT_DIR_TREE DIRECTORY="%DIR%".
  • global_mapper
    global_mapper Administrator
    edited March 2013
    You missed the key change, which is using FILENAME_MASKS=".." . Note that the mask is ".." and not "*.tif". The ".." is special and causes each folder to be hit just once as there is just a single ".." file inside each folder.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/products/global-mapper.php
  • RobertR
    RobertR GlobalMapper Fan! Trusted User
    edited March 2013
    Damn me! Many thanks and sorry for that. Now it's clear.