Automatically choosing export type / projection based on import

George Chandeep Corea

I am writing a python script that generates GMS files for processing directories of files listed in a CSV.

The input types are (mostly) shp/tab/kml for vector and ecw/jpg/tif for Raster. I want the output type to be the same as the input type and the data should be clipped to a fixed boundary.

I can get it to do this for a single file type but I would like it to handle at least the filetypes listed above. Do I need to code each one separately? This will be quite inefficient.

1. Can I get the GMS to automatically set output type to the same as input TYPE.
2. Similarly do I need to set the FILENAME_MASKS?

I tried without these two switches and GMS shows an error when it it run in GM->run script

I have attached the files I am using and the gms files that are produced. 1.gms is for a single file, 2.gms is for a directory.

import os, csv
#print "running script"
os.chdir(r"P:\2014\047_MSC_DataSplit\Working\gms")
#os.system(r"C:\Program Files\GlobalMapper14_64bit\global_mapper14.exe" "GM_Script_Scratchpad.gms")
#os.system("run_GM.bat -InFile \"D:\Scratch\gms\335000_8084000.tif\" -OutFile \"D:\Scratch\gms\export\test.ecw\" -OutFormat \"ECW\"")


RootDir="L:\\"
filecount=1
vector_list=['shp','tab']
raster_list=['ecw','jpg','tif']
out_gms_file=r"P:\2014\047_MSC_DataSplit\Working\gms\\"
out_bat_file=r"P:\2014\047_MSC_DataSplit\Working\gms\file.bat"
##in_file=r"D:\Scratch\gms\335000_8084000.tif"
##out_file=r"D:\Scratch\gms\export\test_crop.ecw"
FileList = csv.reader(open('filelist.csv'))


for File in FileList:
    print File


#Create GMS file
    filename=File[0]
    ext=filename[filename.rfind('.')+1:]
    out_gms = open (out_gms_file+str(filecount)+'.gms', "w")
    out_gms.write("GLOBAL_MAPPER_SCRIPT VERSION=1.00 \n")
    print "Extension: "+ ext
    if filename.rfind('.')==-1:
        out_gms.write("DIR_LOOP_START DIRECTORY="+File[0]+ " FILENAME_MASKS=\"*.shp\"\n")
        out_gms.write("IMPORT FILENAME=\"%FNAME_W_DIR%\" \n")
        out_gms.write("EXPORT_VECTOR FILENAME=\"D:\\scratch\\gms\export\\"+File[0].strip(RootDir)+"\\%FNAME_WO_EXT%.KML\"TYPE=KML")
        out_gms.write("\nDIR_LOOP_END")


    else:
        out_gms.write("IMPORT FILENAME="+File[0] +"\n")
        if ext in raster_list:
            print "raster"
            out_gms.write("EXPORT_RASTER FILENAME=" + "D:\\scratch\\gms\export\\"+File[0].strip(RootDir)+" ")
            out_gms.write("TYPE=" + File[2]+" ")
            out_gms.write("POLYGON_CROP_FILE=D:\Scratch\gms\\fileboundary.shp \n")
        if ext in vector_list:
            print "vector"
            out_gms.write("EXPORT_VECTOR FILENAME=" + "D:\\scratch\\gms\export\\"+File[0].strip(RootDir)+" ")
            out_gms.write("TYPE=" + File[2]+" ")
            out_gms.write("POLYGON_CROP_FILE=D:\Scratch\gms\\fileboundary.shp \n")


    out_gms.close()
    filecount=filecount+1
    #os.system("run_GM.bat")

Python Output

>>> 
['D:\\Scratch\\gms\\335000_8084000.tif', 'D:\\Scratch\\gms\\export\\test.ecw', 'ECW']
Extension: tif
raster
['L:\\Vector_Data\\Administrative\\Cultural\\Native_Title\\SHP', '', 'TAB']
Extension: L:\Vector_Data\Administrative\Cultural\Native_Title\SHP
['L:\\Vector_Data\\Infrastructure\\CAD\\Mareeba\\GPS_Points', '', '']
Extension: L:\Vector_Data\Infrastructure\CAD\Mareeba\GPS_Points
['L:\\Vector_Data\\Infrastructure\\Water_Reticulation\\Eacham\\TAB', '', 'TAB']
Extension: L:\Vector_Data\Infrastructure\Water_Reticulation\Eacham\TAB
['L:\\Vector_Data\\Relief\\Contours\\1_Metre\\SHP\\Lidar_2011_Tiles', '', '']
Extension: L:\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles
['L:\\Vector_Data\\Relief\\Contours\\10Metre\\10m_NQWMP_LGA_Contours\\SHP', '', 'TAB']
Extension: L:\Vector_Data\Relief\Contours\10Metre\10m_NQWMP_LGA_Contours\SHP
['L:\\Raster_Data\\Elevation\\DEM\\Tile_ASCII_DEM\\Atherton\\asc', '', 'ECW']
Extension: L:\Raster_Data\Elevation\DEM\Tile_ASCII_DEM\Atherton\asc
['L:\\Raster_Data\\Aerial_Photography\\12_5_cm_June2011\\Millaa_Millaa\\ECW', '', 'ECW']
Extension: L:\Raster_Data\Aerial_Photography\12_5_cm_June2011\Millaa_Millaa\ECW
>>> 

RobertR

Hello,
You could also add a log file in the .gms file right after "GLOBAL_MAPPER_SCRIPT VERSION=1.00"
Add something like:

SET_LOG_FILE FILENAME="D:\Scratch\gms\my_script.log"

This file can help you see where you have errors.

George Chandeep Corea

Thanks Robert. This will be useful. Anyone

1. Can I get the GMS to automatically set output type to the same as input TYPE.
2. Similarly do I need to set the FILENAME_MASKS?

global_mapper

I have updated the TYPE parameter to specify providing FIRST_LOADED or LAST_LOADED as values to set it to be the type of the first or last loaded file. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

You do need the FILENAME_MASKS parameter, but you can set it to "*" if you just want everything.

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks Mike, LAST_LOADED type will be very useful. Rather than * for all files, can you add a routine to only load known vector, raster or elevation files. I tried * earlier and it imported an XML that it then couldn't process and crashed.

global_mapper

I've added a todo item (#13332) to support special values in the FILENAME_MASKS to match some of the special masks like common vector, raster, and elevation. I'll let you know when you can do that.

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks, do you have an approximate date by when this will be done? If it's going to be months then I will have to code around it by manually including each possible file type. If it's just a couple of weeks then I can delay the project till then. Regards,

global_mapper

I'll probably do it in the next few days, it's on my short-term todo list. Shouldn't be very hard. Might even be tonight

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

global_mapper

I've added support for the following 4 special values in the FILENAME_MASKS field:


COMMON_ALL - a filter with all Commonly Supported Formats
COMMON_ELEV - a filter with all Commonly Supported Elevation Grid Formats
COMMON_RASTER - a filter with all Commonly Supported Raster Formats
COMMON_VECTOR - a filter with all Commonly Supported Vector Formats

I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks, this is great. The issue is when using LAST_LOADED and COMMON_ALL for example is that you still have to specify EXPORT_ as vector/raster/elev but you don't know what the input is. Is there a way to test for this?

Ideally,

[COLOR=#d3d3d3]GLOBAL_MAPPER_SCRIPT VERSION=1.00 [/COLOR]
[COLOR=#d3d3d3]SET_LOG_FILE FILENAME="D:\Scratch\gms\MSC_DataExtract.log"[/COLOR]
[COLOR=#d3d3d3]DIR_LOOP_START DIRECTORY=L:\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles RECURSE_DIR=YES [/COLOR]FILENAME_MASKS=COMMON_ALL
[COLOR=#d3d3d3]IMPORT FILENAME="%FNAME_W_DIR%" [/COLOR]
EXPORT_COMMON_ALL [COLOR=#d3d3d3]FILENAME=D:\scratch\gms\export\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles\%RECURSE_FOLDER%\[/COLOR]%FNAME% TYPE=LAST_LOADED[COLOR=#d3d3d3] POLYGON_CROP_FILE=P:\2014\001_TP_Deamalgamation_Production\Working\base_data\MSC_Boundary_Locgovbdy_dcdb.shp [/COLOR]
[COLOR=#d3d3d3]UNLOAD_ALL 
DIR_LOOP_END

[/COLOR]

global_mapper

That one may be a bit trickier as some export formats might apply to multiple exports. Maybe you could have EXPORT_ELEVATION, EXPORT_RASTER, and EXPORT_VECTOR all in the loop with LAST_LOADED, then it will just log an error if the LAST_LOADED type value doesn't make sense. It would also be easy for me to make the LAST_LOADED silently fail if it isn't known for the export command.

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Yup, I'll write it as you have suggested. I am just wondering how to let the python code that creates the gms know if it is a vector/raster/elevation? Can the GMS look for input file type and then choose the appropriate export command?

global_mapper

I have added a new EXPORT_ANY script command that automatically selected the appropriate EXPORT_* command based on the TYPE parameter. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

Let me know if I can be of further assistance.

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks for your quick updates.

It works great from my current (limited) testing but it try's to read .xml metadata files that are associated with the .shp or other gis files. Can it exclude these automatically? maybe add a switch so that we can exclude any file types in the FILENAME_MASKS=COMMON_ALL option like FILENAME_MASKS=COMMON_ALL EXCLUDE('xml','jpg') etc.

Current Code is

GLOBAL_MAPPER_SCRIPT VERSION=1.00 
SET_LOG_FILE FILENAME="D:\Scratch\gms\MSC_DataExtract.log"
DIR_LOOP_START DIRECTORY=D:\Scratch\gms\input_testing RECURSE_DIR=YES FILENAME_MASKS=COMMON_ALL
IMPORT FILENAME="%FNAME_W_DIR%" 
EXPORT_ANY FILENAME=D:\Scratch\gms\output\%RECURSE_FOLDER%\%FNAME% TYPE=LAST_LOADED POLYGON_CROP_FILE=P:\2014\001_TP_Deamalgamation_Production\Working\base_data\MSC_Boundary_Locgovbdy_dcdb.shp 
UNLOAD_ALL 
DIR_LOOP_END

George Chandeep Corea

Further testing shows...

1.

WARNING: Error loading MapInfo file \Scratch\gms\input_testing\1m\1m_Contour_Index.TAB
Unsupported table type 'SHAPEFILE' in file \Scratch\gms\input_testing\1m\1m_Contour_Index.TAB. This type of .TAB file cannot be read by this library.
MapInfoOverlay.cpp - 65
Version: v15.1.7 (64-bit)
Build Time: Mar 12 2014 17:04:08
Use Ctrl+V to Paste Error

2.

WARNING: No overlays could be loaded from file <D:\Scratch\gms\input_testing\1m\1m_Contour_Index.TAB>.
ERROR: Unknown TYPE value <LAST_LOADED> in EXPORT_ANY command.

3.

WARNING: Exception running script \Scratch\gms\GM_test.gms
Open() failed for \Scratch\gms\output\1m\Walkamin\TAB: invalid filename extension
MapInfoOverlay.cpp - 65
Version: v15.1.7 (64-bit)
Build Time: Mar 12 2014 17:04:08
Use Ctrl+V to Paste Error

It has also skipped xyz, las and ecw files.

sub_dirlist.txt - list of all the files in the input directory. As you can see there is a mix of data types. I haven't included a fgdb or ESRI Grid.

MSC_DataExtract.log.txt - log of process

global_mapper

I've added support for adding FILENAME_MASKS_EXCLUDE to the DIR_LOOP_START command to specify any file extensions to skip. So you could use:

DIR_LOOP_START DIRECTORY="folder_her" FILENAME_MASKS="COMMON_ALL" FILENAME_MASKS_EXCLUDE="*.xml *.jpg"

to get all common formats except .xml and .jpg files.

I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks again for all this.

It works great except when clipping.

The prj file and metadata are not being brought across.

Capture.JPG

As it is working on thousands of files, is it possible to get GM to not write an output file if no data exists within the clip area?

The last line of output is

Removed all loaded overlays.
Importing file L:\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles\Biboohra_Koah\330000_8135000_1k_cont_1m.shp...
Exporting vector data to \scratch\gms\export\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles\Biboohra_Koah\\330000_8135000_1k_cont_1m.shp...
Removed all loaded overlays.

The code is

GLOBAL_MAPPER_SCRIPT VERSION=1.00 
SET_LOG_FILE FILENAME="D:\Scratch\gms\MSC_DataExtract.log"
DIR_LOOP_START DIRECTORY=L:\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles RECURSE_DIR=YES FILENAME_MASKS="COMMON_ALL" FILENAME_MASKS_EXCLUDE="*.xml *.jpg" 
IMPORT FILENAME="%FNAME_W_DIR%" 
EXPORT_ANY FILENAME=D:\scratch\gms\export\Vector_Data\Relief\Contours\1_Metre\SHP\Lidar_2011_Tiles\%RECURSE_FOLDER%\%FNAME% TYPE=LAST_LOADED POLYGON_CROP_FILE=P:\2014\001_TP_Deamalgamation_Production\Working\base_data\MSC_Boundary_Locgovbdy_dcdb.shp 
UNLOAD_ALL 
DIR_LOOP_END

global_mapper

If you add GEN_PRJ_FILE=YES to the EXPORT_ANY command then you should get a PRJ file created for each output file. There isn't currently any facility for creating metadata files, although you could perhaps add a RUN_COMMAND (Global Mapper Scripting Reference) command after the EXPORT_ANY to copy over any .prj and .xml files from the source location to the output if you want to copy those files that GM doesn't generate.

The clipping is a bit more difficult. Any files where no features intersect the bounding box of the clip rectangle should already be skipped, but there would need to be something added internal to each format export to detect that when actually clipped that nothing ended up written, then those files deleted at the end. That would be a rather large effort to do all formats.

Thanks,

Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation

George Chandeep Corea

Thanks Mike, it's all done and working now.

If anyone wants the python code to enable you to use GM to process specific directories of mixed file types from a csv, clip the files and capture the metadata please contact me.

Regards,

Geo

George Chandeep Corea wrote: »

If anyone wants the python code to enable you to use GM to process specific directories of mixed file types from a csv, clip the files and capture the metadata please contact me.

Hello George Chandeep,

Thank you for sharing your code in python.
It will give the final version of the example you gave in first post and let me study how you've built your automation process.

Thanks,
Richard.

PS:
How I use Global Mapper could be also a good place to post it.

George Chandeep Corea

Hi Richard, thanks for the suggestions. I have added the script and instructions to the following

http://www.globalmapperforum.com/forums/how-i-use-global-mapper/11254-use-gm-script-python-process-mix-files-csv.html#post40329

Regards,