Scripting: Batch looping slows down Bug/Memory-Issue?

PeterDrexelPeterDrexel Global Mapper UserPosts: 41Trusted User
edited July 2012 in Bug Report
Mike,
I am using the Script
Tra.gms with a gcp-File (Trans.txt, the full file contains about 5000 Lines) to batch-adjust several hundred tifs to their correct possition.
the job runs really fast for the first 100 tifs, but then slows down. Process Explorer (Sysinternals) shows rising values for Private Bytes (up to more then 10 GB). If I run the Job "Inside" GM (dropping the gms-File into the GM-Window) and cancel the job after 200 Tifs processed the Overlay Control center shows only one image loaded but GM is still using 10 GB of Private Bytes. It seems that the transform-file-data is not unloaded properly...
By the moment I can work splitting the job and run it in smaler pieces.
Thanks
Mike

Tra.gms:

GLOBAL_MAPPER_SCRIPT VERSION="1.00" FILENAME="D:\test.gmw"

UNLOAD_ALL
DEFINE_PROJ PROJ_NAME="AUSTRIA_ZONE4_MGI_AUSTRIA"
Projection Austrian Grid
Datum MILITAR_GEOGRAPHISCHE_INSTITUTE
Zunits NO
Units METERS
Zone 4
Xshift 0.000000
Yshift 0.000000
Parameters
END_DEFINE_PROJ

DIR_LOOP_START DIRECTORY="c:\in\" FILENAME_MASKS="*.TIF" RECURSE_DIR=YES
IMPORT FILENAME="%FNAME_W_DIR%" \
TYPE="GEOTIFF" PROJ_NAME="AUSTRIA_ZONE4_MGI_AUSTRIA" \
SAMPLING_METHOD="BILINEAR" CONTRAST_SHARED="YES" CONTRAST_MODE="NONE" CLIP_COLLAR="NONE" \
TRANSFORM_FILENAME="c:\trans.txt" TRANSPARENT_COLOR="RGB(0,0,0)"
EXPORT_RASTER FILENAME="c:\out\%FNAME_WO_EXT%_31254.tif" \
TYPE="GEOTIFF" SPATIAL_RES=0.1,0.1 GEN_WORLD_FILE=YES SAMPLING_METHOD="BILINEAR" COMPRESSION="LZW"
UNLOAD_ALL
DIR_LOOP_END



__________

Trans.txt:

563051.77000,5230221.53000,-37884.80000,231632.34000
562599.85000,5233619.37000,-38278.51000,235038.59000
565173.54000,5231425.34000,-35741.93000,232800.04000
563877.47000,5231057.10000,-37044.59000,232453.96000
562598.23000,5233620.56000,-38280.11000,235039.80000
563096.93000,5230279.25000,-37838.64000,231689.30000
564856.35000,5223684.27000,-36191.70000,225062.80000
565733.73000,5221607.16000,-35349.69000,222970.17000
565972.49000,5221656.43000,-35110.02000,223015.35000
564776.67000,5221605.45000,-36306.97000,222984.92000
567012.90000,5221851.49000,-34066.09000,223192.63000
566353.30000,5222263.59000,-34718.76000,223616.14000
567908.36000,5222761.91000,-33154.86000,224087.93000
566323.28000,5222314.41000,-34747.92000,223667.48000
565927.33000,5221660.09000,-35155.13000,223019.79000
565922.03000,5221662.63000,-35160.39000,223022.42000
555952.99000,5227418.21000,-45033.14000,228950.00000
555431.23000,5226804.20000,-45565.53000,228344.80000
554443.51000,5227245.33000,-46545.94000,228802.92000
...


Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited July 2012
    Can you provide the trans.txt file and at least one of the .tif files that you are transforming so I can try this and see if there is a memory leak somewhere? I don't see anything wrong with your script itself, but it doesn't sound like some memory isn't being cleaned up somewhere. Do you know if it has the same behavior without the transform .txt file?

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • PeterDrexelPeterDrexel Global Mapper User Posts: 41Trusted User
    edited July 2012
    Mike,
    you can find one Tif, a Script and the Transform-File here:

    ftp://ftp.vorarlberg.at/Peter/gm13.zip
    I think you can reproduce it by just copying the content to c:\temp and run the Script.
    The size of the Working Set is increasing after every transformation.
    Thanks
    Peter
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited July 2012
    Peter,

    I seem to need a username and password to download the file. You can send to gmsupport@bluemarblegeo.com if you don't want to paste it here.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited July 2012
    I have the files now and was able to track down the memory leak (it was indeed only if importing with a TRANSFORM_FILENAME parameter). I have placed a new build at http://www.globalmapper.com/global_mapper13.zip with the change for you to try. Simply download that file and extract the contents into your existing v13.xx installation folder to give it a try. If you are using the 64-bit v13 version there is a new build at http://www.globalmapper.com/global_mapper13_64bit.zip .

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • jeb76jeb76 Global Mapper User Posts: 5

    I am having a similar type issue running a script in GM 17, the longer it runs the more memory it consumes. Here is one line of the script and the first csv  file it creates.  On the csv file it is every 15 ft out 20 nmi (first column) the the second column it is the elevation at that point.  This is done every 0.05 degrees for 360 degrees.  If it cannot fixed in our current version hopefully it works in v19.  Thanks joel berry

    GENERATE_PATH_PROFILE FILENAME="D:\\Current Projects\\Atlantic City\\Script\\gm_Export_ACY_15ft_0.05deg_20nmi_0.csv" ELEV_UNITS=FEET POINT_COUNT=8101 ADD_LAND_USE_CODES=NO APPEND_TO_FILE=YES SAVE_DIST_Z_FILE=YES START_POS=-74.5919611111111,39.4527361111111 END_POS=-74.5919611111111,39.7863476612278

    0 107 15.003 71 30.005 75 45.008 71 60.011 70 75.014 70 90.016 70 105.02 70 120.02 70 135.02 70 150.03 70 165.03 70

Sign In or Register to comment.