Script and Page File usage

veddycent

Hi all,

I have a script that loops through a directory of .asc files.
- Loads the .asc
- Gets asc X,Y max and min positions and store in 4 variables
- Loops hierarchy of shp files
- Gets shp X,Y max and min positions and store in 4 variables
- Test to see if they overlap (simple if comparison)
- If asc does not overlap any shp then unload

10,000 asc files and 1,500 shp files.

This scripts maxes out page file usage (~40GB) freezes then eventually closes.

This happens before it even finds any over lapping data.
Could it be that the page file is not being unloaded for the files that are unloaded?

GM Version: 19.0.2 (b112717)(1)(64-bit)
32GB Ram

bmg_bob

Hello,

This is a guess, since you did not post the actual script, but It sounds like you may not be unloading the .shp files when you are done with them. For each .shp file, you should probably import it, test for overlay, then unload it.

Cheers,
Bob

veddycent

Hi Bob,

Sorry for the late reply.

The shp files only contain 1 square area so they don't use not much memory, definitely not that much any way.

Here is the script:

GLOBAL_MAPPER_SCRIPT VERSION="1.00"

DEFINE_VAR NAME="FILE_FORMAT_TERRAIN" VALUE="asc" PROMPT=YES PROMPT_TEXT="Terrain File Type (e.g. asc, tif, png, etc):"

DEFINE_VAR NAME="FILES_TERRAINS" PROMPT=DIR ABORT_ON_CANCEL=YES

DIR_LOOP_START DIRECTORY="%FILES_TERRAINS%" FILENAME_MASKS="*.%FILE_FORMAT_TERRAIN%" RECURSE_DIR=YES

IMPORT FILENAME="%FNAME_W_DIR%"

//Gets the X Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="UPPER LEFT X" RESULT_VAR="TERRAIN_XMin"

//Gets the X Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="LOWER RIGHT X" RESULT_VAR="TERRAIN_XMax"

//Gets the Y Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="LOWER RIGHT Y" RESULT_VAR="TERRAIN_YMin"

//Gets the Y Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="UPPER LEFT Y" RESULT_VAR="TERRAIN_YMax"

//File Description

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="DESCRIPTION" RESULT_VAR="TERRAIN_FileDesc"

LAYER_LOOP_START FILENAME="*.shp"

//Gets the X Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="UPPER LEFT X" RESULT_VAR="BUFFER_XMin"

//Gets the X Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="LOWER RIGHT X" RESULT_VAR="BUFFER_XMax"

//Gets the Y Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="LOWER RIGHT Y" RESULT_VAR="BUFFER_YMin"

//Gets the Y Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="UPPER LEFT Y" RESULT_VAR="BUFFER_YMax"

IF COMPARE_STR="%TERRAIN_XMax%<=%BUFFER_XMin%"

UNLOAD_LAYER FILENAME="%TERRAIN_FileDesc%" COMPARE_NUM="YES"

ELSE_IF COMPARE_STR="%TERRAIN_XMin%>=%BUFFER_XMax%"

UNLOAD_LAYER FILENAME="%TERRAIN_FileDesc%" COMPARE_NUM="YES"

ELSE_IF COMPARE_STR="%TERRAIN_YMin%>=%BUFFER_YMax%"

UNLOAD_LAYER FILENAME="%TERRAIN_FileDesc%" COMPARE_NUM="YES"

ELSE_IF COMPARE_STR="%TERRAIN_YMax%<=%BUFFER_YMin%"

UNLOAD_LAYER FILENAME="%TERRAIN_FileDesc%" COMPARE_NUM="YES"

END_IF

LAYER_LOOP_END

DIR_LOOP_END

bmg_bob

Hello,

LAYER_LOOP_START works on loaded layers, and the script does not appear to be loading any .shp files. As a result, the layer loop gets skipped (no *.shp files found) and none of your 10,000 .asc files ever gets unloaded. 

There are a couple of options:

1. Change your LAYER_LOOP_START to a DIR_LOOP_START. Inside the directory loop, IMPORT the .shp file, perform your comparisons, then unload the .shp file.
2. If you think your computer has enough memory to have all of your .shp files loaded at once, put a dir loop that loads all of your .shp files before the existing DIR_LOOP_START. Then your LAYER_LOOP_START will have some data to loop through.

The first option will use less memory. The second option will be faster, since each .shp file only gets loaded once.

Cheers,
Bob

veddycent

Hi Bob,

I don't think the issue is with the loaded shp files but something else.

I have the GM file with all shp files open, in view and this are the current stats for memory usage:
Commit (MB) = 87
Working Set (MB) = 355
Shareable (MB) = 273
Private (MB) = 81

As you can see RAM usage is 81MB and page file is 87MB.
So it must be the script but I can't understand why because when only one ASC file is loaded and the loop is checking if it overlaps with the SHP files that's when the page file usage rockets.

After it has looped through the layers once the page file usage is up to 300MB even though only 1 file has been loaded.

I'm wondering if it keeps constantly creating new variables as it iterates through the layers, even though the variable uses the same name?

GM is really good with memory management which is why I'm confused about this one.

NOTE: The script I posted above had an error. It would unload the ASC after checking with the first SHP instead of unloading/keeping after the full LAYER_LOOP has finished. I have added the corrected version below:



Another example, I have a script that exports raster files (2048 x 2048 px) of a shape file area of the UK which is solid white with a black background. It loops a directory (10,000 files) and uses a shape file (contains a simple square area) as the crop area. The export starts of great but after about 15% of the way through it slows down that much that it takes 5 mins to export each raster. There are no variables used in this script.

*
*
*
*

DEFINE_VAR NAME="FILES_TERRAINS" PROMPT=DIR ABORT_ON_CANCEL=YES

DEFINE_VAR NAME="KEEP_FILE" VALUE="YES"

DEFINE_VAR NAME="KEEP_FILE_OVERALL" VALUE="NO"

DIR_LOOP_START DIRECTORY="%FILES_TERRAINS%" FILENAME_MASKS="*.%FILE_FORMAT_TERRAIN%" RECURSE_DIR=YES

IMPORT FILENAME="%FNAME_W_DIR%"

//Gets the X Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="UPPER LEFT X" RESULT_VAR="TERRAIN_XMin"

//Gets the X Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="LOWER RIGHT X" RESULT_VAR="TERRAIN_XMax"

//Gets the Y Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="LOWER RIGHT Y" RESULT_VAR="TERRAIN_YMin"

//Gets the Y Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="UPPER LEFT Y" RESULT_VAR="TERRAIN_YMax"

//File Description

QUERY_LAYER_METADATA METADATA_LAYER="%FNAME_W_DIR%" METADATA_ATTR="DESCRIPTION" RESULT_VAR="TERRAIN_FileDesc"

LAYER_LOOP_START FILENAME="*.shp"

//Gets the X Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="UPPER LEFT X" RESULT_VAR="BUFFER_XMin"

//Gets the X Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="LOWER RIGHT X" RESULT_VAR="BUFFER_XMax"

//Gets the Y Min from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="LOWER RIGHT Y" RESULT_VAR="BUFFER_YMin"

//Gets the Y Max from Metadata for the layer

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="UPPER LEFT Y" RESULT_VAR="BUFFER_YMax"

//File Description

QUERY_LAYER_METADATA METADATA_LAYER="%LAYER_DESC%" METADATA_ATTR="DESCRIPTION" RESULT_VAR="BUFFER_FileDesc"

IF COMPARE_STR="%TERRAIN_XMax%<=%BUFFER_XMin%"

DEFINE_VAR NAME="KEEP_FILE" VALUE="NO"

ELSE_IF COMPARE_STR="%TERRAIN_XMin%>=%BUFFER_XMax%"

DEFINE_VAR NAME="KEEP_FILE" VALUE="NO"

ELSE_IF COMPARE_STR="%TERRAIN_YMin%>=%BUFFER_YMax%"

DEFINE_VAR NAME="KEEP_FILE" VALUE="NO"

ELSE_IF COMPARE_STR="%TERRAIN_YMax%<=%BUFFER_YMin%"

DEFINE_VAR NAME="KEEP_FILE" VALUE="NO"

END_IF

IF COMPARE_STR="%KEEP_FILE_OVERALL%=NO"

DEFINE_VAR NAME="KEEP_FILE_OVERALL" VALUE="%KEEP_FILE%"

END_IF

LAYER_LOOP_END

IF COMPARE_STR="%KEEP_FILE_OVERALL%=NO"

UNLOAD_LAYER FILENAME="%TERRAIN_FileDesc%" COMPARE_NUM="YES"

END_IF

DIR_LOOP_END

bmg_bob

Hello Ved,

This is my understanding of what you are doing:

1. Load all of your .shp files in the UI
2. Run the script with the "Run Script in the Context of the Main View" option checked.
3. Your directory of ASC files has only 1 file (for the purposes of this test)
4. When the loop begins comparing the metadata values, the memory usage skyrockets.

Is this correct? I will look into the scripting engine to see if there is something amiss.

One thing that I should have mentioned earlier: You are using 19.0.2 from November of 2017. The latest version of Global Mapper 19 is dated September 19, 2018.  It might be helpful to download the latest release.

Cheers,
Bob

veddycent

Hi Bob,

Yes that is the process.

I have updated to the latest version since the original post. v19.1.1 (b091718)(680)(64-bit)

Regards
Ved

veddycent

I have uploaded a sample file and script which loops the layers, gets some meta data and does a few IF's
No data is loaded or unloaded and you can see the memory usage increase.

Hopefully this will be useful for any testing.

Regards
Ved