Scripted URL problems
Ice Age Mark
Global Mapper UserTrusted User
Howdy Mike,
I have about a dozen scripts I use that directly download weather charts from the internet, georeference them, and then project them onto the globe for viewing. I haven't used them for a while because winter ended early here and the weather has been rather boring since, but they have begun to not work well most of the time recently. The reasons I suspect GM are as follows:
The scripts work sometimes, but not most of the time.
When I test, say 20 times; first running the script I get a failed download perhaps 18 out of 20 times using the script, but when immediately copying the script generated URL directly to my browser to download it I get a 100% success rate. Also, when I directly paste the URL to GM as an "open file" I get 100% success.
The fail to download error returns on GM much faster than a browser or manual GM downloading, like instantaneously; both other successful methods, as well as regular browsing take at least several seconds to connect and download an image.
I'm thinking that maybe with all of the new great WMS changes that this channel of internet connection has gotten neglected. Would it be possible to check into this, and perhaps get GM to "try harder" (so to speak) or try more than once when connecting to a URL by script? Maybe this could be "smartened up" like the WMS stuff.
Thanks in advance,
Mark
V15.1.8 (B040414)
I have about a dozen scripts I use that directly download weather charts from the internet, georeference them, and then project them onto the globe for viewing. I haven't used them for a while because winter ended early here and the weather has been rather boring since, but they have begun to not work well most of the time recently. The reasons I suspect GM are as follows:
The scripts work sometimes, but not most of the time.
When I test, say 20 times; first running the script I get a failed download perhaps 18 out of 20 times using the script, but when immediately copying the script generated URL directly to my browser to download it I get a 100% success rate. Also, when I directly paste the URL to GM as an "open file" I get 100% success.
The fail to download error returns on GM much faster than a browser or manual GM downloading, like instantaneously; both other successful methods, as well as regular browsing take at least several seconds to connect and download an image.
I'm thinking that maybe with all of the new great WMS changes that this channel of internet connection has gotten neglected. Would it be possible to check into this, and perhaps get GM to "try harder" (so to speak) or try more than once when connecting to a URL by script? Maybe this could be "smartened up" like the WMS stuff.
Thanks in advance,
Mark
V15.1.8 (B040414)
Comments
-
Mark,
Can you provide one of those scripts that fails very quickly? There should be a lengthier timeout, but perhaps something else is happening.
Thanks,
Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
http://www.bluemarblegeo.com/ -
Mike,
This one fails to download most often, but still sometimes works. One odd thing I noticed is that for some reason PNG files are not included on the list of import file "types" in the scripting help documentation. Maybe it's just a typo that it's left out, but a lot of these are PNG files. I switched my scripts to "AUTO" for file type, but it didn't help.
Here's the script: (I load them over another map, so no map layout, load projection or view bounds is used)
GLOBAL_MAPPER_SCRIPT VERSION="1.00" FILENAME="C:\XXXX\XX\XXXX\RAP full.gmw"
SET_BG_COLOR COLOR="RGB(0,0,0)"
DEFINE_PROJ PROJ_NAME="LAMCC_WGS84"
Projection LAMBERT
Datum WGS84
Zunits NO
Units METERS
Xshift 0.000000
Yshift 0.000000
Parameters
1.000000000 /* scale factor at central meridian
50 0 0.000 /* 1st standard parallel
50 0 0.000 /* 2nd standard parallel
-104 0 0.000 /* central meridian
0 0 0.000 /* latitude of the origin
0.000 /* false easting (meters)
0.000 /* false northing (meters)
END_DEFINE_PROJ
DEFINE_SHAPE SHAPE_NAME="CUSTOM_CROP_POLY"
-4967709.666,9691577.868
4960432.346,9691577.868
4960432.346,2257578.850
-4967709.666,2257578.850
-4967709.666,9691577.868
END_DEFINE_SHAPE
DEFINE_VAR NAME="DATE" VALUE="" \
PROMPT="YES" PROMPT_TEXT="ENTER MONTH/DAY (MMDD)"
DEFINE_VAR NAME="CYCLE" VALUE="" \
PROMPT="YES" PROMPT_TEXT="ENTER CYCLE HOUR (CC)"
DEFINE_VAR NAME="PARAMETER" VALUE="" \
PROMPT="YES" PROMPT_TEXT="ENTER FORECAST PARAMETER (pppp)"
DEFINE_VAR NAME="LEVEL" VALUE="" \
PROMPT="YES" PROMPT_TEXT="ENTER LEVEL (LLL)"
DEFINE_VAR NAME="FCSTHOUR" VALUE="" \
PROMPT="YES" PROMPT_TEXT="ENTER FORECAST HOUR (FF)"
IMPORT FILENAME="; \
TYPE="AUTO" PROJ_NAME="LAMCC_WGS84" RECTIFY="Automatic" GCP="Point 1,438.2460342,260.5582482,113429.2234524,7343356.0015800" \
GCP="Point 2,314.9020888,480.2529257,-1491955.7162598,4492119.0159385" GCP="Point 3,582.3320660,584.3874252,1988349.8864657,3033268.1471410" \
GCP="Point 4,519.7549666,633.9015770,1135217.5104357,2350649.2918505" GCP="Point 5,710.4665198,284.7861768,3580209.6662310,7100724.6417813" \
GCP="Point 6,313.5835379,149.6651207,-1480340.4893739,8830096.4333644" GCP="Point 7,601.0781683,476.5804443,2240254.7109323,4546216.7795181" \
LAYER_DESC="%LEVEL% Level %PARAMETER% %FCSTHOUR%00 (%CYCLE%)" LABEL_FIELD_FORCE_OVERWRITE="NO" SAMPLING_METHOD="BICUBIC" AUTO_CONTRAST="NO" \
CONTRAST_SHARED="YES" CONTRAST_MODE="NONE" CLIP_COLLAR="POLY" CLIP_COLLAR_POLY="CUSTOM_CROP_POLY" TEXTURE_MAP="NO"
Thanks again,
Mark -
I forgot that you might not know what variables to supply. Sorry about that. This URL should still be current when you test.
http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040820/full/temp_850_f00.png -
I'll give this a try in the morning and see what is going on.
Thanks,
Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation -
Hello,
Nice topic! It would be nice in these cases to be able and store the failed responses (like everything <> than code 200) and to send those via email to the user. I use Python for this but it would boost up the functionality in GMScripting Envir. I understand that it may be hard to implement, but it is just a thought. For sure in this case it can be fixed other way, but what if you download data from a source that often changes it's link format? It would help there to know that the link changed or cannot be accessed.
Cheers, -
I've been trying this with the URL provided and can't get it to fail. If you run the script with File->Run Script do you see any additional error information when it fails?
I have updated the download to actually add detailed errors to the script log or report them at the end of the load rather than just saying that the download failed. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .
Thanks,
Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation -
Mike,
Here are the error codes returned. The URL itself works outside of the script. Is there something wrong with the way I've set up the URL with variables? It seems to work with other ones. (No XML(?) info should be associated with this, it's just a png image.)
Error downloading file C:\Users\MP\AppData\Local\Temp\GlobalMapper\downloads\rapidrefresh_noaa_gov_RAP_for_web_rap_jet_2014040913_full_wmag_850_f00.png from URL http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png
Download of http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png failed with status code 404.
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Object not found!</title>
<link rev="made" href="mailto:webmaster-amb.gsd@noaa.gov" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
body { color: #000000; background-color: #FFFFFF; }
a:link { color: #0000CC; }
p, address {margin-left: 3em;}
span {font-size: smaller;}
/*]]>*/--></style>
</head>
<body>
WebUtils.cpp - 1198
Version: v15.1.9 (64-bit)
Build Time: Apr 9 2014 08:34:18
Thanks much,
Mark -
Mark,
I'm getting an error trying to load that image in a web browser. The link is:
http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png
I get basically the same text that you see in the error log in Firefox.
Thanks,
Mike
Global Mapper Guru
geohelp@bluemarblegeo.com
Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation -
Mike,
This mystery (in my feeble brain) and the seeming randomness is solved, and GM is not the problem. I finally heard back from ESRL and it seems that the images from some model sources are generated "on the fly" when requested to conserve processing. This was the randomness that was driving me crazy; I would get a fail, check the site to be sure it was there, that would generate the image and it would work. Then I would try another image and get a fail. I was fooling myself and going in circles. Geez!
Anyway, the error logging you provided was the key to figuring this out, so I thank you very much, and am sorry to have sent you on a wild goose chase.
Many thanks again for your always polite and patient help,
Mark
Categories
- 12.7K All Categories
- 5.6K Features Discussion
- 342 Downloading Imagery
- 1.3K Elevation Data
- 380 Georeferencing Imagery Discussion
- 628 GM Script Language
- 53 User Scripts
- 113 GPS Features
- 414 Projection Questions
- 819 Raster Data
- 1.3K Vector Data
- 6.6K Support
- 177 Announcement and News
- 908 Bug Report
- 558 SDK
- 1.2K Suggestion Box
- 3.7K Technical Support
- 562 Other Discussion
- 129 GIS Data Sources
- 27 Global Mapper Showcase
- 233 How I use Global Mapper
- 107 Global Mapper Forum Website