Scripted URL problems

Ice Age MarkIce Age Mark Global Mapper UserPosts: 290Trusted User
edited April 2014 in GM Script Language
Howdy Mike,


I have about a dozen scripts I use that directly download weather charts from the internet, georeference them, and then project them onto the globe for viewing. I haven't used them for a while because winter ended early here and the weather has been rather boring since, but they have begun to not work well most of the time recently. The reasons I suspect GM are as follows:


The scripts work sometimes, but not most of the time.


When I test, say 20 times; first running the script I get a failed download perhaps 18 out of 20 times using the script, but when immediately copying the script generated URL directly to my browser to download it I get a 100% success rate. Also, when I directly paste the URL to GM as an "open file" I get 100% success.


The fail to download error returns on GM much faster than a browser or manual GM downloading, like instantaneously; both other successful methods, as well as regular browsing take at least several seconds to connect and download an image.


I'm thinking that maybe with all of the new great WMS changes that this channel of internet connection has gotten neglected. Would it be possible to check into this, and perhaps get GM to "try harder" (so to speak) or try more than once when connecting to a URL by script? Maybe this could be "smartened up" like the WMS stuff.


Thanks in advance,


Mark

V15.1.8 (B040414)

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited April 2014
    Mark,

    Can you provide one of those scripts that fails very quickly? There should be a lengthier timeout, but perhaps something else is happening.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • Ice Age MarkIce Age Mark Global Mapper User Posts: 290Trusted User
    edited April 2014
    Mike,

    This one fails to download most often, but still sometimes works. One odd thing I noticed is that for some reason PNG files are not included on the list of import file "types" in the scripting help documentation. Maybe it's just a typo that it's left out, but a lot of these are PNG files. I switched my scripts to "AUTO" for file type, but it didn't help.

    Here's the script: (I load them over another map, so no map layout, load projection or view bounds is used)



    GLOBAL_MAPPER_SCRIPT VERSION="1.00" FILENAME="C:\XXXX\XX\XXXX\RAP full.gmw"
    SET_BG_COLOR COLOR="RGB(0,0,0)"


    DEFINE_PROJ PROJ_NAME="LAMCC_WGS84"
    Projection LAMBERT
    Datum WGS84
    Zunits NO
    Units METERS
    Xshift 0.000000
    Yshift 0.000000
    Parameters
    1.000000000 /* scale factor at central meridian


    50 0 0.000 /* 1st standard parallel


    50 0 0.000 /* 2nd standard parallel


    -104 0 0.000 /* central meridian


    0 0 0.000 /* latitude of the origin


    0.000 /* false easting (meters)


    0.000 /* false northing (meters)


    END_DEFINE_PROJ


    DEFINE_SHAPE SHAPE_NAME="CUSTOM_CROP_POLY"
    -4967709.666,9691577.868
    4960432.346,9691577.868
    4960432.346,2257578.850
    -4967709.666,2257578.850
    -4967709.666,9691577.868
    END_DEFINE_SHAPE


    DEFINE_VAR NAME="DATE" VALUE="" \
    PROMPT="YES" PROMPT_TEXT="ENTER MONTH/DAY (MMDD)"


    DEFINE_VAR NAME="CYCLE" VALUE="" \
    PROMPT="YES" PROMPT_TEXT="ENTER CYCLE HOUR (CC)"


    DEFINE_VAR NAME="PARAMETER" VALUE="" \
    PROMPT="YES" PROMPT_TEXT="ENTER FORECAST PARAMETER (pppp)"


    DEFINE_VAR NAME="LEVEL" VALUE="" \
    PROMPT="YES" PROMPT_TEXT="ENTER LEVEL (LLL)"


    DEFINE_VAR NAME="FCSTHOUR" VALUE="" \
    PROMPT="YES" PROMPT_TEXT="ENTER FORECAST HOUR (FF)"












    IMPORT FILENAME="; \
    TYPE="AUTO" PROJ_NAME="LAMCC_WGS84" RECTIFY="Automatic" GCP="Point 1,438.2460342,260.5582482,113429.2234524,7343356.0015800" \
    GCP="Point 2,314.9020888,480.2529257,-1491955.7162598,4492119.0159385" GCP="Point 3,582.3320660,584.3874252,1988349.8864657,3033268.1471410" \
    GCP="Point 4,519.7549666,633.9015770,1135217.5104357,2350649.2918505" GCP="Point 5,710.4665198,284.7861768,3580209.6662310,7100724.6417813" \
    GCP="Point 6,313.5835379,149.6651207,-1480340.4893739,8830096.4333644" GCP="Point 7,601.0781683,476.5804443,2240254.7109323,4546216.7795181" \
    LAYER_DESC="%LEVEL% Level %PARAMETER% %FCSTHOUR%00 (%CYCLE%)" LABEL_FIELD_FORCE_OVERWRITE="NO" SAMPLING_METHOD="BICUBIC" AUTO_CONTRAST="NO" \
    CONTRAST_SHARED="YES" CONTRAST_MODE="NONE" CLIP_COLLAR="POLY" CLIP_COLLAR_POLY="CUSTOM_CROP_POLY" TEXTURE_MAP="NO"

    Thanks again,

    Mark
  • Ice Age MarkIce Age Mark Global Mapper User Posts: 290Trusted User
    edited April 2014
    I forgot that you might not know what variables to supply. Sorry about that. This URL should still be current when you test.

    http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040820/full/temp_850_f00.png
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited April 2014
    I'll give this a try in the morning and see what is going on.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • RobertRRobertR GlobalMapper Fan! Posts: 284Trusted User
    edited April 2014
    Hello,
    Nice topic! It would be nice in these cases to be able and store the failed responses (like everything <> than code 200) and to send those via email to the user. I use Python for this but it would boost up the functionality in GMScripting Envir. I understand that it may be hard to implement, but it is just a thought. For sure in this case it can be fixed other way, but what if you download data from a source that often changes it's link format? It would help there to know that the link changed or cannot be accessed.

    Cheers,
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited April 2014
    I've been trying this with the URL provided and can't get it to fail. If you run the script with File->Run Script do you see any additional error information when it fails?

    I have updated the download to actually add detailed errors to the script log or report them at the end of the load rather than just saying that the download failed. I have placed a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v15.xx installation folder to give it a try. If you are using the 64-bit v15 version there is a new build at http://www.bluemarblegeo.com/downloads/global-mapper/global_mapper15_64bit.zip .

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • Ice Age MarkIce Age Mark Global Mapper User Posts: 290Trusted User
    edited April 2014
    Mike,

    Here are the error codes returned. The URL itself works outside of the script. Is there something wrong with the way I've set up the URL with variables? It seems to work with other ones. (No XML(?) info should be associated with this, it's just a png image.)

    Error downloading file C:\Users\MP\AppData\Local\Temp\GlobalMapper\downloads\rapidrefresh_noaa_gov_RAP_for_web_rap_jet_2014040913_full_wmag_850_f00.png from URL http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png
    Download of http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png failed with status code 404.


    <?xml version="1.0" encoding="ISO-8859-1"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">;
    <html xmlns="http://www.w3.org/1999/xhtml"; lang="en" xml:lang="en">
    <head>
    <title>Object not found!</title>
    <link rev="made" href="mailto:webmaster-amb.gsd@noaa.gov" />
    <style type="text/css"><!--/*--><![CDATA[/*><!--*/
    body { color: #000000; background-color: #FFFFFF; }
    a:link { color: #0000CC; }
    p, address {margin-left: 3em;}
    span {font-size: smaller;}
    /*]]>*/--></style>
    </head>


    <body>


    WebUtils.cpp - 1198
    Version: v15.1.9 (64-bit)
    Build Time: Apr 9 2014 08:34:18

    Thanks much,

    Mark
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited April 2014
    Mark,

    I'm getting an error trying to load that image in a web browser. The link is:

    http://rapidrefresh.noaa.gov/RAP/for_web/rap_jet/2014040913/full/wmag_850_f00.png

    I get basically the same text that you see in the error log in Firefox.

    Thanks,

    Mike
    Global Mapper Guru
    geohelp@bluemarblegeo.com
    Blue Marble Geographics for Coordinate Conversion, Image Reprojection and Vector Translation
  • Ice Age MarkIce Age Mark Global Mapper User Posts: 290Trusted User
    edited April 2014
    Mike,


    This mystery (in my feeble brain) and the seeming randomness is solved, and GM is not the problem. I finally heard back from ESRL and it seems that the images from some model sources are generated "on the fly" when requested to conserve processing. This was the randomness that was driving me crazy; I would get a fail, check the site to be sure it was there, that would generate the image and it would work. Then I would try another image and get a fail. I was fooling myself and going in circles. Geez!


    Anyway, the error logging you provided was the key to figuring this out, so I thank you very much, and am sorry to have sent you on a wild goose chase.


    Many thanks again for your always polite and patient help,


    Mark
Sign In or Register to comment.