Downloading imagery from behind a firewall

VikVik Global Mapper UserPosts: 14
edited August 2012 in Downloading Imagery
Good afternoon,

I'm having issues with a copy of v12.02 (64-bit) trying to download NAIP imagery from behind a network at my workplace that's a little more locked-down than the primary network. The issue is that, while I can use the primary network machine just fine and get the imagery that way, I need to run an export operation on it to convert it to Jpg2000 format and this machine does not have nearly enough horsepower to get it done without processing for 4-5 days straight.

My other machine, on the locked-down network, has the necessary horsepower to get it done in probably half the time, but the imagery download isn't working. Here are the steps to replicate my issue:

1) Open app. Click on "Download Free Maps/Imagery from Online Sources".
2) Under "popular sources" click "NAIP Color Imagery for US (1m resolution)"
3) Click on "Specify Latitude/Longitude Bounds of Area". My exact coordinates probably are not necessary, but I chose 37.5 and 36 degrees N, along with -77 and -75.5 degrees W as my bounds.
4) Click "Ok"

I get a popup saying "Unknown error getting WMS server capabilities" (I didn't get any hits searching for that message on the forums).

Again, this works just fine on my main network, so I think the issue might be firewall rules in the other network. Could you let me know what kind of network traffic I can ask my network security folks to allow me to let in/out in order to get this functionality working? I have WWW access on the machine, so I know my current proxy settings are enough to let in normal htttp traffic over port 80. I checked in the manual, as well as in the UI, for specific proxy settings that Global Mapper might be using, but I didn't find any, so my assumption is that Global Mapper is using the proxy settings I currently have setup in Internet Explorer (these are the same settings which are allowing http traffic in what is otherwise a generally locked-down network).

Any insight would be of great help to me.

Thanks for reading,
Vik

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vik,

    Port 80 is all that you should need open for that source. Which v12.02 build do you have? If not the absolute latest I would first get that. I have placed a new build at http://www.globalmapper.com/global_mapper12.zip with the latest changes for you to try. Simply download that file and extract the contents into your existing v12.xx installation folder to give it a try. If you are using the 64-bit v12 version there is a new build at http://www.globalmapper.com/global_mapper12_64bit.zip .

    There have also been a lot of enhancements to accessing online sources like that since the v12.02 release, in particular in regards to handling unreliable servers like the NAIP server so you can still get a seamless output. I would definitely recommend upgrading to v13 and using that rather than v12 if at all possible.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike,

    I had been running an old build from May 2011. I unzipped the 64 bit zip file, backed up my old executable, and copied the new files into my installation folder. Everything works great now; the imagery downloaded from the locked-down network without a hitch.

    Thank you so much for the quick support.

    Best,
    Vik
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike,

    Now that I have my imagery working properly on the network, I saw that the work was going to take more time than I wanted it to (an estimated completion time of next Tuesday for the export). So, a few tiles of processing into it, I decided to split my "export to JP2 task" in half, by dividing the geographic area by lat/lon extents, and doing half of the export on each machine. I'm sort of puzzled, because the estimated time of completion is only an hour or so different with the data halved. You'd think, intuitively, it would estimate that it would be finished in half the time of the previous estimate. The size of the tiles (I used 10 rows and columns both before I split the task, as well as after) I'm getting is roughly half of what they were before, so it seems like the program is taking just as long to complete half the processing work. If it helps, the CPU isn't close to being taxed (roughly at 10% utilization).

    Also, generally, this export seems to be taking much longer than I'm used to. Is that a side effect of having the imagery pulled down from an online source inside of Global Mapper rather than having all of my raw mrsid data loaded locally, as I have done in the past?

    Any speed tips you could give me would be appreciated.

    Thanks,
    Vik
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vik,

    With a download from an online source nearly all of the time is just waiting for the server to provide the imagery, not much CPU is needed. It will also be much slower than export from a local source. Time estimates will also vary a lot, especially at the start when the estimate doesn't have much history to go on. The estimate depends almost entirely on how fast the server is providing the imagery, which often depends mostly on the load on the server and to a lesser degree the quality of the connection between your machine and the server. The NAIP server is notoriously slow and unreliable compared to others. You might instead export from the World Imagery source as it has NAIP imagery in the US.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike,

    I took your suggestion and tried the World Imagery source. This looks to be quite a nice source of imagery and has the oceanic imagery that the NAIP source was missing. I appreciate the tip.

    I took the same lat/lon extents I listed in my original post and used them to download from the World Imagery source and kicked off an export of that imagery to JP2. It's going much faster than the last process, about twice as fast. My issue is that, when I loaded the first few tiles into STK the bottom half of each tile was mostly blank, gray in color with the message "Map data not yet available" repeated over and over again. The weird thing is that I didn't see this blank data in the original imagery from the server that I was viewing in Global Mapper.

    The other image server was slow, and it didn't have the Chesapeake Bay imagery I wanted, but it didn't have any of these half-filled tiles. Are there any tips you could give me on something that I might be doing wrong? Since the soruce imagery was in a weird Mercator coordinate system I reprojected to Geographic WGS84 before I started the export--could that be a source of problems?

    Thanks,
    Vik
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Quick clarification. I didn't use those exact coordinates for this last run, I had split it up into a smaller chunk. I was using -77 to -75.5 W like in the original post but the northern and southern boundaries were 36.5 to 36 N.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    What resolution are you exporting at? The World Imagery source does have different resolutions of imagery available in different locations and in those areas where you are accessing at a resolution beyond what the source has you may see that 'Data Not Yet Available' graphic. However entire tiles filled with that are detected and replaced with lower resolution tiles of actual data, so this should only be an issue at the border of differing resolutions. If you are exporting at 1m resolution I would expect there to be coverage everywhere though, the problems should only pop up in the US at higher zoom levels.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    I think this is definitely User error. I wasn't aware there was an option, while exporting, to specify the resolution. I would like to export at a fidelity of 1 meter, but I'm not sure how to do that. I've spent the last half hour going through all of the export settings where JP2 is the selected format, and I've tried to search every page of the PDF manual that has the word "resolution" in it, but I'm not sure where to specify what my export resolution should be.

    Is this an option I have to select before the online imagery is pulled down, or is this something I have to do during the export process? Apologies for my unfamiliarity with this aspect of the application.

    Thanks,
    Vik
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vik,

    When you export the 'Sample Spacing' setting is the resolution. Just set that to 1 meter in each direction and see if that does the trick. This also might speed things up quite a bit too if you had been exporting at a higher resolution before.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike, I think you're on the money. For whatever reason, my sample spacing had defaulted to .59 meters along the x and y axes. I'll tweak that to 1 in each direction and try again this morning. Thank you for the excellent support!
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike, I do have one more question regarding the resolution during export. The world imagery comes down in the Mercator projection. I don't want that coordinate system so, before exporting into JP2, I reproject into "Geographic". I click on "Click here to calculate spacing in other units" so that I can specify 1 meter by 1 meter resolution. I get through this step, but the sample spacing in arc degrees of the x and y axes is not the same value, which makes me think the conversion of 1m by 1m did not calculate as I had hoped. If I go back to the other units calculations page and select "meters" from the pulldown, it retrieves the y-axis value as 1m,but the x-axis is always listed as 0.806448642359564m.

    Is there any way to force a resolution specification of 1m by 1m? If I go finer than 1m along the x axis then I fear I will run into the same "Data Not Yet Available" graphic I'd seen before.

    Thanks,
    Vik
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vik,

    The issue is that there isn't a linear conversion between arc degrees (angular units) and linear units, like meters. The number of degrees per meter depends heavily on your location. The conversion used when you specify your units in meters when exporting in degrees is done at the center of your loaded data. What you could do if you want to be absolutely certain is export in Mercators at 1 meter resolution (this again is a bit misleading as 1 Mercator "meter" is not the same as a ground "meter", no projected distance is since the curved surface of the earth can't be represented with no distortion on a 2D map projection, but you can get close for small areas), then load that back in and reproject to Geographic and let the default resolution be used. By then you will be done with the online source so selecting a layer from there won't matter.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
  • VikVik Global Mapper User Posts: 14
    edited August 2012
    Mike, I was afraid the curvature of the Earth was working to sabotage me on this . . .


    I had sort of settled on the fact that I might have to do a Mercator projection export first and from there do another export to Geographic. Was hoping to avoid adding another export to the process, but this way might be safest way to ensure I have the resolution I want. If I do an export twice in JP2 with compression enabled each time, then I'd be losing image fidelity each time: is that correct? If so, then the way to ensure that I'm only losing image fidelity once would be to export into Mercator with 1m resolution and no JP2 compression (lossless) and then add the compression level I'm comfortable with to the second export, when I project into Geographic.

    Does that, offhand, sound correct to you?

    Thanks again,
    Vik
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited August 2012
    Vik,

    That would work, or you could export to another lossless format to start with (like GeoTIFF), then compress into JP2. Really doesn't matter much which way though.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.globalmapper.com
Sign In or Register to comment.