Global Mapper v25.0

Rebuilt missing data in SRTM3 using alternative source (e.g. NED1)

dpluigi
dpluigi Global Mapper User
edited March 2009 in Elevation Data
Hi,

I am new to the forum and GM. I am working on a DEM terrain data for Southern Nevada, including a portion of the Grand Canyon (Arizona side).

The SRTM3 (92m) Version 2 elevation data saved in BIL/HGT format and downloaded from NASA ftp site is missing the canyon walls and bottom. ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SRTM3/ .

Note that the CGIAR-CSI SRTM 3-arc second (92m) DEM version 4 data product is addressing the no-data regions/missing data issues with a hole-filling algorithm used to provide continuous elevational surfaces for SRTM3 data around the globe. Although it is an improvement, it seems to have lost some of the details that are present in the NASA SRTM, probably due to smoothing after interpolations.
Also I observed a slight shift when flipping between the NSAS and CGIAR corrected SRTM3 data. If the coordinate system is placed at bottom left in GlobalMapper display the CGIAR moves away from NASA SRTM3 by (1/2, 1/2) pixels (i.e. zooming at pixels level). Anyone else observed this?

Anyhow, it seems that if another alternative DEM source with similar or better resolution than SRTM3 can used then, the Grand Canyon missing data could be merge back from that source into teh SRTM DEM.

I am wondering if it is acceptable to use a combinatorial approach to recovering SRTM DEM data using NED (1-arc second) from NSGS Seamless Server?

Is there a recommend approach to preserve the most of the original SRTM?

The approach I used is to combine layers with "minimum" operation, choosing the SRTM3 (92m) as layer1, and the NED (30m) as layer 2. Since I need to generate an combination consistent to SRTM3 (92m) I used sampling steps along x and y of 0.00083333333333. I am hoping that this way only the missing data is replaced with the corresponding data points from the NED DEM file. Is this correct and a valid approach?

I also tried to use "filter" operation, but I am not sure I understand how it replaces a value. I thought that for not-data point, which elevatino is not valid, it would replaced. I assumed it to be a boolean approach and even used Alter Elevation value for that.

I would appreciate any suggestion in using NED to rebuild missing data in an SRTM3. Also, are they better alternative DEM data to use with an SRTM3?

Thank you in advance for your answers,
Donat

Comments

  • global_mapper
    global_mapper Administrator
    edited March 2009
    Donat,

    For the Grand Canyon I think the Combine Terrain with a 'minimum' combination would probably work. Unless you are really attached to the SRTM data I would just patch in the NED data and then use feather blending via the Options dialog for the NED layer to smoothly blend it into the SRTM data so you get a seamless transition between the NED and the SRTM data.

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    Hi Mike,

    Thanks for the quick reply. Regarding my SRTM data, I consider it the Base DEM and would like to preserve it. That is because I read it is slightly more accurate than NED. Regarding the difference approach, is using sampling steps along x and y of 0.000833333 deg enough to make sure than the combined image has the same resolution and is formatted like any SRTM (92m)?

    Regarding the Feather Blend approach, I just tried it and I don't think I understand. I used the Feather from the Option menu displayed in Control Center (Alt-C). The Base layer (SRTM 92m file) as File 1 and the second layer (NED 30m file) as File 2. In the order of file in the list of Control Center File 1 is above File 2. Now the idea is to blend feather File 2 onto File 2 only for missing data in File 1. With both files selected/checked, I highlighter File 2 in Control Center, clicked Option>Feathering, and chose feather along one or more edge... For File 1, i left the option "Don't feather this layer".
    What am I doing wrong, the view of both layer is not changed... is there a resulting image or should I use Export Raster/ELevation data?

    Could you explain how I should proceed to "patch in the NED data and then use feather blending via the Options dialog for the NED layer to smoothly blend it into the SRTM data"?

    Regarding the CGIAR SRTM (92m) DEM data, which were corrected and used interpolation and auxiliary images, I feel the details on the Grand Canyon walls are smoother/less detailed than when Combining "Difference" NED into SRTM. This is why I'd like to use NED (30m) as auxiliary DEM to rebuild void regions in my SRTM DEM file.

    Below are the screenshots of a small region of the Grand Canyon illustrating the results of Combining Layer using "Minimum" operation...

    Thank you for your help. Very much appreciated.
    Donat

    Original SRTM3 version 2 DEM Data BIL format file downloaded from NASA ftp site. It is missing the Grand Canyon walls and bottom as well as steep slopes, radar shadow on southern sides of relief...
    srtm3gcoriginal1.th.jpg

    The National Atlas NED (30m) DEM used to combine with NASA SRTM to recover missing data.
    srtm3gcned11.th.jpg

    Combining the NASA SRTM version 2 (92m) DEM with National Atlas NED (30m) to recover missing data. The Combine Layer operation is GlobalMapper was "Minimum".
    srtm3gccombned11.th.jpg

    SRTM3 version 4 DEM Data BIL format file downloaded from CGIAR-CSI, corrected data. It has all the Grand Canyon, but the resolution of the recovered data is lower/smoother than when using NED auxiliary DEM...
    srtm3gccgsiar1.th.jpg
  • global_mapper
    global_mapper Administrator
    edited March 2009
    For the feather blending you need to have the SRTM layer loaded first (as File 1), then the NED data loaded after that (the NED data will be below the SRTM layer in the Control Center), so that the NED layer overlays the SRTM data. Then, select the NED layer, press Options, then go to the Feathering tab and select the option to 'Feather Along One or More Edges'. Also make sure that the 'Border Size in Pixels' value makes sense. This will control the size of the feather blend from the edge of the NED layer in terms of the native NED layer sample size (i.e. 1 arc second or 1/3 arc second).

    I'm guessing that the SRTM and NED data are close enough in value that there will be very little visible difference, but the feather-blending will force a smooth transition between any differences that are there.

    Regarding the SRTM data being less accurate that the NED, that is not something that I have ever heard. I would expect the NED to be even better if you are using the 1/3 arc second (10 meter) NED stuff that is available for most of the country (note sure if it's there for the Grand Canyon, but I would expect it to be).

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    I will try this thank you. Another question, if I wanted to use NSGS' NED, my application would require the elevations to be in reference to the ellipsoid, WSG84 and project in UTM for my project. (See National Elevation Dataset details on the datum used for NED)

    DO you know if that is easily done within GlobalMapper or will I need to carry conversion with another program (some are mentioned in the link about at USGS)?

    It is technical question I don't fully grasp and I apologize if my question is not formulated well.

    Thanks,
    Donat
  • global_mapper
    global_mapper Administrator
    edited March 2009
    Donat,

    It is a simple matter to reproject data in Global Mapper to UTM, you just select UTM on the Projection tab of the Configuration dialog.

    Changing your elevations to be in reference to a different vertical datum is not supported automatically in Global Mapper. You can manually adjust all elevations in a layer by some amount which works fine for vertical datum adjustments over a limited area, but obviously is not practical for data covering a large area as the adjustment would not be constant.

    The NED data is in the NAVD88 vertical datum. I'm honestly not familiar enough with the typical differences between various vertical reference systems to know if that is significantly different than elevations referenced to the WGS84 ellipsoid.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    Hi,

    I just found great explanations about comparing NED and SRTM (NASA) DEMs there:
    Significant Topographic Changes in the United States

    If I understand correctly DEM, GeoData expressed in different datum can be compared once they have either been converted to a common datum (i.e. NAD88, which is common to a lot of dataset in the US) or they are projected in UTM, i.e. a common represenation sapce/common ground?

    The link above, mention that the different vertical datum used to reference SRTM (EGM96 geoid) and NED (NAD88, sea level referenced) are essentially the same and require no transformation. (Which settles what I was asking in another post).

    As I discussed in another thread, there is a 1/2 pixel grid shift for the SRTM file for the CGIAR SRTMs and also the NASA one. I can use the Shift a Specified Layer(s) (by) a Specified Distance/Offset, and apply the required offset (i.e. "by" was omitted, typos?). Note though, that the units for the offset did not have pixel in the selection and I used the approriate arc degrees sampling step - could it be added?

    Although I am looking into using SRTM (92m, 3-arc second) corrected for missing data using auxiliary higher resolution DEMs (e.g. 1 arc-second NED), I am also considering using 1 arc-second NED (30m). In either case the resulting DEMs resolution need to be for now 92m, 3-arc second ( in the future higher res. will be used like 45m, 30m).
    So, regarding the Sampling Spacing when exporting data or Combining layers, could you tell me how it operates?
    Is it using a sampling/thinning or an averaging method when going from say 1-arc second data to 3-arc second data?
    (See p.3 of this document ftp://e0srp01u.ecs.nasa.gov/srtm/version2/Documentation/SRTM_Topo.pdf ).

    I am assuming here that using a sampling step of 0.00083333 for pixel width and height and enforcing square pixel will result in a 3-arc second DEM data set. Am I correct?

    I really appreciated all your help and thanks to you my learning curve of both GlobalMapper and GIS data has been really steep. Thank you for your clear answers so far and looking forward to your clarifications.

    Donat
  • global_mapper
    global_mapper Administrator
    edited March 2009
    Donat,

    In order to spatially compare two data sets they need to be displayed in the same projection system and datum. Luckily Global Mapper automatically does this for you, converting all loaded data into the projection system and datum setup on the Projection tab of the Configuration dialog. This means that so long as data was loaded with the proper projection and datum (SRTM and NED data should automatically be loaded properly), they should overlap in the proper locations to allow for an elevation comparison.

    Of course to compare elevations the vertical datums used for the elevation values need to be comparable, or you need to know the offset. As you discovered SRTM and NED basically use the same vertical datum, so this is not an issue.

    Current the option to shift the selected layers only works for a specific ground distance and not pixels, although you can easily compute the ground distance representing some pixel difference. I will consider adding a direct pixel shift option for a future release.

    Currently when you sub-sample imagery (i.e. reduce the resolution) Global Mapper will just sample the data at an interval of the new sample spacing. By default bilinear interpolation will be used at the new sample location so the result could be a combination of the 4 nearest pixels to a sample location not at a sample center, but there is not any smarter sub-sampling done (at least not yet, maybe in the next release).

    Your sample spacing of 0.00083333 would correspond to 3 arc seconds (I would actually add a few more 3s onto the end to prevent any tiny gaps or overlaps).

    Let me know if I can be of further assistance.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    Mike,

    "Spot on", thanks for the quick and clear answers. I am wondering are you lots of Mike at customer support or are you a very busy bee. :)

    I got it, the NED (using NAD88) and SRTM (using WSG84/EGM96) automatically load and are converted to "the projection system and datum setup on the Projection tab of the Configuration dialog", say WSG84. I just went back to the doc and it makes, I apologize if this was already stated clearly, I am processing so much at the moment.

    So I only need to worry about the Coordinate system referencing for native format NED and SRTM when suitable (i.e. NED are pixel corner, SRTM are pixel centers DEM). Yes,having a "pixel" unit for displacement distance will be a little more user friendly and prevent an easy made typo or not enough digit of accuracy when entering arc-degrees (per pixel) values instead of pixel step size.

    Regarding the sub-sampling:
    Currently when you sub-sample imagery (i.e. reduce the resolution) Global Mapper will just sample the data at an interval of the new sample spacing. By default bilinear interpolation will be used at the new sample location so the result could be a combination of the 4 nearest pixels to a sample location not at a sample center, but there is not any smarter sub-sampling done (at least not yet, maybe in the next release).
    I am confused here, I would have assumed that for re-sampling to a lower resolution, a 3x3 pixel grid was considered (1 pixel surrounded by 8 pixels). Do you really use just 4 pixels? Is it because of the bilinear interpolation used by GlobalMapper?

    I was just reading through these links to refresh my mind:
    Bicubic interpolation - Wikipedia, the free encyclopedia
    What's the point? Interpolation and extrapolation with a regular grid DEM
    http://www.tinaja.com/glib/pixintpl.pdf

    You said, by default bilinear interpolation will be used. Is there a choice to change that to bicubic, which produce less artifact when processing time is not an issue... From wiki link above:
    "In image processing, bicubic interpolation is often chosen over bilinear interpolation or nearest neighbor in image resampling, when speed is not an issue. Images resampled with bicubic interpolation are smoother and have fewer interpolation artifacts."

    If it is not, could this be implemented easily?
    Your sample spacing of 0.00083333 would correspond to 3 arc seconds (I would actually add a few more 3s onto the end to prevent any tiny gaps or overlaps).
    I always do, which is silly because by keep pressing "3" (or the relevant trailing digit) I ended up with several field length of digit... After processing though, it seem that there some funny padding occurring. I am wondering what the number of significant digit then?

    Anyhow, I am again very impressed with your quick replies. This is exciting and helps me a lot to get my mind around DEM processing and related pitfalls for my research projects. I am learning a great deal. Thank you.

    Donat
  • global_mapper
    global_mapper Administrator
    edited March 2009
    There is just the one Mike here, I am quite the busy bee!

    I do plan to eventually add better sub-sampling, but the issues are not always straightforward as in addition to just the basic sub-sampling that is occurring an export may be done at any resolution (not just some multiple of the base resolution) and in any projection, so the actual warping of the surface could be in any shape at any given sample location. I hope to add something for the upcoming v11 release this summer though.

    The issue of where in a pixel coordinates are referenced too should also be taken care of as most formats specify this and Global Mapper should automatically adjust each layer based on the pixel referencing system used.

    When exporting keep in mind that you can either force your exact sample spacing to be obeyed (the default), which can cause your export bounds to be adjusted by up to a full sample spacing to get a whole number of pixels, or you can force the exact export bounds to be obeyed, which can cause the sample spacing to be slightly tweaked if the export size if not an exact multiple of the sample spacing. The option to obeyed the bounds over the spacing is found in the Advanced section of the General tab of the Configuration dialog.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    Hi Mike,

    I hope the "busy bee" is having a break today - it is real windy and rainy here, not a good time to go flying. ;)

    Regarding a better sub-sampling, I hear you, it is not straightforward when considering any projection at export. It sounds like version 11 is going to be a lot of work judging from other posts with other feature suggestions I read the forum today.

    Anyhow, I think you answer and clarify my main questions. I just have few more practical ones, if you don't mind - hopefully the last ones too.

    Regarding the issue of coordinate referencing location in a pixel....
    The issue of where in a pixel coordinates are referenced too should also be taken care of as most formats specify this and Global Mapper should automatically adjust each layer based on the pixel referencing system used.
    [/email]
    The DEM file formats used for my projects are BIL/HGT, GeoTIFF, and sonetimes ArcGrid (i.e. the ones provided by NASA, CGIAR or NSGS). You said it should be taken care and adjusted by Global Mapper, how do I know if it was since GM relies on the format specifying this?

    Speaking of file format, are GeoTIFF and BIL equivalent?
    I am always wondering what is best to work with. They seem to yield similar file size for the same data. Are they issue with accuracy, loss with compression?...
    I uses 3DEM and started using BIL files with it following some tutorials I was using - they are then saved as terrain matrix 16-bit signed Integer format. Geotiff can be read by Photoshop, Matlab... which I also use.

    Anyhow, time to download all my NED (30m) DEMs for the region covered by my project which rounded to integer outer boundary limits is included in Lon: -120 ... -113W and Lat: 38 ... 34N. Note that the UTM zone is mostly zone 11, but ~0.6 deg overlap on zone 12.

    On NSGS the limit per file to download is 250mb. Last practical questions for the day and this thread :)
    Do I need to overlap my selection tiles (e.g. 1degx1deg)/band of NED DEM data when selecting the coordinates in NSGS Seamless Server or edge matching should be clean and can be trusted?
    Conversely/alternatively, I just used for the 1st time the Download online Terrain Map data within Global Mapper and there was no restriction and it uploaded quick too. Is this identical to the latest NED data release from NSGS?
    Again, what would be the recommended data format (BIL, GeoTIFF, ...) if any to save them?

    I feel I have been all over the place with this thread and I hope it is ok. I feel this should be my last post since you clarified all the sticky issues I had. Thanks.

    If I don't hear from you before this coming week, have a great end of week-end Mike.

    Cheers!
    Donat
  • global_mapper
    global_mapper Administrator
    edited March 2009
    Donat,

    The GeoTIFF elevation and BIL formats are basically equivalent in that they store elevation data as uncompressed binary grids. I like the GeoTIFF format better as it has all formatting data internal. If you save a 32-bit GeoTIFF elevation file then there should be no loss of precision, whereas the 16-bit elevation GeoTIFF will round to the nearest whole elevation value.

    The best format for compression, load speed and display will be the Global Mapper Grid format but it will only be usable in Global Mapper.

    I would expect the edges to be fine via the seamless server, but it wouldn't hurt to overlap.

    The NED download in Global Mapper via the File->Download Online Imagery menu command comes from a JPL server (OnEarth, JPL WMS Server). I'm honestly not sure how often that NED data is updated from the master archive.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com
  • dpluigi
    dpluigi Global Mapper User
    edited March 2009
    THanks for the file format recommendations and the explanations.

    Regarding JPL server, I was not able to download anything from it - too busy I suppose. Ironically after trying many days in a row, I was able to download twice for the very first time yesterday morning but the NED elevation map was not saved.
    I did find some WMS data source there and uploaded them like:
    http://toposervices.cr.usgs.gov/wmsconnector/com.esri.wms.Esrimap/USGS_EDNA_geo
    but the NED data was just a straight RBG gray scale file, and not the elevation map I expected... not clear what is it used for? Maybe the Service name:
    WMS&WMTVER=1.1.1&LAYERS=NED_2003_DEM&SRS=EPSG:4326&FORMAT=PNG&BGCOLOR=0xFFFFFF&TRANSPARENT=true&SERVICE=WMS&STYLES=&REQUEST=capabilities
    need to be edited to force a different file format, e.g. GeoTIFF instead of PNG?!

    I downloaded my area from NSGS and but adjusting slightly the area, it did fit in the 1.5Gb limit and saved in six 250MB archived GeoTIFF files. I trusted NSGS server to partition the area in matching tiles and it looks fine.

    I exported this mosaic of NED DEMs into a GeoTIFF 32 (also I think I might only needs 16 bite signed integer later on), sampled at 0.0008333333... arc deg., using the native NAD83 datum (i.e. kept the original geographic projection). This file is now my 3 arc sec (92m) equivalent to my CGIAR 92m SRTMs and I will project it to UTM when I'll need to overlay GeaData, imagery, or compare to other DEMs like my SRTMs.

    I am set now. Thank you for all your help and suggestions.

    Donat
  • global_mapper
    global_mapper Administrator
    edited March 2009
    Donat,

    FYI, the USGS NED WMS server does only contain a grayscale image of the rendered NED data and not anything with the actual elevation values themselves, which makes it less than useful. Hopefully in the future they will add a layer with actual elevation values to make it useful and also hopefully more reliable than the JPL server.

    Thanks,

    Mike
    Global Mapper Support
    support@globalmapper.com