›

Slight difference between geographic point location and LAT/LON location

JakeJake Global Mapper UserPosts: 276Trusted User
edited January 2013 in Bug Report
Windows 7 64 bit, GM 13.2(b100512) 64 bit

I have a WGS84 LAT/LON shapefile. It is attached. When I query points I noticed that the point location is not exactly the same as the LAT/LON location. The point location seems to be subtracting a constant value from the latitude. See the screen grab of the INFO for a sample point. I have highlighted the difference. It is 0.0000000008"

The workspace is WGS84, the layer is WGS84 and the source is WGS84. No transformations or shifts should be occurring.

I realize this is a very small shift but I worry it might be larger in different circumstances or datasets. I suspect it is due to some internal rounding or conversion. I believe the SHP file is DD to six decimal places. I have set the display to DD as well so there should be no conversion.

If you say it is rounding and will not exceed 0.0000000008 I will OK with that. I just have to round the output.

Comments

  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited January 2013
    That's just the rounding for the display. One is only displaying to 8 decimal digits and the other has 10. The .8992 rounds to .90.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • JakeJake Global Mapper User Posts: 276Trusted User
    edited January 2013
    That can't be. The data only contains 7 decimal places (I verified this in another package). So where are the extra decimal places coming from? You can see that the LAT/LON values have a single zero padding the end to make it 8 decimal places. Psdding I don't mind. But the point location has 10 non-zero digits. That is not normal.

    To make an example, if I had a lat/lon of 45, -60 I would expect to see 45.00000000°N, 60.00000000°W. Not 44.99998992, 60.00000000.
  • global_mapperglobal_mapper Administrator Posts: 17,238
    edited January 2013
    The Shapefile format stores the X and Y coordinates as 64-bit double-precision floating point number, not some specific number of digits. How many digits of precision to store is up to the application. I also wouldn't be surprised if an application that saved a Shapefile used 32-bit floating point numbers internally, then just converts to 64-bit double when writing to Shapefile, so you might get round-off at the 32-bit single precision level rather than the 64-bit double precision level.

    For example one value in the Shapefile is stored as '-63.833333299999993'. You would see this in an application to 7 digits as '-63.8333333'. However floating point numbers can't exactly store that, so the internal representation is not exact. To get an exact representation of a number with some number of digits of precision you would need to pre-multiply it by a value like 10,000,000 for 7 digits then round to the nearest integer and store the integer value rather than a floating point. That's not how Shapefile works though. In any case the differences are way back in the noise and inconsequential in terms of location.

    Thanks,

    Mike
    Global Mapper Guru
    gmsupport@bluemarblegeo.com
    http://www.bluemarblegeo.com/
  • JakeJake Global Mapper User Posts: 276Trusted User
    edited January 2013
    Ah thanks for the explanation. I just needed to account for the difference and that explanation fits with what I am seeing. We have seen the effect of single and double precision in the past when dealing with geodetic calculations. I suspect you are correct about the application saving as single precision. If I list 100 vertices of a polygon they will all have 2 trailing zeros. So only 6 decimal places.

    I imagine you already know this but the S57 format does what you are saying,
    Latitude and longitude values are converted from decimal degrees to integers by means of the
    Coordinate Multiplication Factor [COMF] subfield value in the Data Set Parameter [DSPM] field. The
    integer values are encoded in the Coordinate in Y-axis [YCOO] subfield and the Coordinate in X-axis
    [XCOO] subfield. The number of decimal digits is chosen by the data producer and is valid through out
    the data set.
    E.g. : If the producer chooses a resolution of 0.0001° (10^-4), then the value of COMF is 10 000 (10^4).
    A longitude = 34.5678° is converted into XCOO = longitude * COMF = 34.5678*10 000 = 345678.
    The integer value of the converted coordinate is encoded in binary form.
Sign In or Register to comment.