# Problem with standard deviation values

Global Mapper User
edited September 2011
In using the contrast stretch feature on some of my data I was getting unexpected results. (I'm loading a 1-band 16-bit BSQ file as raster image data.) After experimenting a little, I found that the standard deviation seems to be getting computed wrong whenever the mean is greater than zero and the data contains any values of 13574 or more. Even a few dozen values of 13574 out of a million pixels is enough to make a difference I can see. If the mean is small or negative, I don't seem to see the effect.

So I asked myself, what's so special about 13574? I found that (32767 + 13574)^2 is just over 2^31, so I'd guess this is some sort of integer overflow problem. (Sorry - I'm a software engineer myself, so I couldn't resist playing around with the numbers!)

I've attached two sets of files that show the problem clearly. In "bad.bsq", the data ranges from 6426 on the left to 13574 on the right, with a mean of 10000, and is constructed so that the min/max range fits exactly in +/- 2 standard deviations. The file "good.bsq" is the same thing except that I've replaced the value 13574 with 13573 in the far right column of pixels. If you load good.bsq you'll see a graded band of grays from black to white, as expected. But with bad.bsq, you'll see that the about 20% of the image on the right side has been clamped to white by the contrast stretch. To match the results of the good file, you have to increase the contrast stretch to 4 standard deviations. (So in this case it seems to be off by a factor of two, though that was not typical of other cases I tried.)

One other question: I was wondering why the number of standard deviations is limited to 4. I have an application where I sometimes need to use a stretch of up to 5 standard deviations; and I saw another forum poster use 5 standard deviations as an example of what he wanted to do with LiDAR data, so apparently I'm not the only one with a use for it. Is there a technical reason it's limited to 4, or was that just considered the largest value likely to be needed?

Regards,

-- Leland

edited September 2011
Leland,

Thank you for the file and analysis, you are correct there was an integer overflow situation calculating the standard deviation, I was missing a cast to 'double' for some of the math. Once that was fixed the files behave as expected. I also increased the max number of standard deviations to stretch by to 10. I have placed a new build at http://www.globalmapper.com/global_mapper12.zip with the change for you to try. Simply download that file and extract the contents into your existing v12.xx installation folder to give it a try. If you are using the 64-bit v12 version there is a new build at http://www.globalmapper.com/global_mapper12_64bit.zip .

Let me know if I can be of further assistance.

Thanks,

Mike
Global Mapper Support
support@globalmapper.com
• Global Mapper User
edited September 2011
Thanks, Mike!

Your speed in responding to questions and bugs is legendary. I'm glad to see it for myself. The results of the contrast stretch are exactly what I expect now. Problem solved!

I'm having a couple of other issues as well; I'll create new threads for those as soon as I have good test cases to send you.

-- Leland