On Wed, Jul 2, 2014 at 10:36 AM, Julian Taylor < jtaylor.debian@googlemail.com> wrote: we recently fixed a float32/float64 issue in histogram.
It's a good idea to keep the edges in the same dtype as the input data, it will make for fewer surprises, but I'm not sure that it's necessarily any more "correct". A value within an eps of a bin could arbitrarily end up on either side -- that's simply the nature of floating point.
I think it boils down to the use of round() in histogram which is not so great in python as its based on decimals not significant figures (so it does nothing for float32 values > 1e7).
Using decimals rather than sig-figs is a problem regardless of precision, and isn't that the same problem with C libmath round() ? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov