Strange rounding problem
Steven Taschuk
staschuk at telusplanet.net
Sat Mar 15 22:47:49 CET 2003
Quoth Grant Edwards:
> In article <TrCcnZMtctRJ_u6jXTWc3Q at comcast.com>, Marc wrote:
[...]
> > So I still don't understand that if I enter a number to be an exact value of
> > .00000096, why it can't be stored as 9.60000000e-007?
>
> Becausing you entered it in base 10, and it's stored in base 2. Just
> because it's an exact value in base 10 it doesn't mean it can be represented
> exactly in base 2.
For concreteness, it may be interesting to note that
x = 9.6e-7
y = 9.5999999999999991e-7
are in binary approximately:
x = 1.000000011011001010110010100110100100011010010010101101100e-20
y = 1.000000011011001010110010100110100100011010010010101011111e-20
These values first differ at the 53rd significant bit, here: ^.
Even if the value in question had a terminating binary expansion
(which it doesn't), it takes a lot of precision to distinguish
these two values. It's *just* within the capacity of IEEE 754
double-precision floats, I think.
--
Steven Taschuk 7\ 7'Z {&~ .
staschuk at telusplanet.net Y r --/hG-
(__/ )_ 1^1`
More information about the Python-list
mailing list