Strange rounding problem

[Steven Taschuk]

Quoth Grant Edwards:
> In article <TrCcnZMtctRJ_u6jXTWc3Q at comcast.com>, Marc wrote:
  [...]
> > So I still don't understand that if I enter a number to be an exact value of
> > .00000096, why it can't be stored as 9.60000000e-007?
> 
> Becausing you entered it in base 10, and it's stored in base 2.  Just
> because it's an exact value in base 10 it doesn't mean it can be represented
> exactly in base 2.

For concreteness, it may be interesting to note that
    x = 9.6e-7
    y = 9.5999999999999991e-7
are in binary approximately:
    x = 1.000000011011001010110010100110100100011010010010101101100e-20
    y = 1.000000011011001010110010100110100100011010010010101011111e-20
These values first differ at the 53rd significant bit, here: ^.

Even if the value in question had a terminating binary expansion
(which it doesn't), it takes a lot of precision to distinguish
these two values.  It's *just* within the capacity of IEEE 754
double-precision floats, I think.

-- 
Steven Taschuk                                7\ 7'Z {&~         .
staschuk at telusplanet.net                        Y r          --/hG-
                                            (__/ )_             1^1`