Floating point subtraction rounding error (NOT display error)

Keflavich keflavich at gmail.com
Fri Dec 14 09:55:06 EST 2007


On Dec 14, 2:57 am, "Nikos Vergas" <ni... at vergas.gr> wrote:
> > Solved: used round(number,12) in this case for all of the operands of
> > my arcsines.  Not pretty, but at least VIM made it easy...
>
> You might have the same problem though:
>
> >>> round(1.000340100000325235000000235,13)
> 1.0003401000003
> >>> round(1.000340100000325235000000235,12)
>
> 1.0003401000000001

True, but I think that's actually the behavior I'm looking for: the
truncation in this case gives me a smaller number (I hope it's safe to
assume that 1.0003401000000001 < 1.0003401000003), which is exactly
what I want when I'm worried about subtractions giving me numbers very
slightly larger than 1.

Gabriel, Paddy, thanks for the overviews.  Yes, floats deserve more
respect, or at least more caution.

I feel fairly certain, however, that floats are exactly what I want
for my purposes: I need moderately high precision and I'm not
concerned about the least-significant-bit errors except when they
violate function domains.  I guess the overriding lesson is that every
float subtraction needs to be carefully thought through, which is a
disappointing loss of simplicity for me, but I'll deal with it.

Adam



More information about the Python-list mailing list