[Tutor] help: funny numbers
Kirby Urner
urnerk@qwest.net
Tue, 02 Apr 2002 12:03:44 -0800
>
>[66.599999999999994, 333, -1, 333, 1, 1234.5, 333]
> >>>
>
>In the above list what happened to 66.6 when I redisplayed
>the list?
>
> Gord
This is the age-old problem with floating point numbers.
Internally, they're converted from decimal to base 2,
and what come off as terminating decimals often end up
being non-terminal/repeating in binary. Some truncation
occurs, and you notice the effects upon converting back
to decimal.
You will recall, for example, that 1/3 in decimal is
.33333... whereas 1/8 is .125 exactly. With 1/3, we
have only a finite amount of space for digits, and so
have to truncate or round at some point. The stored
decimal is no longer precisely 1/3. Likewise, with
binary numbers, 6.65 isn't exactly captured in the
space allotted, so you get stuff like:
>>> a = 6.65
>>> a
6.6500000000000004
We're used to thinking of finite decimals as approximations
of fractions, it's just that we forget 665/100 (i.e. 6.65)
is, for the computer, a division problem in base 2, and so
we keep being surprised by "floating point noise" in our
calculations.
Kirby