[Tutor] help: funny numbers

Kirby Urner urnerk@qwest.net
Tue, 02 Apr 2002 12:03:44 -0800


>
>[66.599999999999994, 333, -1, 333, 1, 1234.5, 333]
> >>>
>
>In the above list what happened to 66.6 when I redisplayed
>the list?
>
>                                             Gord

This is the age-old problem with floating point numbers.

Internally, they're converted from decimal to base 2,
and what come off as terminating decimals often end up
being non-terminal/repeating in binary.  Some truncation
occurs, and you notice the effects upon converting back
to decimal.

You will recall, for example, that 1/3 in decimal is
.33333... whereas 1/8 is .125 exactly.  With 1/3, we
have only a finite amount of space for digits, and so
have to truncate or round at some point.  The stored
decimal is no longer precisely 1/3.  Likewise, with
binary numbers, 6.65 isn't exactly captured in the
space allotted, so you get stuff like:

  >>> a = 6.65
  >>> a
  6.6500000000000004

We're used to thinking of finite decimals as approximations
of fractions, it's just that we forget 665/100 (i.e. 6.65)
is, for the computer, a division problem in base 2, and so
we keep being surprised by "floating point noise" in our
calculations.

Kirby