Decimal arithmetic, with example code
Chris Gonnerman
chris.gonnerman at newcenturycomputers.net
Tue Oct 1 21:43:50 EDT 2002
----- Original Message -----
From: "Paul Boddie" <paul at boddie.net>
> Here are some example results from an extended
> version of the original program:
>
> PCT AMT Fixed Fixed Fixed Scaled Scaled Float
> 2dp 4dp -> 2dp 2dp 4dp
<< hack >>
> 0.10 0.45 0.04 0.0450 0.04 0.05 0.0450 0.05
> 0.10 0.65 0.06 0.0650 0.06 0.07 0.0650 0.07
> 0.10 0.85 0.08 0.0850 0.08 0.09 0.0850 0.09
Whoops... can someone on the FixedPoint team explain this
to me? When did 0.85 * 0.10, rounded to 2 places, become
0.08? It should be 0.09.
Perhaps I misunderstood. Am I supposed to use higher
precision and then lower it after computation?
This isn't any better than floats.
Chris Gonnerman -- chris.gonnerman at newcenturycomputers.net
http://newcenturycomputers.net
More information about the Python-list
mailing list