[Tutor] precision handling
Dennis Lee Bieber
wlfraed at ix.netcom.com
Thu Sep 30 10:43:23 EDT 2021
On Thu, 30 Sep 2021 13:37:03 +0530, Msd De <mmssdd1920 at gmail.com> declaimed
the following:
>Hello,
>The two same numbers generated in python 2.7 when subtracted should give
>zero. Instead it gives - 4.7393372903e-07.
>The number are given below:
>
>176697092.941 - 176697092.941 = - 4.7393372903e-07
>
>How would I increase the precision for the whole document in python?
What "document"?
You state "generated" -- but don't show the code doing that generation.
You have just encountered the fact that floating point arithmetic is not
exact -- there may be some small tail-end that is not displayed, etc.
Note that entering the exact text you have above does produce 0.0
PythonWin 3.8.2 (default, Aug 25 2020, 15:54:26) [MSC v.1900 64 bit
(AMD64)] on win32.
Portions Copyright 1994-2018 Mark Hammond - see 'Help/About PythonWin' for
further copyright information.
>>> 176697092.941 - 176697092.941
0.0
>>>
>>> a = 176697092.0 + 0.941
>>> a
176697092.941
>>> a - 176697092.0
0.9410000145435333
>>>
>>> a - 176697092.0 - 0.941
1.454353337759784e-08
>>>
Try using the Decimal package -- but you may need to modify a lot of
code to ensure the numbers stay decimal throughout.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed at ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
More information about the Tutor
mailing list