Strange rounding problem

Marc mnations at airmail.net
Fri Mar 14 18:40:52 EST 2003


Hi all,

I have a number that slightly changes value at different points. I
haven't been able to find any information on how to "stabilize" it.
Here's the code snippet:


... in function getGap ...
if gap <= .00000096:
    gap = .00000096
print "GAP IN ROUTINE " + str(gap)
return [list, float(gap), self.slotID]

... call to function...
gapList = slot.getGap(iteration, size)
print "GAP BEFORE CONVERSION " + str(gapList[1])
gap = int(gapList[1]/pow(10,-9))
print "GAP AFTER CONVERSION " + str(gap)
print "gapList = " + str(gapList)

... output ...
GAP IN ROUTINE 9.6e-007
GAP BEFORE CONVERSION 9.6e-007
GAP AFTER CONVERSION 959
gapList = [[], 9.5999999999999991e-007, '3']


This appears to be confusing. The variable is set to .00000096 but is
passed through as 9.5999999999999991e-007. It appears accurate when
printed before the value is returned, is inaccurate when the full
return list is printed, accurate when it's printed by itself after the
return, and then inaccurate again when the division is made. Why does
it do this, and how can I avoid it?

Thanks,
Marc




More information about the Python-list mailing list