C extension + libm oddity [fmod(2.0, 2.0) == nan ?!]
Paul Rubin
http
Fri Nov 4 20:13:26 EST 2005
"Lonnie Princehouse" <finite.automaton at gmail.com> writes:
> Now, fmod(2.0, 2.0) should be 0.0. The problem? ans is getting
> assigned nan! I have stepped through it in the debugger now dozens of
> times. Either fmod is putting the wrong return value on the stack, or
> the stack is getting corrupted by something else and "ans" is getting
> assigned the wrong value.
Have you compiled the C extension with all optimization turned off?
Especially with optimizations on, you can't really tell what is going
to get assigned in a variable because the code isn't computing the
intermediate values you might expect it to. I suggest disassembling
it and stepping through it instruction by instruction if you haven't
done that.
More information about the Python-list
mailing list