29 Jan
2007
29 Jan
'07
7:33 p.m.
On 1/29/07, Keith Goodman
On 1/29/07, Charles R Harris
wrote: That's odd, the LSB bit of the double precision mantissa is only about 2.2e-16, so you can't *get* differences as small as 8.4e-22 without about 70 bit mantissa's. Hmmm, and extended double precision only has 63 bit mantissa's. Are you sure you are computing the error correctly?
That is odd.
8.4e-22 is just the output of the test script: abs(z - z0).max(). That abs is from python.
By playing around with x and y I can get all sorts of values for abs(z - z0).max(). I can get down to the e-23 range and to 2.2e-16. I've also seen e-18 and e-22.