strange seterr persistence between sessions
In trying to track down a bug in matplotlib, I have come across tsome very strange numpy behavior. Basically, whether or not I call np.seterr('raise') or not in a matplotlib demo affects the behavior of seterr in another (pure numpy) script, run in a separate process. Something about the numpy state is persisting between python sessions. This appears to be platform specific, because I have only been able to verify it on 1 platform (quad code xeon 64 bit running fedora) but not on another (solaris x86). Here are the gory details. Below is a cut-and-paste from a single xterm session, with some comments sprinkled in. Some version info:: ~> uname -a Linux bic128.bic.berkeley.edu 2.6.25.10-47.fc8 #1 SMP Mon Jul 7 18:31:41 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux ~> python -V Python 2.5.1 ~> python -c 'import numpy; print numpy.__version__' 1.2.0.dev5564 ~> python -c 'import matplotlib; print matplotlib.__version__' 0.98.3rc1 With mpl svn, head over to the examples directory and grab the data file needed to show this bug:: ~> cd mpl/examples/pylab_examples/ pylab_examples> wget http://matplotlib.sourceforge.net/tmp/alpha.npy --11:22:08-- http://matplotlib.sourceforge.net/tmp/alpha.npy => `alpha.npy' Resolving matplotlib.sourceforge.net... 66.35.250.209 Connecting to matplotlib.sourceforge.net|66.35.250.209|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 688 [text/plain] 100%[===================================================================>] 688 --.--K/s 11:22:08 (111.19 MB/s) - `alpha.npy' saved [688/688] Run the geo_demo.py example. This has np.seterr set to "raise". It will issue a floating point error:: pylab_examples> head -5 geo_demo.py import numpy as np np.seterr("raise") from pylab import * pylab_examples> python geo_demo.py Traceback (most recent call last): File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/backends/backend_gtk.py", line 333, in expose_event self._render_figure(self._pixmap, w, h) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/backends/backend_gtkagg.py", line 75, in _render_figure FigureCanvasAgg.draw(self) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/backends/backend_agg.py", line 261, in draw self.figure.draw(self.renderer) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/figure.py", line 759, in draw for a in self.axes: a.draw(renderer) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/axes.py", line 1523, in draw a.draw(renderer) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/axis.py", line 718, in draw tick.draw(renderer) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/axis.py", line 186, in draw self.gridline.draw(renderer) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/lines.py", line 423, in draw tpath, affine = self._transformed_path.get_transformed_path_and_affine() File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/transforms.py", line 2089, in get_transformed_path_and_affine self._transform.transform_path_non_affine(self._path) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/transforms.py", line 1828, in transform_path_non_affine self._a.transform_path(path)) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/transforms.py", line 1828, in transform_path_non_affine self._a.transform_path(path)) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/transforms.py", line 1816, in transform_path self._a.transform_path(path)) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/projections/geo.py", line 264, in transform_path return Path(self.transform(ipath.vertices), ipath.codes) File "/home/jdhunter/dev/lib64/python2.5/site-packages/matplotlib/projections/geo.py", line 249, in transform sinc_alpha = ma.sin(alpha) / alpha File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 1887, in __div__ return divide(self, other) File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 638, in __call__ t = narray(self.domain(d1, d2), copy=False) File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 413, in __call__ return umath.absolute(a) * self.tolerance >= umath.absolute(b) FloatingPointError: underflow encountered in multiply OK, now run the pure numpy test script in a separate python process. It also has np.seterr set to raise, and it raises the same error. Nothing too strange (yet):: pylab_examples> cat test.py import numpy as np np.seterr("raise") import numpy.ma as ma alpha = np.load('alpha.npy') alpham = ma.MaskedArray(alpha) sinc_alpha_ma = ma.sin(alpham) / alpham pylab_examples> python test.py Traceback (most recent call last): File "test.py", line 7, in <module> sinc_alpha_ma = ma.sin(alpham) / alpham File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 1887, in __div__ return divide(self, other) File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 638, in __call__ t = narray(self.domain(d1, d2), copy=False) File "/home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.py", line 413, in __call__ return umath.absolute(a) * self.tolerance >= umath.absolute(b) FloatingPointError: underflow encountered in multiply OK, in your editor, comment out the np.seterr line from the geo_demo.py script, and rerun it. The demo runs fine w/o error this time and a figure window pops up. Again, nothing surprising. pylab_examples> head -5 geo_demo.py import numpy as np #np.seterr("raise") from pylab import * pylab_examples> python geo_demo.py OK, now this is where it starts getting funky. Rerun the numpy test script, with no changes (seterr is still set to raise):: pylab_examples> cat test.py import numpy as np np.seterr("raise") import numpy.ma as ma alpha = np.load('alpha.npy') alpham = ma.MaskedArray(alpha) sinc_alpha_ma = ma.sin(alpham) / alpham pylab_examples> python test.py pylab_examples> This time it ran w/o errors (and will continue to do so on successive runs). Same script, same data, same error codes. I can repeat this many times: if I turn errors back on in geo_demo, the error is raised in subsequent runs of test.py. If I turn it back off in geo_demo.py, it is not raised in subsequent runs of test.py. I tried this on a solaris x86 box and did not see this behavior. In mostly unrelated news, I find the exception in the ma divide here a bit confusing, because using np sin and divide does not raise this error, and none of the values are masked. I would expect the divides for the unmasked portions to have the same floating point behavior. Eg, only the second divide raises in the example below:: import numpy as np np.seterr("raise") import numpy.ma as ma alpha = np.load('alpha.npy') alpham = ma.MaskedArray(alpha) sinc_alpha_ma = np.sin(alpham.data) / alpham.data sinc_alpha_ma = ma.sin(alpham) / alpham JDH
On Mon, Jul 28, 2008 at 13:56, John Hunter <jdh2358@gmail.com> wrote:
In trying to track down a bug in matplotlib, I have come across tsome very strange numpy behavior. Basically, whether or not I call np.seterr('raise') or not in a matplotlib demo affects the behavior of seterr in another (pure numpy) script, run in a separate process. Something about the numpy state is persisting between python sessions. This appears to be platform specific, because I have only been able to verify it on 1 platform (quad code xeon 64 bit running fedora) but not on another (solaris x86).
Can you make a new, smaller self-contained example? I suspect stale .pyc files. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Mon, Jul 28, 2008 at 2:02 PM, Robert Kern <robert.kern@gmail.com> wrote:
On Mon, Jul 28, 2008 at 13:56, John Hunter <jdh2358@gmail.com> wrote:
In trying to track down a bug in matplotlib, I have come across tsome very strange numpy behavior. Basically, whether or not I call np.seterr('raise') or not in a matplotlib demo affects the behavior of seterr in another (pure numpy) script, run in a separate process. Something about the numpy state is persisting between python sessions. This appears to be platform specific, because I have only been able to verify it on 1 platform (quad code xeon 64 bit running fedora) but not on another (solaris x86).
Can you make a new, smaller self-contained example? I suspect stale .pyc files.
I'm not sure exactly what you mean by self-contained (since the behavior requires at least two files). Do you mean trying to come up with two numpy only examples files, or one that does away with the npy file? Or both.... As for the stale files, I'm not sure what you are thinking but these are clean builds and installs of numpy and mpl. So if you'll give me a little more guidance in terms of what you are looking for in a self contained example, I'll be happy to try and put it together. But I am not sure what it is about the loading of the geo_demo that is triggering the behavior (numpy extension code, large memory footprint, ??). I tried running a python snippet that would fill a lot of memory to see if that would clear the persistence. I was wondering if there is some kernel memory caching and some empty numpy memory that is not getting initialized properly and thus is picking up some memory from a prior session) but it did not. JDH
On Mon, Jul 28, 2008 at 14:30, John Hunter <jdh2358@gmail.com> wrote:
On Mon, Jul 28, 2008 at 2:02 PM, Robert Kern <robert.kern@gmail.com> wrote:
On Mon, Jul 28, 2008 at 13:56, John Hunter <jdh2358@gmail.com> wrote:
In trying to track down a bug in matplotlib, I have come across tsome very strange numpy behavior. Basically, whether or not I call np.seterr('raise') or not in a matplotlib demo affects the behavior of seterr in another (pure numpy) script, run in a separate process. Something about the numpy state is persisting between python sessions. This appears to be platform specific, because I have only been able to verify it on 1 platform (quad code xeon 64 bit running fedora) but not on another (solaris x86).
Can you make a new, smaller self-contained example? I suspect stale .pyc files.
I'm not sure exactly what you mean by self-contained (since the behavior requires at least two files). Do you mean trying to come up with two numpy only examples files, or one that does away with the npy file? Or both....
Both, if the behavior exhibits itself without the npy file. If it only exhibits itself with an npy involved, then we have some more information about where the problem might be.
As for the stale files, I'm not sure what you are thinking but these are clean builds and installs of numpy and mpl.
And of the files you are editing and anything they might import? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On Mon, Jul 28, 2008 at 2:35 PM, Robert Kern <robert.kern@gmail.com> wrote:
Both, if the behavior exhibits itself without the npy file. If it only exhibits itself with an npy involved, then we have some more information about where the problem might be.
OK, I'll see what I can come up with. In the mean time, as I was trying to strip out the npy component and put the data directly into the file, I find it strange that I am getting a floating point error on this operation import numpy as np np.seterr("raise") import numpy.ma as ma x = 1.50375883 m = ma.MaskedArray([x]) sinc_alpha_ma = ma.sin(m) / m --------------------------------------------------------------------------- FloatingPointError Traceback (most recent call last) /home/jdhunter/<ipython console> in <module>() /home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.pyc in __div__(self, other) 1885 def __div__(self, other): 1886 "Divide other into self, and return a new masked array." -> 1887 return divide(self, other) 1888 # 1889 def __truediv__(self, other): /home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.pyc in __call__(self, a, b) 636 d1 = getdata(a) 637 d2 = get_data(b) --> 638 t = narray(self.domain(d1, d2), copy=False) 639 if t.any(None): 640 mb = mask_or(mb, t) /home/jdhunter/dev/lib64/python2.5/site-packages/numpy/ma/core.pyc in __call__(self, a, b) 411 if self.tolerance is None: 412 self.tolerance = np.finfo(float).tiny --> 413 return umath.absolute(a) * self.tolerance >= umath.absolute(b) 414 #............................ 415 class _DomainGreater: FloatingPointError: underflow encountered in multiply I am no floating point expert, but I don't see why a numerator of 0.99775383 and a denominator of 1.50375883 should be triggering an underflow error. It looks more like a bug in the ma core logic since umath.absolute(a) * self.tolerance is more or less guaranteed to fail if np.seterr("raise") is set JDH
Mon, 28 Jul 2008 13:56:52 -0500, John Hunter wrote:
In trying to track down a bug in matplotlib, I have come across tsome very strange numpy behavior. Basically, whether or not I call np.seterr('raise') or not in a matplotlib demo affects the behavior of seterr in another (pure numpy) script, run in a separate process. Something about the numpy state is persisting between python sessions. This appears to be platform specific, because I have only been able to verify it on 1 platform (quad code xeon 64 bit running fedora) but not on another (solaris x86). [clip]
I don't see this on Python 2.5.2, Linux, AMD Athlon x86, numpy SVN r5542: test.py always raises the FloatingPointError. But on a related note: On 1.2.0.dev5542, I always see the FloatingPointError in test.py. On 1.1.0, I don't see any FloatingPointErrors. Still, nothing depends on running geo_data.py. Are your numpy versions the same on these platforms? Do you have two numpy versions installed. Could it be possible that somehow running the scripts switches between numpy versions? (Sounds very strange, and of course, this is easy to check for in test.py...) -- Pauli Virtanen
participants (3)
-
John Hunter -
Pauli Virtanen -
Robert Kern