object array binary pickling and nan
It seems that object arrays will not pickle if they have a nan when the pickle protocol is set to binary. From what I can tell the object array simply delegates to python (which makes sense) which in turn cannot pickle nans in binary format. It is unfortunate because it is very useful to have heterogenous arrays that include nans. What do other people do in this situation? Does anyone know why python has this limitation? Is there an intelligent workaround other than search and replace? Would it be worth it to put an intelligent workaround into numpy so it is transparent to the user? I was wondering what people thought. Code that reproduces the problem: This is regular python: pickle.dumps(numpy.nan, 2) SystemError: frexp() result out of range This is fine in numpy: pickle.dumps(numpy.array([numpy.nan]), 2) This breaks: pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) SystemError: frexp() result out of range --Tom
Tom Denniston wrote:
Code that reproduces the problem:
This is regular python: pickle.dumps(numpy.nan, 2) SystemError: frexp() result out of range
This is fine in numpy: pickle.dumps(numpy.array([numpy.nan]), 2)
This breaks: pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) SystemError: frexp() result out of range
It seems to me that the example that shows what you want is the only example that numpy has control over, anyway. In the other two cases, you're really using a Python object. So it's really a Python question as to why it can't pickle nans. My understanding on this is weak, but here's one summary, albeit dated: http://www.python.org/dev/summary/2000-10-1.html Here's something more recent: http://mail.python.org/pipermail/python-list/2005-July/290272.html NumPy sidesteps these issues by "forcing" IEEE-754 behavior with clever C written AFAIK by the numarray people. These apparently work even if the C compiler doesn't "officially" support IEEE-754, but it's not going to work on every platform that Python targets. You may also want to take a look at http://scipy.org/FAQ#head-fff4d6fce7528974185715153cfbc1a191dcb915
On Fri, 28 Jul 2006 12:17:39 -0500
"Tom Denniston"
It seems that object arrays will not pickle if they have a nan when the pickle protocol is set to binary. From what I can tell the object array simply delegates to python (which makes sense) which in turn cannot pickle nans in binary format. It is unfortunate because it is very useful to have heterogenous arrays that include nans. What do other people do in this situation? Does anyone know why python has this limitation? Is there an intelligent workaround other than search and replace? Would it be worth it to put an intelligent workaround into numpy so it is transparent to the user? I was wondering what people thought.
Code that reproduces the problem:
This is regular python: pickle.dumps(numpy.nan, 2) SystemError: frexp() result out of range
This is fine in numpy: pickle.dumps(numpy.array([numpy.nan]), 2)
This breaks: pickle.dumps(numpy.array([numpy.nan], numpy.PyObject), 2) SystemError: frexp() result out of range
I believe this will be fixed in Python 2.5 (see http://www.python.org/sf/445484). I guess your best alternative would be to use a numpy scalar object (like double) instead of a Python float for NaN's. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
participants (3)
-
Andrew Straw
-
David M. Cooke
-
Tom Denniston