pickling arrays: numpy 1.0 can't unpickle numpy 1.0.1
I'm running numpy 1.0 and 1.0.1 on several hosts and today I've found that pickling arrays in 1.0.1 generates problems to 1.0. An example: --- numpy 1.0.1 --- import numpy import pickle a = numpy.array([1,2,3]) f=open('test1.pickle','w') pickle.dump(a,f) f.close() --- If I unpickle test1.pickle in numpy 1.0 I got: --- numpy 1.0
import numpy import pickle f=open('test1.pickle') a=pickle.load(f) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 1370, in load return Unpickler(file).load() File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 858, in load dispatch[key](self) File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 1217, in load_build setstate(state) TypeError: argument 1 must be sequence of length 5, not 8
How can I let access pickled arrays made in numpy 1.0.1 to numpy 1.0 ? Help! Thanks in advance, Emanuele
On 12/7/06, Emanuele Olivetti <olivetti@itc.it> wrote:
How can I let access pickled arrays made in numpy 1.0.1 to numpy 1.0 ?
If you pickle in 1.0.1, I bet you can read it in 1.0. I don't know why the pickle format keeps changing. But I understand why an old version of software can't always read data generated by a new version of software.
On 12/7/06, Keith Goodman <kwgoodman@gmail.com> wrote:
On 12/7/06, Emanuele Olivetti <olivetti@itc.it> wrote:
How can I let access pickled arrays made in numpy 1.0.1 to numpy 1.0 ?
If you pickle in 1.0.1, I bet you can read it in 1.0.
I don't know why the pickle format keeps changing. But I understand why an old version of software can't always read data generated by a new version of software.
Sorry. I meant if you pickle in 1.0, I bet you can read it in 1.0.1.
On 12/7/06, Keith Goodman <kwgoodman@gmail.com> wrote:
On 12/7/06, Keith Goodman <kwgoodman@gmail.com> wrote:
On 12/7/06, Emanuele Olivetti <olivetti@itc.it> wrote:
How can I let access pickled arrays made in numpy 1.0.1 to numpy 1.0 ?
If you pickle in 1.0.1, I bet you can read it in 1.0.
I don't know why the pickle format keeps changing. But I understand why an old version of software can't always read data generated by a new version of software.
The 1.0.x versions are supposed to be compatible. I don't see any changes to pickle in svn since before the 1.0 release, so there might be another problem here. Are there other differences between the machines? Python version, OS, endianess, 32 vs 64 bit, compiler, etc. Chuck
On Thu, December 7, 2006 6:42 pm, Charles R Harris wrote:
The 1.0.x versions are supposed to be compatible. I don't see any changes to pickle in svn since before the 1.0 release, so there might be another problem here. Are there other differences between the machines? Python version, OS, endianess, 32 vs 64 bit, compiler, etc.
Both hosts are 32bit i386 with python 2.5 and different version of numpy. Gcc version is pretty similar: - host1 : gcc version 3.4.5 20051201 (Red Hat 3.4.5-2) - host2 : gcc version 3.4.6 20060404 (Red Hat 3.4.6-3) Try my example yourself and tell me if it works for you. Thanks, Emanuele
Emanuele Olivetti wrote:
I'm running numpy 1.0 and 1.0.1 on several hosts and today I've found that pickling arrays in 1.0.1 generates problems to 1.0. An example: --- numpy 1.0.1 --- import numpy import pickle a = numpy.array([1,2,3]) f=open('test1.pickle','w') pickle.dump(a,f) f.close() ---
If I unpickle test1.pickle in numpy 1.0 I got: --- numpy 1.0
import numpy import pickle f=open('test1.pickle') a=pickle.load(f) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 1370, in load return Unpickler(file).load() File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 858, in load dispatch[key](self) File "/hardmnt/virgo0/sra/olivetti/myapps/lib/python2.5/pickle.py", line 1217, in load_build setstate(state) TypeError: argument 1 must be sequence of length 5, not 8
Please show which version of numpy you are using. There were no changes to pickle from released numpy 1.0 to 1.0.1 (at least that I'm aware of). There might, however, be bugs. -Travis
Emanuele Olivetti wrote:
I'm running numpy 1.0 and 1.0.1 on several hosts and today I've found that pickling arrays in 1.0.1 generates problems to 1.0. An example:
I correct my previous statement. Yes, this is true. Pickles generated with 1.0.1 cannot be read by version 1.0 However, pickles generated with 1.0 can be read by 1.0.1. It is typically not the case that pickles created with newer versions of the code will work with older versions. I obviously didn't think that was something to be concerned about because it slipped my mind. Changeset 3411 is the reason where the hasobject member of the data-type object was given more bits (and therefore needed to be saved). Sorry for the trouble, -Travis
Travis E. Oliphant wrote:
I correct my previous statement. Yes, this is true. Pickles generated with 1.0.1 cannot be read by version 1.0
However, pickles generated with 1.0 can be read by 1.0.1. It is typically not the case that pickles created with newer versions of the code will work with older versions. I obviously didn't think that was something to be concerned about because it slipped my mind.
Changeset 3411 is the reason where the hasobject member of the data-type object was given more bits (and therefore needed to be saved).
Thank you for the detailed explanation. I can easily upgrade the central host that collects results from other hosts in order to be able to read all results (1.0 or 1.0.1). Emanuele
participants (7)
-
Charles R Harris -
Emanuele Olivetti -
Emanuele Olivetti -
emanuele＠relativita.com
-
Keith Goodman -
Travis E. Oliphant -
Travis Oliphant