[Numpy-discussion] numpy pickling problem - python 2 vs. python 3

Arnd Baecker arnd.baecker at web.de
Fri Mar 6 09:48:38 EST 2015


On Fri, 6 Mar 2015, Pauli Virtanen wrote:

> Arnd Baecker <arnd.baecker <at> web.de> writes:
> [clip]
>> Still I would have thought that this should be working out-of-the box,
>> i.e. without the pickle.loads trick?
>
> Pickle files should be considered incompatible between Python 2 and Python 3.
>
> Python 3 interprets all bytes objects saved by Python 2 as str and attempts
> to decode them under some unicode locale. The default locale is ASCII, so it
> will simply just fail in most cases if the files contain any binary data.
>
> Failing by default is also the right thing to do, since the saved bytes
> objects might actually represent strings in some locale, and ASCII is the
> safest guess.
>
> This behavior is that of Python's pickle module, and does not depend on Numpy.

Thank's a lot for the explanation!

So what is then the recommded way to save data under python 2 so that
they can still be loaded under python 3?

For example using np.save with a list of arrays works fine 
either on python 2 or on python 3.
However it does not work if one tries to open under python 3
a file generated before on python 2.
(Again, because pickle is involved internally
   "python3.4/site-packages/numpy/lib/npyio.py",
   line 393, in load  return format.read_array(fid)
   File "python34/lib/python3.4/site-packages/numpy/lib/format.py",
   line 602, in read_array  array = pickle.load(fp)
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0  ...

Just to be clear: I don't want to beat a dead horse here - for my usage
via pytables I was able to solve the loading of old files following
Ryan's solutions. Personally I don't use .npy files.
Maybe saving a list containing arrays is an unusual example ...

Still, I am a little bit worried about backwards-compatibility:
being able to load old data files is an important issue
as by this it is possible to check whether current code still
reproduces previously obtained (maybe also published) results.

Best, Arnd




More information about the NumPy-Discussion mailing list