[Numpy-discussion] numpy pickling problem - python 2 vs. python 3

Pauli Virtanen pav at iki.fi
Sat Mar 7 04:54:06 EST 2015


07.03.2015, 01:29, Julian Taylor kirjoitti:
> On 07.03.2015 00:20, Pauli Virtanen wrote:
>> 06.03.2015, 22:43, Eric Firing kirjoitti:
>>> On 2015/03/06 10:23 AM, Pauli Virtanen wrote:
>>>> 06.03.2015, 20:00, Benjamin Root kirjoitti:
>>>>> A slightly different way to look at this is one of sharing data. If I am
>>>>> working on a system with 3.4 and I want to share data with others who may
>>>>> be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
>>>>> less attractive.
>>>>
>>>> pickle is used in npy files only if there are object arrays in them.
>>>> Of course, savez could just decline saving object arrays.
>>>
>>> Or issue a prominent warning.
>>
>> https://github.com/numpy/numpy/pull/5641
>>
> 
> I think the ship for a warning has long sailed. At this point its
> probably more an annoyance for python3 users and will not prevent many
> more python2 users from saving files that can't be loaded into python3.

How about an extra use_pickle=True kwarg that can be used to disable
using pickle altogether in these routines?

Another reason to do this is arbitrary code execution when loading
pickles: https://www.cs.jhu.edu/~s/musings/pickle.html

Easily demonstrated also with npy files (loading this file will only
print something unexpected, nothing more malicious):
http://pav.iki.fi/tmp/unexpected.npy




More information about the NumPy-Discussion mailing list