[Numpy-discussion] Preserving NumPy views when pickling

Robert Kern robert.kern at gmail.com
Tue Oct 25 19:28:22 EDT 2016


On Tue, Oct 25, 2016 at 3:07 PM, Stephan Hoyer <shoyer at gmail.com> wrote:
>
> On Tue, Oct 25, 2016 at 1:07 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Concretely, what do would you suggest should happen with:
>>
>> base = np.zeros(100000000)
>> view = base[:10]
>>
>> # case 1
>> pickle.dump(view, file)
>>
>> # case 2
>> pickle.dump(base, file)
>> pickle.dump(view, file)
>>
>> # case 3
>> pickle.dump(view, file)
>> pickle.dump(base, file)
>>
>> ?
>
> I see what you're getting at here. We would need a rule for when to
include the base in the pickle and when not to. Otherwise,
pickle.dump(view, file) always contains data from the base pickle, even
with view is much smaller than base.
>
> The safe answer is "only use views in the pickle when base is already
being pickled", but that isn't possible to check unless all the arrays are
together in a custom container. So, this isn't really feasible for NumPy.

It would be possible with a custom Pickler/Unpickler since they already
keep track of objects previously (un)pickled. That would handle [base,
view] okay but not [view, base], so it's probably not going to be all that
useful outside of special situations. It would make a neat recipe, but I
probably would not provide it in numpy itself.

--
Robert Kern
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20161025/130b2f27/attachment.html>


More information about the NumPy-Discussion mailing list