[Python-Dev] Pickler/Unpickler API clarification

[Greg Ewing]

Collin Winter wrote:

> Reusing the Pickler without clearing the
> memo will produce pickles that are, as best I can see, invalid

I'm not sure what you mean by "reusing the pickler" here,
and how it can produce an invalid pickle.

I think what the docs mean by it is continuing to pickle
objects to the same file, but in a logically separate
block that doesn't share any references with the previous
one, e.g.

    pickle obj1
    pickle obj2
    ---clear memo---
    pickle obj3

The whole thing is still a valid pickle containing 3 objects,
whether the memo is cleared at any point or not, and can
be unpickled using 3 corresponding unpickle calls to a
single unpickler.

> 1) Should Pickler/Unpickler objects automatically clear their memos
> when dumping/loading?

If you mean should every call to Pickler.dump() or
Unpickler.load() clear the memo first, definitely *NOT*.
It's explicitly part of the specification that you can
make multiple calls to dump() to build up a single pickle
that shares state, as long as you unpickle it using a
corresponding number of load() calls.

> 2) Is memo an intentionally exposed, supported part of the
> Pickler/Unpickler API, despite the lack of documentation and tests?

I think the 2.4 and later docs make it clear that it's
no longer considered part of the public API, if it ever
was.

If seeding the memo is considered a legitimate need, an
API could be provided for doing that.

-- 
Greg