[Python-Dev] Pickle alternative in stdlib (Was: On breaking modules into packages)

[Nick Coghlan]

On Fri, Nov 5, 2010 at 10:56 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Nick Coghlan wrote:
>
>> As a tool for communicating between different instances of the *same*
>> version of Python though, pickle is fine.
>
> I'm using pickle to pass a list and dict of floats and strings from Python
> 2.6 to 3.1. I've never had any problems with it. Am I living in a state of
> sin or is that okay?

Builtins are generally fine, and we do try reasonably hard to keep the
pickle formats properly compatible across versions. It's corner cases
(like pickling unittest objects) that may sometimes break, since
pickle implicitly depends on things that *should* be disregarded as
implementation details. Specifically, without explicit directions to
do otherwise, pickle encodes objects based on what they *are*, which
may include implementation details, such as optional acceleration
modules, platform specific variants of classes returned by a factory
function, etc. Technically such things are bugs in an object's
pickling support, but they're *really* non-obvious (and genuinely
harmless in most cases).

As I see it, there are at least 3 levels of pickling support:
1. Complete, version independent (implementation details are weeded
out from the pickle, or deliberately kept the same across versions to
preserve pickle compatibility)
2. Partial, potentially version dependent (pickles may be infected
with implementation details that affect cross-version compatibility if
they happen to change)
3. None (can't even pickle it in the first place)

Builtins are in category 1, but there are plenty of things in the
standard library (like unittest classes) that rely on default pickling
behaviour and hence fit into category 2 (we just very, very rarely
move anything around, so such classes may as well be in category 1
most of the time).

Notably, this mostly causes problems when reading pickles generated
with a *new* version of Python in an *old* version. When going the
other way, we can adjust the unpickling process to cope with any
discrepancies (for the "relying on implementation details case",
usually by the simple expedient of keeping both sets of names around).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia