[Tutor] portability of pickle and shelve across platforms and different python versions?

[Steven D'Aprano]

On Thu, 6 May 2010 09:12:24 am Garry Willgoose wrote:

> How portable are files containing pickle'd and shelve'd data? I'm
> thinking issues like simply O/S portability, through
> big-end/little-end hardware (for floats/integers which I use a lot),
> and then for unicode/non-unicode string,  64/32 bit and V2.6/V3.1
> implementations of python. Does the version of the encoder in pickle
> make any difference for this? One post I've seen suggests that as
> long as the file is opened binary (ie. 'b') all should be well for
> platform independence.

Technically, reading the file isn't a matter of pickle, but a matter of 
the operating system not mangling the contents of the file before it 
reaches pickle. I would expect that text pickles (protocol 0) would be 
just fine with opening the file in text mode, but I haven't tried it.

Because Python makes no guarantees about floats, but is just a thin 
wrapper around your platform's C floating point library, you may find 
occasional differences when pickling floats. E.g. I wouldn't trust 
pickling NANs and INFs to be platform independent unless the 
documentation explicitly says so.

Unfortunately transferring floats from one platform to another is a hard 
problem: given a float x on platform A, there's no guarantee that x is 
even representable on platform B. You can make stronger promises about 
transferring floats if you know both platforms use IEEE floats, 
although the C maths libraries differ in their handling of subnormals, 
overflow, NANs, INFs, and signed zeroes. If these features are 
important to you, you've probably already discovered that your 
calculations differ on platforms A and B unless you're using a 
dedicated numeric library that doesn't rely on the native C maths 
routines. If this is gobbledygook to you, then don't worry about it, it 
should Just Work well enough that you won't notice the difference.



> My core question if I give a pickled file to somebody else can i
> guarantee they can read/load it OK. The other person will be using
> exactly the same python code to open it as used to create it.

By default, pickling uses protocol 0, which uses the repr() of objects. 
Nobody can *guarantee* platform independence, because you might feed 
Python an object like this:

class Silly:
    def __init__(self, arg):
        self.arg = arg
    def __repr__(self):
        if sys.platform == 'posix':
            # I hate Linux.
            return "Screw you hippies, I'm going home!"
        return "Silly(%r)" % self.arg


which will break pickling. Similarly if your class has a __setstate__ 
method which does something stupid. Python is a "consenting adults" 
language: if you want to shoot yourself in the foot by writing broken 
code, Python doesn't try to stop you.

But for built-ins, with the possible exception of floats depending on 
the specific platforms in question, you should be safe.



-- 
Steven D'Aprano