importing class objects from a pickled file

[Owen Jacobson]

On 2011-05-03 20:18:33 -0400, Catherine Moroney said:

> Hello,
> 
> I have an object of class X that I am writing to a pickled file.  The 
> pickling part goes fine, but I am having some problems reading the 
> object back out, as I get complaints about "unable to import module X".
> 
> The only way I have found around it is to run the read-file code out of 
> the same directory that contains the X.py file, but this is obviously 
> not a portable way of doing things.
> 
> Even when I put statements into the code such as "from Y.X import X"
> where Y is the name of the python package that contains the X,py file,
> the import statement works, but I am still unable to read the object 
> from the pickled file, running into the same "unable to import module X"
> error.
> 
> Am I explaining myself properly?  Why doesn't the code that loads the
> object from the pickled file work unless I am sitting in the same directory?

That's not the actual requirement.

The pickled representation of an instance of a user-defined class contains:

	* the name of the class
	* the name of the class's module
	* the object's attributes, pickled recursively

You can have a look at the representation using protocol zero, which is 
printable and not terribly hard to read. Here's an example, lightly 
annotated:

                >>> import pickle
                >>> class Example(object):
                ...   def __init__(self, x):
                ...     self.x = x
                ...
                >>> one = Example(1)
                >>> print pickle.dumps(one, 0)
                ccopy_reg
                _reconstructor
                p0
module name     (c__main__
class name      Example
                p1
                c__builtin__
                object
                p2
                Ntp3
                Rp4
                (dp5
field name      S'x'
                p6
field value     I1
                sb.
                >>>

Obviously, since the pickled representation only contains the *name and 
location* of the class and not the class itself, you need to have the 
definition of the class handy when unpickling an object. This means 
that the module that defines the class must be on the module search 
path - either because it's in the current directory (or a package in 
the current directory, or..), or because it's in PYTHONPATH, or because 
it's somewhere else on sys.path.

> The code that writes the pickled file has the statement
> "from Y.X import X" statement" at the top, as does the reading code, but
> even though that import statement succeeds, the read still fails with 
> the import error.

The unpickle code uses something morally equivalent to __import__ to 
convert the module name in the pickled data stream into a module 
object. The symbols defined in the module you called pickle.load from 
don't matter at all, just like any other function defined in a separate 
module.

Pickle isn't meant for transmitting data between two unrelated 
programs. It's designed to be a low-development-effort way to preserve 
and restore objects for a single program or a group of related programs 
that share libraries. If you want to share data between unrelated 
programs, use something a little more interoperable - json, xml, or 
some other program-agnostic representation.

-o

(And if you were thinking of using pickle over the internet, forget it: 
the pickle encoding is a small stack-based programming language in its 
own right, and can be used to invoke semi-arbitrary module-scoped 
funtions.)