[Python-bugs-list] [ python-Bugs-451547 ] pickle / cPickle can't load lambdas

noreply@sourceforge.net noreply@sourceforge.net
Wed, 30 Oct 2002 15:47:17 -0800


Bugs item #451547, was opened at 2001-08-16 09:33
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=451547&group_id=5470

Category: Python Library
Group: None
>Status: Closed
Resolution: Fixed
Priority: 3
Submitted By: Gordon B. McMillan (gmcm)
Assigned to: Guido van Rossum (gvanrossum)
Summary: pickle / cPickle can't load lambdas

Initial Comment:
pickle and cPickle will happily dump a lambda, but on 
load, both report:
SystemError: Failed to import class <lambda> from 
module __main__

Seen on Py 2.1 & 1.5.2

>>> f = lambda x: x in (1,2,3)
>>> o = cPickle.dumps(f)
>>> f2 = cPickle.loads(o)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
SystemError: Failed to import class <lambda> from 
module __main__
>>> o = pickle.dumps(f)
>>> f2 = pickle.loads(o)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "c:\python21\lib\pickle.py", line 951, in loads
    return Unpickler(file).load()
  File "c:\python21\lib\pickle.py", line 567, in load
    dispatch[key](self)
  File "c:\python21\lib\pickle.py", line 780, in 
load_global
    klass = self.find_class(module, name)
  File "c:\python21\lib\pickle.py", line 790, in 
find_class
    raise SystemError, \
SystemError: Failed to import class <lambda> from 
module __main__



----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-10-30 18:47

Message:
Logged In: YES 
user_id=6380

The problem really lies with reload(); pickle is just one of
the many victims. E.g. this fails too:

import foo
x = foo.C()
assert isinstance(x, foo.C) # OK
reload(foo)
assert isinstance(x, foo.C) # Fails

If you have a suggestion for how to warn about reload(),
please submit a *new* bug report with Category set to
Documentation.

----------------------------------------------------------------------

Comment By: Dave Cole (davecole)
Date: 2002-10-29 18:52

Message:
Logged In: YES 
user_id=28658

I realise that there are lots of things that you can do to
prevent or confuse introspection working in the pickle
module.  You can dynamically create classes on the fly too.
 It just seems a shame that it is possible to prevent the
most basic of class definitions from working in the pickler.

I can program around the problem but it took the better part
of a day to work out why I was experiencing the problem.

Maybe the correct bug fix is not to change code but to
improve the documentation of the pickle module.


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-10-29 17:21

Message:
Logged In: YES 
user_id=6380

Well, the problem is that there are lots of other cases that
the fix also catches. For example:

>>> class C:
          class C:
              pass
    
>>> a = C.C()
>>> pickle.dumps(a,1)

I tend to think that catching this is more important than
handling reload() -- since reload() has lots of other
problems like this, it's better to blame reload() and not
try to fix it at the cost of other situations.


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-10-28 08:26

Message:
Logged In: YES 
user_id=6380

Reopening so I won't forget about this.

----------------------------------------------------------------------

Comment By: Dave Cole (davecole)
Date: 2002-10-25 09:44

Message:
Logged In: YES 
user_id=28658

I seem to be suffering from some unintentional consequences
of this fix.  I do not think that the pickle should fail in
the follwing case.  Wouldn't it be a better idea to just
check for a successful of the pickled class instead of
requiring the imported class to be stored at the same memory
location?

>>> import pickle, copy
>>> o = copy._EmptyClass()
>>> reload(copy)
<module 'copy' from '/usr/lib/python2.2/copy.pyc'>
>>> pickle.dumps(o, 1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.2/pickle.py", line 978, in dumps
    Pickler(file, bin).dump(object)
  File "/usr/lib/python2.2/pickle.py", line 115, in dump
    self.save(object)
  File "/usr/lib/python2.2/pickle.py", line 225, in save
    f(self, object)
  File "/usr/lib/python2.2/pickle.py", line 477, in save_inst
    save(cls)
  File "/usr/lib/python2.2/pickle.py", line 225, in save
    f(self, object)
  File "/usr/lib/python2.2/pickle.py", line 524, in save_global
    raise PicklingError(
pickle.PicklingError: Can't pickle <class copy._EmptyClass
at 0x819493c>: it's not the same object as copy._EmptyClass


----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-18 17:23

Message:
Logged In: YES 
user_id=6380

I applied Gordon's patch, so this can be closed now.


----------------------------------------------------------------------

Comment By: Gordon B. McMillan (gmcm)
Date: 2001-08-17 17:45

Message:
Logged In: YES 
user_id=4923

Patch # 452239 is my attempt to patch cPickle.c to match 
Guido's patch to pickle.py.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-17 14:51

Message:
Logged In: YES 
user_id=6380

I figured it out.

In save_global(), I now try to import the module and then
extract the name from the module; this must give us the same
object. If it doesn't, raise PicklingError.

Checked in as pickle.py rev 1.50.  I can't close this yet
because cPickle needs
a similar patch.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-16 12:31

Message:
Logged In: YES 
user_id=6380

I guess what you're missing is that pickling a function
doesn't pickle the bytecode! It pickles the name instead. So
a reference to "foo.bar" is pickled as "foo.bar", and the
unpickler imports bar from foo.

----------------------------------------------------------------------

Comment By: Gordon B. McMillan (gmcm)
Date: 2001-08-16 12:03

Message:
Logged In: YES 
user_id=4923

Hmm. In the (simplistic) case I was trying, I can't see any 
significant difference between the lambda and the 
equivalent function (names differ, and the func has 2 
appended & apparently unreachable bytecodes, but otherwise 
the func_* and func_code.co_* attributes match). So what am 
I missing?

Lowering priority - I can live without it easily enough.

But if lambda's won't load, they probably shouldn't dump.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-16 11:11

Message:
Logged In: YES 
user_id=6380

I think I'll have to close this with a won't fix, or "then
don't do that" resolution.

The problem is that whenever a function or class is pickled,
pickle must accept on blind faith that it can also be
unpickled. How would you check that this is indeed the case?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=451547&group_id=5470