[Python-bugs-list] [ python-Bugs-451547 ] pickle / cPickle can't load lambdas
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 29 Oct 2002 15:52:21 -0800
Bugs item #451547, was opened at 2001-08-16 13:33
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=451547&group_id=5470
Category: Python Library
Group: None
Status: Open
Resolution: Fixed
Priority: 3
Submitted By: Gordon B. McMillan (gmcm)
Assigned to: Guido van Rossum (gvanrossum)
Summary: pickle / cPickle can't load lambdas
Initial Comment:
pickle and cPickle will happily dump a lambda, but on
load, both report:
SystemError: Failed to import class <lambda> from
module __main__
Seen on Py 2.1 & 1.5.2
>>> f = lambda x: x in (1,2,3)
>>> o = cPickle.dumps(f)
>>> f2 = cPickle.loads(o)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
SystemError: Failed to import class <lambda> from
module __main__
>>> o = pickle.dumps(f)
>>> f2 = pickle.loads(o)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "c:\python21\lib\pickle.py", line 951, in loads
return Unpickler(file).load()
File "c:\python21\lib\pickle.py", line 567, in load
dispatch[key](self)
File "c:\python21\lib\pickle.py", line 780, in
load_global
klass = self.find_class(module, name)
File "c:\python21\lib\pickle.py", line 790, in
find_class
raise SystemError, \
SystemError: Failed to import class <lambda> from
module __main__
----------------------------------------------------------------------
Comment By: Dave Cole (davecole)
Date: 2002-10-29 23:52
Message:
Logged In: YES
user_id=28658
I realise that there are lots of things that you can do to
prevent or confuse introspection working in the pickle
module. You can dynamically create classes on the fly too.
It just seems a shame that it is possible to prevent the
most basic of class definitions from working in the pickler.
I can program around the problem but it took the better part
of a day to work out why I was experiencing the problem.
Maybe the correct bug fix is not to change code but to
improve the documentation of the pickle module.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-10-29 22:21
Message:
Logged In: YES
user_id=6380
Well, the problem is that there are lots of other cases that
the fix also catches. For example:
>>> class C:
class C:
pass
>>> a = C.C()
>>> pickle.dumps(a,1)
I tend to think that catching this is more important than
handling reload() -- since reload() has lots of other
problems like this, it's better to blame reload() and not
try to fix it at the cost of other situations.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2002-10-28 13:26
Message:
Logged In: YES
user_id=6380
Reopening so I won't forget about this.
----------------------------------------------------------------------
Comment By: Dave Cole (davecole)
Date: 2002-10-25 13:44
Message:
Logged In: YES
user_id=28658
I seem to be suffering from some unintentional consequences
of this fix. I do not think that the pickle should fail in
the follwing case. Wouldn't it be a better idea to just
check for a successful of the pickled class instead of
requiring the imported class to be stored at the same memory
location?
>>> import pickle, copy
>>> o = copy._EmptyClass()
>>> reload(copy)
<module 'copy' from '/usr/lib/python2.2/copy.pyc'>
>>> pickle.dumps(o, 1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.2/pickle.py", line 978, in dumps
Pickler(file, bin).dump(object)
File "/usr/lib/python2.2/pickle.py", line 115, in dump
self.save(object)
File "/usr/lib/python2.2/pickle.py", line 225, in save
f(self, object)
File "/usr/lib/python2.2/pickle.py", line 477, in save_inst
save(cls)
File "/usr/lib/python2.2/pickle.py", line 225, in save
f(self, object)
File "/usr/lib/python2.2/pickle.py", line 524, in save_global
raise PicklingError(
pickle.PicklingError: Can't pickle <class copy._EmptyClass
at 0x819493c>: it's not the same object as copy._EmptyClass
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-18 21:23
Message:
Logged In: YES
user_id=6380
I applied Gordon's patch, so this can be closed now.
----------------------------------------------------------------------
Comment By: Gordon B. McMillan (gmcm)
Date: 2001-08-17 21:45
Message:
Logged In: YES
user_id=4923
Patch # 452239 is my attempt to patch cPickle.c to match
Guido's patch to pickle.py.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-17 18:51
Message:
Logged In: YES
user_id=6380
I figured it out.
In save_global(), I now try to import the module and then
extract the name from the module; this must give us the same
object. If it doesn't, raise PicklingError.
Checked in as pickle.py rev 1.50. I can't close this yet
because cPickle needs
a similar patch.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-16 16:31
Message:
Logged In: YES
user_id=6380
I guess what you're missing is that pickling a function
doesn't pickle the bytecode! It pickles the name instead. So
a reference to "foo.bar" is pickled as "foo.bar", and the
unpickler imports bar from foo.
----------------------------------------------------------------------
Comment By: Gordon B. McMillan (gmcm)
Date: 2001-08-16 16:03
Message:
Logged In: YES
user_id=4923
Hmm. In the (simplistic) case I was trying, I can't see any
significant difference between the lambda and the
equivalent function (names differ, and the func has 2
appended & apparently unreachable bytecodes, but otherwise
the func_* and func_code.co_* attributes match). So what am
I missing?
Lowering priority - I can live without it easily enough.
But if lambda's won't load, they probably shouldn't dump.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-08-16 15:11
Message:
Logged In: YES
user_id=6380
I think I'll have to close this with a won't fix, or "then
don't do that" resolution.
The problem is that whenever a function or class is pickled,
pickle must accept on blind faith that it can also be
unpickled. How would you check that this is indeed the case?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=451547&group_id=5470