[Python-Dev] Pickling problems are hard to debug
Gary Poster
gary at modernsongs.com
Sun Mar 26 04:07:20 CEST 2006
On Mar 25, 2006, at 8:13 PM, Greg Ewing wrote:
> There seems to be a need for better diagnostics
> when pickle encounters something that can't be
> pickled.
>
> Recently when attempting to pickle a rather
> large and complicated data structure, I got
> the following incomprehensible message:
>
> cPickle.PicklingError: args[0]
> from __newobj__ args has the wrong class
>
> Trying again with protocol 1 instead of 2,
> I get
>
> TypeError: can't pickle function objects
>
> which I'm *guessing* is because somewhere I've
> tried to pickle a nested function or a bound
> method. But it still doesn't give me any idea
> *which* function I tried to pickle or where
> abouts it turns up in the data structure.
>
> Anyone have any ideas how the situation could
> be improved? At the very least, it could
> include some info about the type and identity
> of the offending object.
You are asking for ideas on how to change the pickle story to help.
However, just reading your issue, I thought I might have done a
debugging hack like this, at least for the protocol 1 traceback.
We'll assume that the error is more mysterious than what I've
manufactured here.
>>> import cPickle
>>> cPickle.dumps({'foo': lambda: 42})
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/Library/Frameworks/Python.framework/Versions/2.4/lib/
python2.4/copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects
>>> import copy_reg
>>> def debug(obj):
... import pdb; pdb.set_trace()
...
>>> import types
>>> copy_reg.pickle(types.FunctionType, debug)
>>> cPickle.dumps({'foo': lambda: 42})
--Return--
> <stdin>(2)debug()->None
(Pdb) p obj
<function <lambda> at 0x63230>
(Pdb) p obj.__module__
'__main__'
I also might have used pickle, rather than cPickle, to try and see
what happened, if that ended up being necessary.
I don't use protocol 2 much: that error message in particular looked
a bit difficult, and my hack might not be any help there. I agree
that it would be nice to have a better message there, in particular.
back to lurking...
Gary
More information about the Python-Dev
mailing list