Pickling problems are hard to debug
There seems to be a need for better diagnostics when pickle encounters something that can't be pickled. Recently when attempting to pickle a rather large and complicated data structure, I got the following incomprehensible message: cPickle.PicklingError: args[0] from __newobj__ args has the wrong class Trying again with protocol 1 instead of 2, I get TypeError: can't pickle function objects which I'm *guessing* is because somewhere I've tried to pickle a nested function or a bound method. But it still doesn't give me any idea *which* function I tried to pickle or where abouts it turns up in the data structure. Anyone have any ideas how the situation could be improved? At the very least, it could include some info about the type and identity of the offending object. Greg
On Mar 25, 2006, at 8:13 PM, Greg Ewing wrote:
There seems to be a need for better diagnostics when pickle encounters something that can't be pickled.
Recently when attempting to pickle a rather large and complicated data structure, I got the following incomprehensible message:
cPickle.PicklingError: args[0] from __newobj__ args has the wrong class
Trying again with protocol 1 instead of 2, I get
TypeError: can't pickle function objects
which I'm *guessing* is because somewhere I've tried to pickle a nested function or a bound method. But it still doesn't give me any idea *which* function I tried to pickle or where abouts it turns up in the data structure.
Anyone have any ideas how the situation could be improved? At the very least, it could include some info about the type and identity of the offending object.
import cPickle cPickle.dumps({'foo': lambda: 42}) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/Library/Frameworks/Python.framework/Versions/2.4/lib/
You are asking for ideas on how to change the pickle story to help. However, just reading your issue, I thought I might have done a debugging hack like this, at least for the protocol 1 traceback. We'll assume that the error is more mysterious than what I've manufactured here. python2.4/copy_reg.py", line 69, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle function objects
import copy_reg def debug(obj): ... import pdb; pdb.set_trace() ... import types copy_reg.pickle(types.FunctionType, debug) cPickle.dumps({'foo': lambda: 42}) --Return-- <stdin>(2)debug()->None (Pdb) p obj
(Pdb) p obj.__module__ '__main__'
I also might have used pickle, rather than cPickle, to try and see what happened, if that ended up being necessary. I don't use protocol 2 much: that error message in particular looked a bit difficult, and my hack might not be any help there. I agree that it would be nice to have a better message there, in particular. back to lurking... Gary
Greg Ewing wrote:
Anyone have any ideas how the situation could be improved?
As always: on a case-by-case basis. If you find a specific case where you think the diagnosis should be better, make it better for this case. Perhaps some generalization arises while doing so, but if not, atleast this specific case gets improved. While doing so, try to cover similar cases in the process, e.g. different pickle protocols, and different pickle implementations. Regards, Martin
participants (3)
-
"Martin v. Löwis"
-
Gary Poster
-
Greg Ewing