BaseException pickle issue
Some subclasses of Exception are no longer pickleable in Python 2.5. An example: class A(Exception): def __init__(self, foo): self.foo = foo The key problem here is if you do not somehow set self.args to the correct arguments (via Exception.__init__ or setting self.args directly), you will get a TypeError when you try to unpickle it: TypeError: __init__() takes exactly 2 arguments (1 given) I do not think this is unusual. I found a few examples in Python's standard library that have this problem: subprocess.CalledProcessError HTMLParser.HTMLParseError httplib.UnknownProtocol, httplib.IncompleteRead, httplib.BadStatusLine optparse.OptParseError pickle._Stop and on and on... Does anyone have any thoughts about this? Is it a bug? I can imagine one could argue that exceptions should call the base __init__ method to properly set args, but there are so many exceptions out there that do not do this that it would be very difficult to track them all down. I removed the __reduce__ and __setstate__ methods from BaseException and everything seems to just work. Pickling/unpickling works for all protocols whether or not you set self.args. Is this an appropriate solution? I'm not sure what the motivation for having these methods is. -Eric
Eric Huss wrote:
Some subclasses of Exception are no longer pickleable in Python 2.5. An example:
[snip]
Does anyone have any thoughts about this? Is it a bug?
I can imagine one could argue that exceptions should call the base __init__ method to properly set args, but there are so many exceptions out there that do not do this that it would be very difficult to track them all down.
I removed the __reduce__ and __setstate__ methods from BaseException and everything seems to just work. Pickling/unpickling works for all protocols whether or not you set self.args. Is this an appropriate solution? I'm not sure what the motivation for having these methods is.
I think that this is a bug, but removing those methods is not the right solution. The __reduce__ method is needed because builtin exceptions don't store their attributes in the __dict__ anymore; if you remove it, then those attributes will be lost during pickling. The __setstate__ method was added to help with unpickling old exceptions, which did store all their attributes in the __dict__. See this patch for details: http://www.python.org/sf/1498571 A better solution would be to move the initial args attribute assignment to BaseException.__new__. See this patch: http://www.python.org/sf/1692335 Could you check if that fixes your problem? Ziga
I think that this is a bug, but removing those methods is not the right solution. The __reduce__ method is needed because builtin exceptions don't store their attributes in the __dict__ anymore; if you remove it, then those attributes will be lost during pickling. The __setstate__ method was added to help with unpickling old exceptions, which did store all their attributes in the __dict__. See this patch for details:
http://www.python.org/sf/1498571
A better solution would be to move the initial args attribute assignment to BaseException.__new__. See this patch:
http://www.python.org/sf/1692335
Could you check if that fixes your problem?
Thanks, it certainly does! I wrote a very extensive unittest to try out various permutations. I'm not sure if there's an expectation that pickled strings of user-defined exceptions from previous versions work in newer versions. If you define your own __init__ method, and try to load a pickled string from 2.4, I still get a TypeError when trying to unpickle it. I will send you my test file directly. This isn't a problem for me since we don't store pickled exceptions. -Eric
participants (2)
-
Eric Huss
-
Žiga Seilnacht