UnpicklingError: NEWOBJ class argument isn't a type object

[skunkwerk]

On Monday, July 8, 2013 12:45:55 AM UTC-7, Peter Otten wrote:
> skunkwerk wrote:
> 
> 
> 
> > Hi,
> 
> >   I'm using a custom pickler that replaces any un-pickleable objects (such
> 
> >   as sockets or files) with a string representation of them, based on the
> 
> >   code from Shane Hathaway here:
> 
> > http://stackoverflow.com/questions/4080688/python-pickling-a-dict-with-
> 
> some-unpicklable-items
> 
> > 
> 
> > It works most of the time, but when I try to unpickle a Django
> 
> > HttpResponse, I get the following error: UnpicklingError: NEWOBJ class
> 
> > argument isn't a type object
> 
> > 
> 
> > I have no clue what the error actually means.  If it pickles okay, why
> 
> > should it not be able to unpickle?  Any ideas?
> 
> 
> 
> A simple way to provoke the error is to rebind the name referring to the 
> 
> class of the pickled object:
> 
> 
> 
> >>> import cPickle
> 
> >>> class A(object): pass
> 
> ... 
> 
> >>> p = cPickle.dumps(A(), -1)
> 
> >>> cPickle.loads(p)
> 
> <__main__.A object at 0x7fce7bb58c50>
> 
> >>> A = 42
> 
> >>> cPickle.loads(p)
> 
> Traceback (most recent call last):
> 
>   File "<stdin>", line 1, in <module>
> 
> cPickle.UnpicklingError: NEWOBJ class argument isn't a type object
> 
> 
> 
> You may be doing something to that effect.

Hey Peter,
  I tried unpickling even from another file with no other code in it, but came up with the same error - so I don't think it's a rebinding issue.

But I got the error to disappear when I removed the "hasattr(obj, '__getstate__')" from this line of code in the persistent_id function:
if not hasattr(obj, '__getstate__') and isinstance(obj,(basestring, bool, int, long, float, complex, tuple, list, set, dict)):
    return ["filtered:%s" % type(obj)]

When I do that, I get a few more FilteredObjects in the result, for things like:
<class 'django.core.handlers.wsgi.WSGIRequest'>
<class 'MySQLdb.connections.Connection'>

I figured these classes must have __getstate__ methods which leads to them being pickled without a persistent_id (it turns out they actually have __repr__ methods).

So these classes get pickled fine, but run into problems when trying to unpickle them.  I understand why ImportErrors would happen if the necessary modules haven't been loaded, but this NEWOBJ error is still kind of mystifying.
I guess I just won't pickle any classes for now, if unpickling them is going to be dicey.

thanks for the help guys,
imran