[Python-Dev] How to suppress instance __dict__?

[David Abrahams]

Guido van Rossum <guido@python.org> writes:

>> Guido van Rossum <guido@python.org> writes:
>> >> > I think you could subclass the metaclass, override __new__, and delete
>> >> > the bogus __getstate__ from the type's __dict__.  Then you'll get the
>> >> > default pickling behavior which ignores slots; that should work just
>> >> > fine in your case. :-)
>
> [David]
>> >> Ooh, that's sneaky!  But I can't quite see how it works.  The error
>> >> message I quoted at the top about __getstate__ happens when you try to
>> >> pickle an instance of the class.  If I delete __getstate__ during
>> >> __new__, it won't be there for pickle to find when I try to do the
>> >> pickling.  What will keep it from inducing the same error?
>
> [Guido]
>> > Just try it.  There are many ways to customize pickling, and if
>> > __getstate__ doesn't exist, pickling is done differently.
>> 
>> Since this doesn't work:
>> 
>>     >>> d = type('d', (object,), { '__slots__' : ['foo'] } )
>>     >>> pickle.dumps(d())
>
> Um, you're changing the rules in the middle of the game.  You said you
> had an *empty* __slots__.  

I did.  I just stuck something in there so I could verify that things
were working in the expected way.

> My recommendation only applied to that case.  I also thought you
> were doing this from C, not from Python, but I may be mistaken.

You're not mistaken; Just like Python gives a productivity boost over
C/C++ for ordinary programming, I find I can learn a lot more about
the Python core in a short period of time by writing Python code than
by writing 'C' code, so I usually try that first.

>> I'm still baffled as to why this works:
>> 
>>     >>> class mc(type):
>>     ...     def __new__(self, *args):
>>     ...             x = type.__new__(self, *args)
>>     ...             del args[2]['__getstate__']
>
> Hm.  I don't think that x.__dict__ is args[2]; it's a copy, and
> deleting __getstate__ from the arguments doesn't make any difference
> to this example.

...as I discovered...

>>     ...             return x
>>     ...
>>     >>> c = mc('c', (object,), { '__slots__' : ['foo'], '__getstate__' : lambda self: tuple() } )
>
> Why are you passing a __getstate__ in?  The point was getting rid of
> the __getstate__ that type.__new__ inserts.

Because I didn't understand your intention, nor did I know that the
automatic __getstate__ was responsible for generating the error
message.  I thought the idea was to define a __getstate__, which is a
known way to suppress the error message, and then kill it in __new__.
I figured that pickle was looking for __getstate__ and when it wasn't
there but __slots__ was, rasing the exception.  This may explain why I
didn't see how the approach could work.

Now I understand what you meant.

>>     >>> pickle.dumps(c())
>>     'ccopy_reg\n_reconstructor\np0\n(c__main__\nc\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
>> 
>> especially since:
>> 
>>     >>> dir(d) == dir(c)
>>     1
>
> I think you have been testing something very different from what you
> think you did here.  dir(d) == dir(c) because they both have a
> __getstate__; but d.__getstate__ is a built-in that raises an
> exception, while c.__getstate__ is the lambda you passed in.

Yeah, I found that out below.

> And have you tried unpickling yet?  I expect it to fail.

Nope.

>> I don't see the logic in the source for object.__reduce__(), so where
>> is it?  OK, I see it in typeobject.c.  But now:
>> 
>>     >>> c.__getstate__
>>     <unbound method c.<lambda>>
>> 
>> OK, this seems to indicate that my attempt to remove __getstate__ from
>> the class __dict__ was a failure.  That explains why pickling c works,
>> but not why you suggested that I remove __getstate__ inside of
>> __new__.  Did you mean for me to do something different?
>
> Yes.  I was assuming you'd do this at the C level.  To do what I
> suggested in Python, I think you'd have to write this:
>
>     class M(type):
>       def __new__(cls, name, bases, dict):
>       C = type.__new__(cls, name, bases, dict)
>       del C.__getstate__
>       return C

I tried to get too fancy with del C.__dict__['__getstate__'] which
didn't work of course.  Anyway, thanks for spelling it out for me.  I
think I understand everything now.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com