Re: [Python-Dev] How to suppress instance __dict__?

Guido van Rossum <guido@python.org> writes:
Since this doesn't work: >>> d = type('d', (object,), { '__slots__' : ['foo'] } ) >>> pickle.dumps(d()) I'm still baffled as to why this works: >>> class mc(type): ... def __new__(self, *args): ... x = type.__new__(self, *args) ... del args[2]['__getstate__'] ... return x ... >>> c = mc('c', (object,), { '__slots__' : ['foo'], '__getstate__' : lambda self: tuple() } ) >>> pickle.dumps(c()) 'ccopy_reg\n_reconstructor\np0\n(c__main__\nc\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.' especially since: >>> dir(d) == dir(c) 1 I don't see the logic in the source for object.__reduce__(), so where is it? OK, I see it in typeobject.c. But now: >>> c.__getstate__ <unbound method c.<lambda>> OK, this seems to indicate that my attempt to remove __getstate__ from the class __dict__ was a failure. That explains why pickling c works, but not why you suggested that I remove __getstate__ inside of __new__. Did you mean for me to do something different? I note that c's __slots__ aren't pickled at all, which I guess was the point of the __getstate__ requirement: >>> x = c() >>> x.foo = 1 >>> pickle.dumps(x) == pickle.dumps(c()) 1 Fortunately, in our case the __slots__ are empty so it doesn't matter. -- Dave Abrahams Boost Consulting www.boost-consulting.com

[David]
[Guido]
Um, you're changing the rules in the middle of the game. You said you had an *empty* __slots__. My recommendation only applied to that case. I also thought you were doing this from C, not from Python, but I may be mistaken.
Hm. I don't think that x.__dict__ is args[2]; it's a copy, and deleting __getstate__ from the arguments doesn't make any difference to this example.
Why are you passing a __getstate__ in? The point was getting rid of the __getstate__ that type.__new__ inserts.
I think you have been testing something very different from what you think you did here. dir(d) == dir(c) because they both have a __getstate__; but d.__getstate__ is a built-in that raises an exception, while c.__getstate__ is the lambda you passed in. And have you tried unpickling yet? I expect it to fail.
Yes. I was assuming you'd do this at the C level. To do what I suggested in Python, I think you'd have to write this: class M(type): def __new__(cls, name, bases, dict): C = type.__new__(cls, name, bases, dict) del C.__getstate__ return C
Right. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:
I did. I just stuck something in there so I could verify that things were working in the expected way.
My recommendation only applied to that case. I also thought you were doing this from C, not from Python, but I may be mistaken.
You're not mistaken; Just like Python gives a productivity boost over C/C++ for ordinary programming, I find I can learn a lot more about the Python core in a short period of time by writing Python code than by writing 'C' code, so I usually try that first.
...as I discovered...
Because I didn't understand your intention, nor did I know that the automatic __getstate__ was responsible for generating the error message. I thought the idea was to define a __getstate__, which is a known way to suppress the error message, and then kill it in __new__. I figured that pickle was looking for __getstate__ and when it wasn't there but __slots__ was, rasing the exception. This may explain why I didn't see how the approach could work. Now I understand what you meant.
Yeah, I found that out below.
And have you tried unpickling yet? I expect it to fail.
Nope.
I tried to get too fancy with del C.__dict__['__getstate__'] which didn't work of course. Anyway, thanks for spelling it out for me. I think I understand everything now. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Dave Abrahams wrote:
Guido van Rossum wrote:
Hi, Ok, I'm lost. Please be easy with me, I'm still learning the C API interfacing with Python :) Here's what I have so far. Emulating the desired behavior in Python, I can do: class EnumMeta(type): def __new__(cls, name, bases, dict): C = type.__new__(cls, name, bases, dict) del C.__getstate__ return C class Enum(int): __metaclass__ = EnumMeta __slots__ = () x = Enum(1964) print x import pickle print "SAVING" out_x = pickle.dumps(x) print "LOADING" xl = pickle.loads(out_x) print xl I'm trying to rewrite this in C/C++ with the intent to patch Boost.Python to allow pickling on enums. I took on this task to learn more about the low level details of Python C interfacing. So far, I have implemented EnumMeta in C that does not override anything yet and installed that as the metaclass of Enum. I was wondering... Is there some C code somewhere that I can see that implements some sort of meta-stuff? I read PEP253 and 253 and "Unifying Types and Classes in Python 2.2". The examples there (specifically the class autoprop) is written in Python. I tried searching for examples in C from the current CVS snapsot of 2.3 but I failed in doing so. I'm sure it's there, but I don't know where to find. To be specific, I'm lost in trying to implement tp_new of PyTypeObject. How do I call the default tp_new for metaclasses? TIA, -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net

Hi, Joel -- I don't think this is more than marginally appropriate for python-dev, and probably we shouldn't bother Guido about it until I've failed to help you first. Everybody else can ignore the rest of this message unless they have a sick fascination with Boost.Python... "Joel de Guzman" <joel@boost-consulting.com> writes:
We have some in Boost.Python already, and I'm about to check in some more to implement static data members.
Actually there are very few metaclasses in Python proper. AFAIK, PyType_Type is the only metaclass in the core.
To be specific, I'm lost in trying to implement tp_new of PyTypeObject. How do I call the default tp_new for metaclasses?
PyTypeObject.tp_new( /*args here*/ ) should work. HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com

[David]
[Guido]
Um, you're changing the rules in the middle of the game. You said you had an *empty* __slots__. My recommendation only applied to that case. I also thought you were doing this from C, not from Python, but I may be mistaken.
Hm. I don't think that x.__dict__ is args[2]; it's a copy, and deleting __getstate__ from the arguments doesn't make any difference to this example.
Why are you passing a __getstate__ in? The point was getting rid of the __getstate__ that type.__new__ inserts.
I think you have been testing something very different from what you think you did here. dir(d) == dir(c) because they both have a __getstate__; but d.__getstate__ is a built-in that raises an exception, while c.__getstate__ is the lambda you passed in. And have you tried unpickling yet? I expect it to fail.
Yes. I was assuming you'd do this at the C level. To do what I suggested in Python, I think you'd have to write this: class M(type): def __new__(cls, name, bases, dict): C = type.__new__(cls, name, bases, dict) del C.__getstate__ return C
Right. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:
I did. I just stuck something in there so I could verify that things were working in the expected way.
My recommendation only applied to that case. I also thought you were doing this from C, not from Python, but I may be mistaken.
You're not mistaken; Just like Python gives a productivity boost over C/C++ for ordinary programming, I find I can learn a lot more about the Python core in a short period of time by writing Python code than by writing 'C' code, so I usually try that first.
...as I discovered...
Because I didn't understand your intention, nor did I know that the automatic __getstate__ was responsible for generating the error message. I thought the idea was to define a __getstate__, which is a known way to suppress the error message, and then kill it in __new__. I figured that pickle was looking for __getstate__ and when it wasn't there but __slots__ was, rasing the exception. This may explain why I didn't see how the approach could work. Now I understand what you meant.
Yeah, I found that out below.
And have you tried unpickling yet? I expect it to fail.
Nope.
I tried to get too fancy with del C.__dict__['__getstate__'] which didn't work of course. Anyway, thanks for spelling it out for me. I think I understand everything now. -- Dave Abrahams Boost Consulting www.boost-consulting.com

Dave Abrahams wrote:
Guido van Rossum wrote:
Hi, Ok, I'm lost. Please be easy with me, I'm still learning the C API interfacing with Python :) Here's what I have so far. Emulating the desired behavior in Python, I can do: class EnumMeta(type): def __new__(cls, name, bases, dict): C = type.__new__(cls, name, bases, dict) del C.__getstate__ return C class Enum(int): __metaclass__ = EnumMeta __slots__ = () x = Enum(1964) print x import pickle print "SAVING" out_x = pickle.dumps(x) print "LOADING" xl = pickle.loads(out_x) print xl I'm trying to rewrite this in C/C++ with the intent to patch Boost.Python to allow pickling on enums. I took on this task to learn more about the low level details of Python C interfacing. So far, I have implemented EnumMeta in C that does not override anything yet and installed that as the metaclass of Enum. I was wondering... Is there some C code somewhere that I can see that implements some sort of meta-stuff? I read PEP253 and 253 and "Unifying Types and Classes in Python 2.2". The examples there (specifically the class autoprop) is written in Python. I tried searching for examples in C from the current CVS snapsot of 2.3 but I failed in doing so. I'm sure it's there, but I don't know where to find. To be specific, I'm lost in trying to implement tp_new of PyTypeObject. How do I call the default tp_new for metaclasses? TIA, -- Joel de Guzman joel at boost-consulting.com http://www.boost-consulting.com http://spirit.sf.net

Hi, Joel -- I don't think this is more than marginally appropriate for python-dev, and probably we shouldn't bother Guido about it until I've failed to help you first. Everybody else can ignore the rest of this message unless they have a sick fascination with Boost.Python... "Joel de Guzman" <joel@boost-consulting.com> writes:
We have some in Boost.Python already, and I'm about to check in some more to implement static data members.
Actually there are very few metaclasses in Python proper. AFAIK, PyType_Type is the only metaclass in the core.
To be specific, I'm lost in trying to implement tp_new of PyTypeObject. How do I call the default tp_new for metaclasses?
PyTypeObject.tp_new( /*args here*/ ) should work. HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com
participants (3)
-
David Abrahams
-
Guido van Rossum
-
Joel de Guzman