Hooking into super() attribute resolution
Hi, Below is a very preliminary draft PEP for adding a special method that can be used to hook into the attribute resolution process of the super object. The primary usecase for using this special method are classes that perform custom logic in their __getattribute__ method, where the default behavior of super (peekin the the class __dict__) is not appropriate. The primary reason I wrote this proposal is PyObjC: it dynamicly looks up methods in its __getattribute__ and caches the result in the class __dict__, because of this super() will often not work correctly and therefore I'm currently shipping a custom subclass of super() that basicly contains an in-line implementation of the hook that would be used by PyObjC. I have a partial implementation of the hook system in issue 18181 and a PyObjC patch that uses it. The implementation currently does not contain tests, and I'm sure that I'll find edge cases that I haven't thought about yet when I add tests. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013 Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, otherwise it will peek in the class ``tp_dict``. The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, PyObject* name); The function should perform attribute lookup for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. In Python code -------------- A Python class can contain a definition for a method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(self, cls, name): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? References ========== * `Issue 18181`_ contains a prototype implementation The prototype uses different names than this proposal. Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super
I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. (Apologies for the HTML top-post) Sent from my Windows Phone ________________________________ From: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Sent: 7/6/2013 0:47 To: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Cc: python-dev@python.org Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Hooking into super() attribute resolution I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.co...
On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower@microsoft.com> wrote:
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?
What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).
And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.
__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald
(Apologies for the HTML top-post)
I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.
The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done). I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal. I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal. AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable. (The proxy objects idea was a red herring, sorry :) ) Steve Sent from my Windows Phone ________________________________ From: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Sent: 7/7/2013 12:37 To: Steve Dower<mailto:Steve.Dower@microsoft.com> Cc: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Hooking into super() attribute resolution On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower@microsoft.com> wrote:
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?
What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).
And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.
__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald
(Apologies for the HTML top-post)
I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.
On 8 Jul, 2013, at 17:19, Steve Dower <Steve.Dower@microsoft.com> wrote:
The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done).
The signature is as complex as it is to be able to call descr.__get__ with the correct arguments. I ended up with the current signature when I added __getattribute_super__ to object and removed the tp_dict peeking code from super's tp_getattro. A way to get a simpler interface again would be a method that returns an attribute *without* performing calls to descr.__get__. That could then be used for both __getattribute__ and super.__getattribute__, instead of peeking in a class' dictionary. I must admit that I haven't thought about the ramifactions of this (both functionally and performance wise). This might end up being easier to explain: both normal attribute resolution and super's resolution would end up using the same mechanism, with the differences being that super doesn't begin resolution at the start of the mro and ignores the instance __dict__. The disadvantage is introducing a new way to affect attribute resolution (do I use "__getattribute__" or this new method?). The new interface would be something like: @classmethod def __getlocalname__(cls, object, name): pass Or as you mentioned later as a __getlocalname__ method on the metaclass. The "object" argument wouldn't be necessary to reproduce current functionality, and isn't necessary for my usecase as well, but a hook for attribute resolution on an instance that doesn't have access to that instance feels wrong.
I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal.
I'd have to think about this, but on first glance this would mean a change in the semantics that a metaclass' __getattr__ currently has.
I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal.
AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable.
I'm not necessarily oppossed to a solution that requires using a metaclass, I already have metaclasses with custom metaclasses in PyObjC and this wouldn't add that much complexity to that :-) Ronald
Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) Explaining the new method is easiest if the default implementation is (literally): def __getlocalname__(self, name): try: return self.__dict__[name] except KeyError: raise AttributeError(name) which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members.
Exactly. Probably not a great idea to change that.
I assumed you were - when I was working on similar sort of code they made life extremely easy.
Ronald
Steve
On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
I like this idea and will experiment with implementing this later this week. The only thing I'm not sure about is how to indicate that the name could not be found, raising an exception could end up being to expensive if the __getlocalname__ hook gets used in object.__getattribute__ as well. I guess I'll have to run benchmarks to determine if this really is a problem. Ronald
On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
I've documented this (with a different name) in the current PEP draf (<http://www.python.org/dev/peps/pep-0447/>) and am working on an implementation. Using a lookup method on the metaclass has a nice side-effect: the same method could be used by object.__getattribute__ (aka PyObject_GenericGetAttr) as well which could simplify my primary usecase for this new API. There is a problem with that though: the type attribute cache in Object/typeobject.c, using that cache isn't valid if _PyType_Lookup calls a method instead of peeking in tp_dict (the cache cannot be in the default __getlocalname__ implementation because a primary usecase for the cache appears to be avoiding walking the MRO [1]). I haven't decided yet what to do about this, a number of options: * Don't use __getlocalname__ in _PyType_Lookup * Use __getlocalname__ as a fallback after peeking in tp_dict (that is, use it like __getattr__ instead of __getattribute__) and only cache the attribute when it is fetched from tp_dict. * Don't add a default __getlocalname__ and disable the attribute cache for types with a metatype that does have this method (this might be non-trivial because a metatype might grow a __getlocalname__ slot after instances of the metatype have been created; that would be ugly code but is possbile) Ronald [1] "appears" because I haven't found documentation for the cache yet
I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects? And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have. (Apologies for the HTML top-post) Sent from my Windows Phone ________________________________ From: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Sent: 7/6/2013 0:47 To: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Cc: python-dev@python.org Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Hooking into super() attribute resolution I've updated the implementation in issue 18181 <http://bugs.python.org/issue18181> while adding some tests, and have updated the proposal as well. The proposal has some open issues at the moment, most important of which is the actual signature for the new special method; in particular I haven't been able to decide if this should be an instance-, class- or static method. It is a static method in the proposal and prototype, but I'm not convinced that that is the right solution. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, ? Abstract ======== In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale ========= Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook ==================================== In C code --------- A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, and will raise an exception when it is not set (which shouldn't happen because the method is implemented for :class:`object`). The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* cls, PyObject* name, PyObject* object, PyObject* owner); The function should perform attribute lookup on *object* for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. The implementation of the slot for the :class:`object` type is ``PyObject_GenericGetAttrSuper``, which peeks in the ``tp_dict`` for *cls*. Note that *owner* and *object* will be the same object when using a class-mode super. In Python code -------------- A Python class can contain a definition for a static method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(cls, name, object, owner): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* XXX: I haven't got a clue at the moment if the method should be an instance-, class- or staticmethod. The prototype uses a staticmethod. XXX: My prototype automagicly makes this a static method, just like __new__ is made into a static method. That's more convenient, but also (too?) magical. XXX: Should this raise AttributeError or return a magic value to signal that an attribute cannot be found (such as NotImplemented, used in the comparison operators)? I'm currently using an exception, a magical return value would be slightly more efficient because the exception machinery is not invoked. Alternative proposals --------------------- Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues =========== * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? -> Yes? The method looks up a named attribute name of an object in a specific class. Is also likely needed to deal with @classmethod and super(Class, Class) * Should ``__getattribute_super__`` be defined on object? -> Yes: makes it easier to delegate to the default implementation * This doesn't necessarily work for class method super class (e.g. super(object, object))... References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.co...
On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower@microsoft.com> wrote:
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?
What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).
And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.
__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald
(Apologies for the HTML top-post)
I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.
The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done). I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal. I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal. AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable. (The proxy objects idea was a red herring, sorry :) ) Steve Sent from my Windows Phone ________________________________ From: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Sent: 7/7/2013 12:37 To: Steve Dower<mailto:Steve.Dower@microsoft.com> Cc: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Hooking into super() attribute resolution On 7 Jul, 2013, at 17:17, Steve Dower <Steve.Dower@microsoft.com> wrote:
Could the same result be achieved by hooking the MRO that super uses and returning a list of proxy objects?
What is the advantage over adding a hook to the class itself? That seems to be the right place to add such a hook, super already looks in the classes along the MRO and my proposal would add a formal interface for that instead of having super peek into the class __dict__. I have thought of using a custom mapping object for the tp_dict slot to intercept this, but that won't work because super assumes that tp_dict is an actual PyDictObject (and likely other parts of the interpreter do so as well).
And then wouldn't you only really need a __getattribute__ that doesn't recurse (__getlocalattribute__)? The end result may be conceptually simpler, but you've thought through the edge cases better than I have.
__getattribute_super__ already is a kind of __getlocalattribute__, the primairy difference being __getattribute_super__ is a staticmethod instead of an instance method. To be honest I'm not sure if a staticmethod is the right solution, I'm having a hard time to determine if this should be a class, instance or static method. Currently super(StartClass, x) basicly does (assuming x is an instance method): def __getattribute__(self, name): mro = type(x).mro() idx = mro.index(StartClass) while idx < len(mro): dct = mro[idx].__dict__ try: result = dct[name] # deal with descriptors here return result except KeyError: continue return object.__getattribute__(self, name) With my proposal 'dct' would no longer be needed and 'result = dct[name]' would be 'mro[idx].__getattribute_super__(mro[idx], name, x, StartClass)' (I may have the last argument for the call to __getattribute_super__ wrong, but that's the idea). Given that the first argument of __get...super__ is the same as the object the method get getattr-ed from I guess the method should be a classmethod instead of an staticmethod. Changing that would be easy enough. I'm still interested in feedback on the basic idea, I'd love to here that my proposal isn't necessary because there is already a way to get the behavior I'm looking for although that's nog going to happen ;-). Ronald
(Apologies for the HTML top-post)
I don't mind. PS. Does anyone know if the pep editors are away (conferences, holidays, ...)? I could just check in my proposal in the peps repository, but as this is my first PEP I'd prefer to follow the documented procedure and have someone that knows what he's doing look at the metadata before checking in.
On 8 Jul, 2013, at 17:19, Steve Dower <Steve.Dower@microsoft.com> wrote:
The only real advantage is a simpler signature and more easily explained use (assuming the person you're explaining it to is familiar with metaclasses, so most of the hard explaining has been done).
The signature is as complex as it is to be able to call descr.__get__ with the correct arguments. I ended up with the current signature when I added __getattribute_super__ to object and removed the tp_dict peeking code from super's tp_getattro. A way to get a simpler interface again would be a method that returns an attribute *without* performing calls to descr.__get__. That could then be used for both __getattribute__ and super.__getattribute__, instead of peeking in a class' dictionary. I must admit that I haven't thought about the ramifactions of this (both functionally and performance wise). This might end up being easier to explain: both normal attribute resolution and super's resolution would end up using the same mechanism, with the differences being that super doesn't begin resolution at the start of the mro and ignores the instance __dict__. The disadvantage is introducing a new way to affect attribute resolution (do I use "__getattribute__" or this new method?). The new interface would be something like: @classmethod def __getlocalname__(cls, object, name): pass Or as you mentioned later as a __getlocalname__ method on the metaclass. The "object" argument wouldn't be necessary to reproduce current functionality, and isn't necessary for my usecase as well, but a hook for attribute resolution on an instance that doesn't have access to that instance feels wrong.
I'm still not sure that this isn't simply a bug in super. If the superclass's metaclass provides a __getattr__ then it should probably use it and abandon it's own MRO traversal.
I'd have to think about this, but on first glance this would mean a change in the semantics that a metaclass' __getattr__ currently has.
I still haven't thought the edge cases through, and it seems like there'd be some with that change, so that's where __getattribute_super__ comes in - super can call it without abandoning its MRO traversal.
AFAICT, the difference between that and __getlocalattribute__ is that the latter would be implemented on a metaclass while the former takes extra parameters. I think this functionality is advanced enough that requiring a metaclass isn't unreasonable.
I'm not necessarily oppossed to a solution that requires using a metaclass, I already have metaclasses with custom metaclasses in PyObjC and this wouldn't add that much complexity to that :-) Ronald
Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) Explaining the new method is easiest if the default implementation is (literally): def __getlocalname__(self, name): try: return self.__dict__[name] except KeyError: raise AttributeError(name) which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members.
Exactly. Probably not a great idea to change that.
I assumed you were - when I was working on similar sort of code they made life extremely easy.
Ronald
Steve
On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
I like this idea and will experiment with implementing this later this week. The only thing I'm not sure about is how to indicate that the name could not be found, raising an exception could end up being to expensive if the __getlocalname__ hook gets used in object.__getattribute__ as well. I guess I'll have to run benchmarks to determine if this really is a problem. Ronald
On 9 Jul, 2013, at 1:21, Steve Dower <Steve.Dower@microsoft.com> wrote:
I've documented this (with a different name) in the current PEP draf (<http://www.python.org/dev/peps/pep-0447/>) and am working on an implementation. Using a lookup method on the metaclass has a nice side-effect: the same method could be used by object.__getattribute__ (aka PyObject_GenericGetAttr) as well which could simplify my primary usecase for this new API. There is a problem with that though: the type attribute cache in Object/typeobject.c, using that cache isn't valid if _PyType_Lookup calls a method instead of peeking in tp_dict (the cache cannot be in the default __getlocalname__ implementation because a primary usecase for the cache appears to be avoiding walking the MRO [1]). I haven't decided yet what to do about this, a number of options: * Don't use __getlocalname__ in _PyType_Lookup * Use __getlocalname__ as a fallback after peeking in tp_dict (that is, use it like __getattr__ instead of __getattribute__) and only cache the attribute when it is fetched from tp_dict. * Don't add a default __getlocalname__ and disable the attribute cache for types with a metatype that does have this method (this might be non-trivial because a metatype might grow a __getlocalname__ slot after instances of the metatype have been created; that would be ugly code but is possbile) Ronald [1] "appears" because I haven't found documentation for the cache yet
participants (3)
-
Guido van Rossum -
Ronald Oussoren -
Steve Dower