PEP 447 (type.__getdescriptor__)
Hi, Another summer with another EuroPython, which means its time again to try to revive PEP 447… I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP. The PEP proposes to to replace direct access to the class __dict__ in object.__getattribute__ and super.__getattribute__ by calls to a new special method to give the metaclass more control over attribute lookup, especially for access using a super() object. This is needed for classes that cannot store (all) descriptors in the class dict for some reason, see the PEP text for a real-world example of that. Regards, Ronald The PEP text (with an outdated section with benchmarks removed): PEP: 447 Title: Add __getdescriptor__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015 Abstract ======== Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__getdescriptor__`` method to a metaclass that replaces this behavior and gives more control over attribute lookup, especially when using a `super`_ object. That is, the MRO walking loop in ``_PyType_Lookup`` and ``super.__getattribute__`` gets changed from:: def lookup(mro_list, name): for cls in mro_list: if name in cls.__dict__: return cls.__dict__ return NotFound to:: def lookup(mro_list, name): for cls in mro_list: try: return cls.__getdescriptor__(name) except AttributeError: pass return NotFound The default implemention of ``__getdescriptor__`` looks in the class dictionary:: class type: def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None Rationale ========= It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand. The ``__getdescriptor__`` method makes it possible to dynamically add attributes even when looking them up using the `super class`_. The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency and to have a single place to implement dynamic attribute resolution for classes. Background ---------- The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with `super`_ objects. Because of this PyObjC currently includes a custom `super`_ that must be used with its classes, as well as completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute access. The API in this PEP makes it possible to remove the custom `super`_ and simplifies the implementation because the custom lookup behavior can be added in a central location. .. note:: `PyObjC`_ cannot precalculate the contents of the class ``__dict__`` because Objective-C classes can grow new methods at runtime. Furthermore Objective-C classes tend to contain a lot of methods while most Python code will only use a small subset of them, this makes precalculating unnecessarily expensive. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code) walk an object's MRO and currently peek in the class' ``__dict__`` to look up attributes. With this proposal both lookup methods no longer peek in the class ``__dict__`` but call the special method ``__getdescriptor__``, which is a slot defined on the metaclass. The default implementation of that method looks up the name the class ``__dict__``, which means that attribute lookup is unchanged unless a metatype actually defines the new special method. Aside: Attribute resolution algorithm in Python ----------------------------------------------- The attribute resolution proces as implemented by ``object.__getattribute__`` (or PyObject_GenericGetAttr`` in CPython's implementation) is fairly straightforward, but not entirely so without reading C code. The current CPython implementation of object.__getattribute__ is basicly equivalent to the following (pseudo-) Python code (excluding some house keeping and speed tricks):: def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple) for base in mro: assert isinstance(base, type) # PEP 447 will change these lines: try: return base.__dict__[name] except KeyError: pass return None class object: def __getattribute__(self, name): assert isinstance(name, str) tp = type(self) descr = _PyType_Lookup(tp, name) f = None if descr is not None: f = descr.__get__ if f is not None and descr.__set__ is not None: # Data descriptor return f(descr, self, type(self)) dict = self.__dict__ if dict is not None: try: return self.__dict__[name] except KeyError: pass if f is not None: # Non-data descriptor return f(descr, self, type(self)) if descr is not None: # Regular class attribute return descr raise AttributeError(name) class super: def __getattribute__(self, name): assert isinstance(name, unicode) if name != '__class__': starttype = self.__self_type__ mro = startype.mro() try: idx = mro.index(self.__thisclass__) except ValueError: pass else: for base in mro[idx+1:]: # PEP 447 will change these lines: try: descr = base.__dict__[name] except KeyError: continue f = descr.__get__ if f is not None: return f(descr, None if (self.__self__ is self.__self_type__) else self.__self__, starttype) else: return descr return object.__getattribute__(self, name) This PEP should change the dict lookup at the lines starting at "# PEP 447" with a method call to perform the actual lookup, making is possible to affect that lookup both for normal attribute access and access through the `super proxy`_. Note that specific classes can already completely override the default behaviour by implementing their own ``__getattribute__`` slot (with or without calling the super class implementation). In Python code -------------- A meta type can define a method ``__getdescriptor__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType(type): def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The ``__getdescriptor__`` method has as its arguments a class (which is an instance of the meta type) and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, and should raise `AttributeError`_ when the name cannot be found. The `type`_ class provides a default implementation for ``__getdescriptor__``, that looks up the name in the class dictionary. Example usage ............. The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names:: class UpperCaseAccess (type): def __getdescriptor__(cls, name): try: return cls.__dict__[name.upper()] except KeyError: raise AttributeError(name) from None class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42 def M(self): return "fourtytwo" obj = SillyObject() assert obj.m() == "fortytwo" As mentioned earlier in this PEP a more realistic use case of this functionallity is a ``__getdescriptor__`` method that dynamicly populates the class ``__dict__`` based on attribute access, primarily when it is not possible to reliably keep the class dict in sync with its source, for example because the source used to populate ``__dict__`` is dynamic as well and does not have triggers that can be used to detect changes to that source. An example of that are the class bridges in PyObjC: the class bridge is a Python object (class) that represents an Objective-C class and conceptually has a Python method for every Objective-C method in the Objective-C class. As with Python it is possible to add new methods to an Objective-C class, or replace existing ones, and there are no callbacks that can be used to detect this. In C code --------- A new slot ``tp_getdescriptor`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__getdescriptor__`` method on `type`_. The slot has the following prototype:: PyObject* (*getdescriptorfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Use of this hook by the interpreter ----------------------------------- The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__getdescriptor__`` method when walking the MRO. Other changes to the implementation ----------------------------------- The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__getdescriptor__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API. The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__getdescriptor__``, because using the cache might not be valid for such classes. Impact of this PEP on introspection ----------------------------------- Use of the method introduced in this PEP can affect introspection of classes with a metaclass that uses a custom ``__getdescriptor__`` method. This section lists those changes. The items listed below are only affected by custom ``__getdescriptor__`` methods, the default implementation for ``object`` won't cause problems because that still only uses the class ``__dict__`` and won't cause visible changes to the visible behaviour of the ``object.__getattribute__``. * ``dir`` might not show all attributes As with a custom ``__getattribute__`` method `dir()`_ might not see all (instance) attributes when using the ``__getdescriptor__()`` method to dynamicly resolve attributes. The solution for that is quite simple: classes using ``__getdescriptor__`` should also implement `__dir__()`_ if they want full support for the builtin `dir()`_ function. * ``inspect.getattr_static`` might not show all attributes The function ``inspect.getattr_static`` intentionally does not invoke ``__getattribute__`` and descriptors to avoid invoking user code during introspection with this function. The ``__getdescriptor__`` method will also be ignored and is another way in which the result of ``inspect.getattr_static`` can be different from that of ``builtin.getattr``. * ``inspect.getmembers`` and ``inspect.get_class_attrs`` Both of these functions directly access the class __dict__ of classes along the MRO, and hence can be affected by a custom ``__getdescriptor__`` method. **TODO**: I haven't fully worked out what the impact of this is, and if there are mitigations for those using either updates to these functions, or additional methods that users should implement to be fully compatible with these functions. One possible mitigation is to have a custom ``__getattribute__`` for these classes that fills ``__dict__`` before returning and and defers to the default implementation for other attributes. * Direct introspection of the class ``__dict__`` Any code that directly access the class ``__dict__`` for introspection can be affected by a custom ``__getdescriptor__`` method. Performance impact ------------------ **WARNING**: The benchmark results in this section are old, and will be updated when I've ported the patch to the current trunk. I don't expect significant changes to the results in this section. [snipped] Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Alternate placement of the new method ..................................... This PEP proposes to add ``__getdescriptor__`` as a method on the metaclass. An alternative would be to add it as a class method on the class itself (simular to how ``__new__`` is a `staticmethod`_ of the class and not a method of the metaclass). The two are functionally equivalent, and there's something to be said about not requiring the use of a meta class. References ========== * `Issue 18181`_ contains an out of date prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`super proxy`: http://docs.python.org/3/library/functions.html#super .. _`super`: http://docs.python.org/3/library/functions.html#super .. _`dir()`: http://docs.python.org/3/library/functions.html#dir .. _`staticmethod`: http://docs.python.org/3/library/functions.html#staticmethod .. _`__dir__()`: https://docs.python.org/3/reference/datamodel.html#object.__dir__ .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError .. _`PyObjC`: http://pyobjc.sourceforge.net/ .. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this. There are two fairly obvious non-esoteric questions: 1. How does this impact speed (updated section needed)? 2. Is this useful, that you can think of, for anything other than connecting to Objective C? -- Terry Jan Reedy
Terry Reedy wrote:
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this.
There are two fairly obvious non-esoteric questions:
1. How does this impact speed (updated section needed)?
Agreed, this is important. But hopefully it's just a C indirection (or better yet, a null check) for objects that don't override __getdescriptor__.
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
There are other object models that would benefit from this, but I don't recall that we came up with uses other than "helps proxy to objects where listing all members eagerly is expensive and/or potentially incorrect". Maybe once you list all the operating systems that are now using dynamic object-oriented APIs rather than flat APIs (Windows, iOS, Android, ... others?) this is good enough? FWIW, I'm still +1 on this, pending performance testing. Cheers, Steve
-- Terry Jan Reedy
On 22 Jul 2015, at 19:12, Steve Dower <Steve.Dower@microsoft.com> wrote:
Terry Reedy wrote:
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this.
There are two fairly obvious non-esoteric questions:
1. How does this impact speed (updated section needed)?
Agreed, this is important. But hopefully it's just a C indirection (or better yet, a null check) for objects that don't override __getdescriptor__.
Not a null check, but a check for a specific function pointer. That way you can be sure that super classes always have the slot which IMHO gives a nicer user experience.
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
There are other object models that would benefit from this, but I don't recall that we came up with uses other than "helps proxy to objects where listing all members eagerly is expensive and/or potentially incorrect". Maybe once you list all the operating systems that are now using dynamic object-oriented APIs rather than flat APIs (Windows, iOS, Android, ... others?) this is good enough?
FWIW, I'm still +1 on this, pending performance testing.
The PEP on the website contains performance test data, but that’s out of data. I don’t think the implementation of attribute lookup has changed enough to really invalidate those test results, but I will rerun the tests once I’ve updated the patch because hunches don’t count when evaluating performance. Ronald
Cheers, Steve
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com <https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com>
On 23 July 2015 at 03:12, Steve Dower <Steve.Dower@microsoft.com> wrote:
Terry Reedy wrote:
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this.
There are two fairly obvious non-esoteric questions:
1. How does this impact speed (updated section needed)?
Agreed, this is important. But hopefully it's just a C indirection (or better yet, a null check) for objects that don't override __getdescriptor__.
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
There are other object models that would benefit from this, but I don't recall that we came up with uses other than "helps proxy to objects where listing all members eagerly is expensive and/or potentially incorrect". Maybe once you list all the operating systems that are now using dynamic object-oriented APIs rather than flat APIs (Windows, iOS, Android, ... others?) this is good enough?
"better bridging to other languages and runtimes" is a good enough rationale for me, although I also wonder if it might be useful for making some interesting COM and dbus based API wrappers. Ronald, could you dig up a reference to the last thread (or threads) on this? My recollection is that we were actually pretty happy with it, and it was just set aside through lack of time to push it through to completion. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 Jul 2015, at 16:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 23 July 2015 at 03:12, Steve Dower <Steve.Dower@microsoft.com <mailto:Steve.Dower@microsoft.com>> wrote:
Terry Reedy wrote:
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this.
There are two fairly obvious non-esoteric questions:
1. How does this impact speed (updated section needed)?
Agreed, this is important. But hopefully it's just a C indirection (or better yet, a null check) for objects that don't override __getdescriptor__.
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
There are other object models that would benefit from this, but I don't recall that we came up with uses other than "helps proxy to objects where listing all members eagerly is expensive and/or potentially incorrect". Maybe once you list all the operating systems that are now using dynamic object-oriented APIs rather than flat APIs (Windows, iOS, Android, ... others?) this is good enough?
"better bridging to other languages and runtimes" is a good enough rationale for me, although I also wonder if it might be useful for making some interesting COM and dbus based API wrappers.
Ronald, could you dig up a reference to the last thread (or threads) on this? My recollection is that we were actually pretty happy with it, and it was just set aside through lack of time to push it through to completion.
I’ll do some digging in my archives. From what I recall you and Steve were positive the last time around and others didn’t have much to add at the time. FWIW Guido was positive about the idea, but would really like to see up to date benchmark results and some specific micro benchmarking to see if the change has negative performance impact. I do have a API design question now that I’m working on this again: the PEP proposed to add a __getdescriptor__ method to the meta type, that is you’d define it as: class MyMeta (type): def __getdescriptor__(self, name): … class MyType (object, metaclass=MyMeta): pass This doesn’t match how other special slots are done, in particular __new__. I’d like to switch the definition to: class MyType: @classmethod def __getdescriptor__(cls, name): … I have two questions about that: (1) is this indeed a better interface and (2) should users explicitly use the classmethod decorator or would it be better to match the behaviour for __new__ by leaving that out? Personally I do think that this is a better interface, but am not sure about requiring the decorator. Ronald P.S. Fighting with refcounting between sessions, forward porting of the patch for this PEP seems to have introduced a refcount problem. Nothing that cannot be fixed during the sprints though.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com <https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com>
On Fri, Jul 24, 2015, 07:43 Ronald Oussoren <ronaldoussoren@mac.com> wrote: On 24 Jul 2015, at 16:17, Nick Coghlan <ncoghlan@gmail.com> wrote: On 23 July 2015 at 03:12, Steve Dower <Steve.Dower@microsoft.com> wrote: Terry Reedy wrote: On 7/22/2015 3:25 AM, Ronald Oussoren wrote: Hi, Another summer with another EuroPython, which means its time again to try to revive PEP 447… I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP. Yeh, a bit too esoteric for most of us to review. For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed. If Guido does not want to review this, you need to find a PEP BDFL for this. There are two fairly obvious non-esoteric questions: 1. How does this impact speed (updated section needed)? Agreed, this is important. But hopefully it's just a C indirection (or better yet, a null check) for objects that don't override __getdescriptor__. 2. Is this useful, that you can think of, for anything other than connecting to Objective C? There are other object models that would benefit from this, but I don't recall that we came up with uses other than "helps proxy to objects where listing all members eagerly is expensive and/or potentially incorrect". Maybe once you list all the operating systems that are now using dynamic object-oriented APIs rather than flat APIs (Windows, iOS, Android, ... others?) this is good enough? "better bridging to other languages and runtimes" is a good enough rationale for me, although I also wonder if it might be useful for making some interesting COM and dbus based API wrappers. Ronald, could you dig up a reference to the last thread (or threads) on this? My recollection is that we were actually pretty happy with it, and it was just set aside through lack of time to push it through to completion. I’ll do some digging in my archives. From what I recall you and Steve were positive the last time around and others didn’t have much to add at the time. FWIW Guido was positive about the idea, but would really like to see up to date benchmark results and some specific micro benchmarking to see if the change has negative performance impact. I do have a API design question now that I’m working on this again: the PEP proposed to add a __getdescriptor__ method to the meta type, that is you’d define it as: class MyMeta (type): def __getdescriptor__(self, name): … class MyType (object, metaclass=MyMeta): pass This doesn’t match how other special slots are done, in particular __new__. I’d like to switch the definition to: class MyType: @classmethod def __getdescriptor__(cls, name): … I have two questions about that: (1) is this indeed a better interface and (2) should users explicitly use the classmethod decorator or would it be better to match the behaviour for __new__ by leaving that out? Personally I do think that this is a better interface, but am not sure about requiring the decorator. Leave the decorator out like __new__, otherwise people are bound to forget it and have a hard time debugging why their code doesn't work. -Brett Ronald P.S. Fighting with refcounting between sessions, forward porting of the patch for this PEP seems to have introduced a refcount problem. Nothing that cannot be fixed during the sprints though. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On 25 July 2015 at 00:50, Brett Cannon <brett@python.org> wrote:
Leave the decorator out like __new__, otherwise people are bound to forget it and have a hard time debugging why their code doesn't work.
I'd actually advocate for keeping this as a metaclass method, rather than making it available to any type instance. The key thing to consider for me is "What additional power does making it a method on the class itself grant to mixin types?" With PEP 487, the __init_subclass__ proposal only grants mixins the power to implicitly run additional code when new subclasses are defined. They have no additional ability to influence the behaviour of the specific class adding the mixin into the inheritance hierarchy. With PEP 447, as currently written, a mixin that wants to alter how descriptors are looked up will be able to do so implicitly as long as there are no other custom metaclasses in the picture. As soon as there are *two* custom metaclasses involved, you'll get an error at definition time and have to sort out how you want the metaclass inheritance to work and have a chance to notice if there are two competing __getdescriptor__ implementations. However, if __getdescriptor__ moves to being a class method on object rather than an instance method on type, then you'll lose that assistance from the metaclass checker - if you have two classes in your MRO with mutually incompatible __getdescriptor__ implementations, you're likely to be in for a world of pain as you try to figure out the source of any related bugs. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 Jul 2015, at 17:29, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 25 July 2015 at 00:50, Brett Cannon <brett@python.org> wrote:
Leave the decorator out like __new__, otherwise people are bound to forget it and have a hard time debugging why their code doesn't work.
I'd actually advocate for keeping this as a metaclass method, rather than making it available to any type instance. The key thing to consider for me is "What additional power does making it a method on the class itself grant to mixin types?”
To be honest, I hadn’t considered mixin types yet.
With PEP 487, the __init_subclass__ proposal only grants mixins the power to implicitly run additional code when new subclasses are defined. They have no additional ability to influence the behaviour of the specific class adding the mixin into the inheritance hierarchy.
With PEP 447, as currently written, a mixin that wants to alter how descriptors are looked up will be able to do so implicitly as long as there are no other custom metaclasses in the picture. As soon as there are *two* custom metaclasses involved, you'll get an error at definition time and have to sort out how you want the metaclass inheritance to work and have a chance to notice if there are two competing __getdescriptor__ implementations.
However, if __getdescriptor__ moves to being a class method on object rather than an instance method on type, then you'll lose that assistance from the metaclass checker - if you have two classes in your MRO with mutually incompatible __getdescriptor__ implementations, you're likely to be in for a world of pain as you try to figure out the source of any related bugs.
That’s a good point, and something that will move something that I’ve wanted to look into forward on my list: the difference between a classmethod and a method on the class defined through a metaclass. The semantics I’d like to have is that __getdescriptor__ is a local decision, defining __getdescriptor__ for a class should only affect that class and its subclass, and shouldn’t affect how superclasses are handled by __getattribute__. That is something that can be done by defining __getdescriptor__ on a metaclass, and AFAIK requires active cooperation when using a @classmethod. It should be possible to demonstrate the differences in a pure Python prototype. Ronald
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
I’ve pushed a minor update to the PEP to the repository. The benchmark results are still out of date, I need want to run those on an idle machine to get reliable results. The PEP has one significant change w.r.t. the previous version: it now requires the use of a new type flag to enable the usage of the new slot in C code. This is due to concerns that loading old extensions might crash the interpreter otherwise. References to earlier discussions (also added to the PEP): * http://marc.info/?l=python-dev&m=137510220928964&w=2 * https://mail.python.org/pipermail/python-ideas/2014-July/028420.html * https://mail.python.org/pipermail/python-dev/2013-July/127321.html And finally, I’ve updated the implementation in issue 18181. The implementation passes the test suite with the current trunk and is good enough to play around with. There is still an important issue though: I’ve done some micro benchmarking and those indicate that the method_cache mechanism in typeobject.c doesn’t work with my changes and that has a clear performance impact and must be fixed. That shouldn’t be too hard to fix, it’s probably just a botched check before the blocks of code that use and update the cache. Ronald
On 24 Jul 2015, at 19:55, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 24 Jul 2015, at 17:29, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
On 25 July 2015 at 00:50, Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote:
Leave the decorator out like __new__, otherwise people are bound to forget it and have a hard time debugging why their code doesn't work.
I'd actually advocate for keeping this as a metaclass method, rather than making it available to any type instance. The key thing to consider for me is "What additional power does making it a method on the class itself grant to mixin types?”
To be honest, I hadn’t considered mixin types yet.
With PEP 487, the __init_subclass__ proposal only grants mixins the power to implicitly run additional code when new subclasses are defined. They have no additional ability to influence the behaviour of the specific class adding the mixin into the inheritance hierarchy.
With PEP 447, as currently written, a mixin that wants to alter how descriptors are looked up will be able to do so implicitly as long as there are no other custom metaclasses in the picture. As soon as there are *two* custom metaclasses involved, you'll get an error at definition time and have to sort out how you want the metaclass inheritance to work and have a chance to notice if there are two competing __getdescriptor__ implementations.
However, if __getdescriptor__ moves to being a class method on object rather than an instance method on type, then you'll lose that assistance from the metaclass checker - if you have two classes in your MRO with mutually incompatible __getdescriptor__ implementations, you're likely to be in for a world of pain as you try to figure out the source of any related bugs.
That’s a good point, and something that will move something that I’ve wanted to look into forward on my list: the difference between a classmethod and a method on the class defined through a metaclass.
The semantics I’d like to have is that __getdescriptor__ is a local decision, defining __getdescriptor__ for a class should only affect that class and its subclass, and shouldn’t affect how superclasses are handled by __getattribute__. That is something that can be done by defining __getdescriptor__ on a metaclass, and AFAIK requires active cooperation when using a @classmethod.
It should be possible to demonstrate the differences in a pure Python prototype.
Ronald
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com <https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com>
On 22 Jul 2015, at 18:02, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/22/2015 3:25 AM, Ronald Oussoren wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
Yeh, a bit too esoteric for most of us to review.
I noticed that in my previous attempts as well. There is only a limited number of people the really grok how Python’s attribute lookup works, and a smaller subset of those understand how that’s implemented in CPython.
For instance, it is not obvious to me, not familiar with internal details, after reading the intro, why a custom __getattribute__ is not enough and why __getdescriptor__ would be needed.
That means the PEP text needs some more work. Using __getattribute__ works for normal attribute access, but not when you look for a superclass implementation using super() because super currently *only* looks in the __dict__ of classes further along the MRO and offers no way to influence the search. That’s a problem when classes can grow methods dynamically.
If Guido does not want to review this, you need to find a PEP BDFL for this.
I’ll see if I can corner him at EP :-). Its surprisingly hard to find people at conferences.
There are two fairly obvious non-esoteric questions:
1. How does this impact speed (updated section needed)?
The speed impact should be minimal, the initial version of the patch (which needs some updating which I’ll try to do during the EP sprints) uses shortcuts to avoid actually calling the __getdescriptor__ method in the usual case.
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
Not immediately. But then again, I initially thought that decorators would have limited appeal as well :-). I guess this could be useful for other proxy-like objects as well, especially when preloading the __dict__ is relatively expensive. Apart from direct usefulness this closes a hole in the way you can influence attribute lookup. Ronald
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
On 22 July 2015 at 14:21, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
2. Is this useful, that you can think of, for anything other than connecting to Objective C?
Not immediately. But then again, I initially thought that decorators would have limited appeal as well :-). I guess this could be useful for other proxy-like objects as well, especially when preloading the __dict__ is relatively expensive.
I think one place this could be immediately useful is in code using remote method invocation/remote attribute access. This allows one to subclass proxy types for remote objects, and call methods that resolve remotely in a seamless way. (And without having to download and pre-populate an entire API into the proxy-class __dict__) +1 I found the solution rather concise for the flexibility it adds as well.
Apart from direct usefulness this closes a hole in the way you can influence attribute lookup.
Ronald
Hey I've taken time to read the PEP, my 2c... actually 1c: Most important is to explain how this changes behaviour of Python programs. A comprehensive set of Python examples where behaviour is changed (for better or worse) please. While I understand the concern of "superclasses of objects that gain or lose attributes at runtime" on the theoretical level, please translate that into actual Python examples. d. On 22 July 2015 at 09:25, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
The PEP proposes to to replace direct access to the class __dict__ in object.__getattribute__ and super.__getattribute__ by calls to a new special method to give the metaclass more control over attribute lookup, especially for access using a super() object. This is needed for classes that cannot store (all) descriptors in the class dict for some reason, see the PEP text for a real-world example of that.
Regards,
Ronald
The PEP text (with an outdated section with benchmarks removed):
PEP: 447 Title: Add __getdescriptor__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
Abstract ========
Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__getdescriptor__`` method to a metaclass that replaces this behavior and gives more control over attribute lookup, especially when using a `super`_ object.
That is, the MRO walking loop in ``_PyType_Lookup`` and ``super.__getattribute__`` gets changed from::
def lookup(mro_list, name): for cls in mro_list: if name in cls.__dict__: return cls.__dict__
return NotFound
to::
def lookup(mro_list, name): for cls in mro_list: try: return cls.__getdescriptor__(name) except AttributeError: pass
return NotFound
The default implemention of ``__getdescriptor__`` looks in the class dictionary::
class type: def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
Rationale =========
It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand.
The ``__getdescriptor__`` method makes it possible to dynamically add attributes even when looking them up using the `super class`_.
The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency and to have a single place to implement dynamic attribute resolution for classes.
Background ----------
The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with `super`_ objects. Because of this PyObjC currently includes a custom `super`_ that must be used with its classes, as well as completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute access.
The API in this PEP makes it possible to remove the custom `super`_ and simplifies the implementation because the custom lookup behavior can be added in a central location.
.. note::
`PyObjC`_ cannot precalculate the contents of the class ``__dict__`` because Objective-C classes can grow new methods at runtime. Furthermore Objective-C classes tend to contain a lot of methods while most Python code will only use a small subset of them, this makes precalculating unnecessarily expensive.
The superclass attribute lookup hook ====================================
Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code) walk an object's MRO and currently peek in the class' ``__dict__`` to look up attributes.
With this proposal both lookup methods no longer peek in the class ``__dict__`` but call the special method ``__getdescriptor__``, which is a slot defined on the metaclass. The default implementation of that method looks up the name the class ``__dict__``, which means that attribute lookup is unchanged unless a metatype actually defines the new special method.
Aside: Attribute resolution algorithm in Python -----------------------------------------------
The attribute resolution proces as implemented by ``object.__getattribute__`` (or PyObject_GenericGetAttr`` in CPython's implementation) is fairly straightforward, but not entirely so without reading C code.
The current CPython implementation of object.__getattribute__ is basicly equivalent to the following (pseudo-) Python code (excluding some house keeping and speed tricks)::
def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple)
for base in mro: assert isinstance(base, type)
# PEP 447 will change these lines: try: return base.__dict__[name] except KeyError: pass
return None
class object: def __getattribute__(self, name): assert isinstance(name, str)
tp = type(self) descr = _PyType_Lookup(tp, name)
f = None if descr is not None: f = descr.__get__ if f is not None and descr.__set__ is not None: # Data descriptor return f(descr, self, type(self))
dict = self.__dict__ if dict is not None: try: return self.__dict__[name] except KeyError: pass
if f is not None: # Non-data descriptor return f(descr, self, type(self))
if descr is not None: # Regular class attribute return descr
raise AttributeError(name)
class super: def __getattribute__(self, name): assert isinstance(name, unicode)
if name != '__class__': starttype = self.__self_type__ mro = startype.mro()
try: idx = mro.index(self.__thisclass__)
except ValueError: pass
else: for base in mro[idx+1:]: # PEP 447 will change these lines: try: descr = base.__dict__[name] except KeyError: continue
f = descr.__get__ if f is not None: return f(descr, None if (self.__self__ is self.__self_type__) else self.__self__, starttype)
else: return descr
return object.__getattribute__(self, name)
This PEP should change the dict lookup at the lines starting at "# PEP 447" with a method call to perform the actual lookup, making is possible to affect that lookup both for normal attribute access and access through the `super proxy`_.
Note that specific classes can already completely override the default behaviour by implementing their own ``__getattribute__`` slot (with or without calling the super class implementation).
In Python code --------------
A meta type can define a method ``__getdescriptor__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::
class MetaType(type): def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
The ``__getdescriptor__`` method has as its arguments a class (which is an instance of the meta type) and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, and should raise `AttributeError`_ when the name cannot be found.
The `type`_ class provides a default implementation for ``__getdescriptor__``, that looks up the name in the class dictionary.
Example usage .............
The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names::
class UpperCaseAccess (type): def __getdescriptor__(cls, name): try: return cls.__dict__[name.upper()] except KeyError: raise AttributeError(name) from None
class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42
def M(self): return "fourtytwo"
obj = SillyObject() assert obj.m() == "fortytwo"
As mentioned earlier in this PEP a more realistic use case of this functionallity is a ``__getdescriptor__`` method that dynamicly populates the class ``__dict__`` based on attribute access, primarily when it is not possible to reliably keep the class dict in sync with its source, for example because the source used to populate ``__dict__`` is dynamic as well and does not have triggers that can be used to detect changes to that source.
An example of that are the class bridges in PyObjC: the class bridge is a Python object (class) that represents an Objective-C class and conceptually has a Python method for every Objective-C method in the Objective-C class. As with Python it is possible to add new methods to an Objective-C class, or replace existing ones, and there are no callbacks that can be used to detect this.
In C code ---------
A new slot ``tp_getdescriptor`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__getdescriptor__`` method on `type`_.
The slot has the following prototype::
PyObject* (*getdescriptorfunc)(PyTypeObject* cls, PyObject* name);
This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).
Use of this hook by the interpreter -----------------------------------
The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__getdescriptor__`` method when walking the MRO.
Other changes to the implementation -----------------------------------
The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__getdescriptor__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API.
The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__getdescriptor__``, because using the cache might not be valid for such classes.
Impact of this PEP on introspection -----------------------------------
Use of the method introduced in this PEP can affect introspection of classes with a metaclass that uses a custom ``__getdescriptor__`` method. This section lists those changes.
The items listed below are only affected by custom ``__getdescriptor__`` methods, the default implementation for ``object`` won't cause problems because that still only uses the class ``__dict__`` and won't cause visible changes to the visible behaviour of the ``object.__getattribute__``.
* ``dir`` might not show all attributes
As with a custom ``__getattribute__`` method `dir()`_ might not see all (instance) attributes when using the ``__getdescriptor__()`` method to dynamicly resolve attributes.
The solution for that is quite simple: classes using ``__getdescriptor__`` should also implement `__dir__()`_ if they want full support for the builtin `dir()`_ function.
* ``inspect.getattr_static`` might not show all attributes
The function ``inspect.getattr_static`` intentionally does not invoke ``__getattribute__`` and descriptors to avoid invoking user code during introspection with this function. The ``__getdescriptor__`` method will also be ignored and is another way in which the result of ``inspect.getattr_static`` can be different from that of ``builtin.getattr``.
* ``inspect.getmembers`` and ``inspect.get_class_attrs``
Both of these functions directly access the class __dict__ of classes along the MRO, and hence can be affected by a custom ``__getdescriptor__`` method.
**TODO**: I haven't fully worked out what the impact of this is, and if there are mitigations for those using either updates to these functions, or additional methods that users should implement to be fully compatible with these functions.
One possible mitigation is to have a custom ``__getattribute__`` for these classes that fills ``__dict__`` before returning and and defers to the default implementation for other attributes.
* Direct introspection of the class ``__dict__``
Any code that directly access the class ``__dict__`` for introspection can be affected by a custom ``__getdescriptor__`` method.
Performance impact ------------------
**WARNING**: The benchmark results in this section are old, and will be updated when I've ported the patch to the current trunk. I don't expect significant changes to the results in this section.
[snipped]
Alternative proposals ---------------------
``__getattribute_super__`` ..........................
An earlier version of this PEP used the following static method on classes::
def __getattribute_super__(cls, name, object, owner): pass
This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``.
Reuse ``tp_getattro`` .....................
It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO.
That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_.
Alternate placement of the new method .....................................
This PEP proposes to add ``__getdescriptor__`` as a method on the metaclass. An alternative would be to add it as a class method on the class itself (simular to how ``__new__`` is a `staticmethod`_ of the class and not a method of the metaclass).
The two are functionally equivalent, and there's something to be said about not requiring the use of a meta class.
References ==========
* `Issue 18181`_ contains an out of date prototype implementation
Copyright =========
This document has been placed in the public domain.
.. _`Issue 18181`: http://bugs.python.org/issue18181
.. _`super class`: http://docs.python.org/3/library/functions.html#super
.. _`super proxy`: http://docs.python.org/3/library/functions.html#super
.. _`super`: http://docs.python.org/3/library/functions.html#super
.. _`dir()`: http://docs.python.org/3/library/functions.html#dir
.. _`staticmethod`: http://docs.python.org/3/library/functions.html#staticmethod
.. _`__dir__()`: https://docs.python.org/3/reference/datamodel.html#object.__dir__
.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented
.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr
.. _`type`: http://docs.python.org/3/library/functions.html#type
.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError
.. _`PyObjC`: http://pyobjc.sourceforge.net/
.. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dimaqq%40gmail.com
On 23 Jul 2015, at 11:29, Dima Tisnek <dimaqq@gmail.com> wrote:
Hey I've taken time to read the PEP, my 2c... actually 1c:
Most important is to explain how this changes behaviour of Python programs.
A comprehensive set of Python examples where behaviour is changed (for better or worse) please.
The behaviour of existing python code is not changed at all. Code that directly looks in the class __dict__ might be impacted, but only when running into classes with a custom __getdescriptor__ method. I’ve listed the code in the stdlib that could be affected, but have to do a new pass of the stdlib to check if anything relevant changed since I wrote the section. I general you run into the same issues when adding a custom __getattribute__ or __getattr__ method, both of which will confuse introspection tools that assume regular attribute lookup semantics.
While I understand the concern of "superclasses of objects that gain or lose attributes at runtime" on the theoretical level, please translate that into actual Python examples.
The primary use case I have for this are classes that are proxies for external systems. There may be other uses as well, but I don’t have examples of that (other than the contrived example in the PEP). The reason I wrote the PEP in the first place is PyObjC: this project defines a proxy later between Python and Objective-C, with the goal to making it possible to write programs for Mac OS X in Python while being able to make full use of Apple’s high-level APIs. The proxy layer is mostly completely dynamic: proxies for Objective-C classes and their methods are created at runtime by inspecting the Objective-C runtime with optionally extra annotations (provided by the project) for stuff that cannot be extracted from the runtime. That is, at runtime PyObjC creates a Python class “NSObject” that corresponds to the Objective-C class “NSObject” as defined by Apple. Every method of the Objective-C class is make available as a method on the Python proxy class as well. It is not possible to 100% reliably set up the Python proxy class for “NSObject” with all methods because Objective-C classes can grow new methods at runtime, and the introspection API that Apple provides does not have a way to detect this other than by polling. Older versions of PyObjC did poll, but even that was not relialble enough and left a race condition: def someAction_(self, sender): self.someMethod() self.button.setTitle_(“click me”) super().someOtherMethod() The call to “someMethod” used to poll the Objective-C runtime for changes. The call through super() of someOtherMethod() does not do so because of the current semantics of super (which PEP 447 tries to change). That’s a problem because “self.button.setTitle_” might load a bundle that adds “someOtherMethod” to one of our super classes. That sadly enough is not a theoretical concern, I’ve seen something like this in the past. Because of this PyObjC contains its own version of builtins.super which must be used with it (and is fully compatible with builtin.super for other classes). Recent versions of PyObjC no longer poll, primarily because polling is costly and because Objective-C classes tend to have fat APIs most of which is never used by any one program. What bothers me with PyObjC’s current approach is one the one hand that a custom super is inherently incompatible with any other library that might have a simular need, and on there other hand that I have to reimplement all logic in both object.__getattribute__ and super.__getattribute__ to be able to have a customisation of one small aspect of attribute lookup. Ronald
d.
On 22 July 2015 at 09:25, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
The PEP proposes to to replace direct access to the class __dict__ in object.__getattribute__ and super.__getattribute__ by calls to a new special method to give the metaclass more control over attribute lookup, especially for access using a super() object. This is needed for classes that cannot store (all) descriptors in the class dict for some reason, see the PEP text for a real-world example of that.
Regards,
Ronald
The PEP text (with an outdated section with benchmarks removed):
PEP: 447 Title: Add __getdescriptor__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
Abstract ========
Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__getdescriptor__`` method to a metaclass that replaces this behavior and gives more control over attribute lookup, especially when using a `super`_ object.
That is, the MRO walking loop in ``_PyType_Lookup`` and ``super.__getattribute__`` gets changed from::
def lookup(mro_list, name): for cls in mro_list: if name in cls.__dict__: return cls.__dict__
return NotFound
to::
def lookup(mro_list, name): for cls in mro_list: try: return cls.__getdescriptor__(name) except AttributeError: pass
return NotFound
The default implemention of ``__getdescriptor__`` looks in the class dictionary::
class type: def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
Rationale =========
It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand.
The ``__getdescriptor__`` method makes it possible to dynamically add attributes even when looking them up using the `super class`_.
The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency and to have a single place to implement dynamic attribute resolution for classes.
Background ----------
The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with `super`_ objects. Because of this PyObjC currently includes a custom `super`_ that must be used with its classes, as well as completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute access.
The API in this PEP makes it possible to remove the custom `super`_ and simplifies the implementation because the custom lookup behavior can be added in a central location.
.. note::
`PyObjC`_ cannot precalculate the contents of the class ``__dict__`` because Objective-C classes can grow new methods at runtime. Furthermore Objective-C classes tend to contain a lot of methods while most Python code will only use a small subset of them, this makes precalculating unnecessarily expensive.
The superclass attribute lookup hook ====================================
Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code) walk an object's MRO and currently peek in the class' ``__dict__`` to look up attributes.
With this proposal both lookup methods no longer peek in the class ``__dict__`` but call the special method ``__getdescriptor__``, which is a slot defined on the metaclass. The default implementation of that method looks up the name the class ``__dict__``, which means that attribute lookup is unchanged unless a metatype actually defines the new special method.
Aside: Attribute resolution algorithm in Python -----------------------------------------------
The attribute resolution proces as implemented by ``object.__getattribute__`` (or PyObject_GenericGetAttr`` in CPython's implementation) is fairly straightforward, but not entirely so without reading C code.
The current CPython implementation of object.__getattribute__ is basicly equivalent to the following (pseudo-) Python code (excluding some house keeping and speed tricks)::
def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple)
for base in mro: assert isinstance(base, type)
# PEP 447 will change these lines: try: return base.__dict__[name] except KeyError: pass
return None
class object: def __getattribute__(self, name): assert isinstance(name, str)
tp = type(self) descr = _PyType_Lookup(tp, name)
f = None if descr is not None: f = descr.__get__ if f is not None and descr.__set__ is not None: # Data descriptor return f(descr, self, type(self))
dict = self.__dict__ if dict is not None: try: return self.__dict__[name] except KeyError: pass
if f is not None: # Non-data descriptor return f(descr, self, type(self))
if descr is not None: # Regular class attribute return descr
raise AttributeError(name)
class super: def __getattribute__(self, name): assert isinstance(name, unicode)
if name != '__class__': starttype = self.__self_type__ mro = startype.mro()
try: idx = mro.index(self.__thisclass__)
except ValueError: pass
else: for base in mro[idx+1:]: # PEP 447 will change these lines: try: descr = base.__dict__[name] except KeyError: continue
f = descr.__get__ if f is not None: return f(descr, None if (self.__self__ is self.__self_type__) else self.__self__, starttype)
else: return descr
return object.__getattribute__(self, name)
This PEP should change the dict lookup at the lines starting at "# PEP 447" with a method call to perform the actual lookup, making is possible to affect that lookup both for normal attribute access and access through the `super proxy`_.
Note that specific classes can already completely override the default behaviour by implementing their own ``__getattribute__`` slot (with or without calling the super class implementation).
In Python code --------------
A meta type can define a method ``__getdescriptor__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::
class MetaType(type): def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
The ``__getdescriptor__`` method has as its arguments a class (which is an instance of the meta type) and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, and should raise `AttributeError`_ when the name cannot be found.
The `type`_ class provides a default implementation for ``__getdescriptor__``, that looks up the name in the class dictionary.
Example usage .............
The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names::
class UpperCaseAccess (type): def __getdescriptor__(cls, name): try: return cls.__dict__[name.upper()] except KeyError: raise AttributeError(name) from None
class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42
def M(self): return "fourtytwo"
obj = SillyObject() assert obj.m() == "fortytwo"
As mentioned earlier in this PEP a more realistic use case of this functionallity is a ``__getdescriptor__`` method that dynamicly populates the class ``__dict__`` based on attribute access, primarily when it is not possible to reliably keep the class dict in sync with its source, for example because the source used to populate ``__dict__`` is dynamic as well and does not have triggers that can be used to detect changes to that source.
An example of that are the class bridges in PyObjC: the class bridge is a Python object (class) that represents an Objective-C class and conceptually has a Python method for every Objective-C method in the Objective-C class. As with Python it is possible to add new methods to an Objective-C class, or replace existing ones, and there are no callbacks that can be used to detect this.
In C code ---------
A new slot ``tp_getdescriptor`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__getdescriptor__`` method on `type`_.
The slot has the following prototype::
PyObject* (*getdescriptorfunc)(PyTypeObject* cls, PyObject* name);
This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).
Use of this hook by the interpreter -----------------------------------
The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__getdescriptor__`` method when walking the MRO.
Other changes to the implementation -----------------------------------
The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__getdescriptor__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API.
The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__getdescriptor__``, because using the cache might not be valid for such classes.
Impact of this PEP on introspection -----------------------------------
Use of the method introduced in this PEP can affect introspection of classes with a metaclass that uses a custom ``__getdescriptor__`` method. This section lists those changes.
The items listed below are only affected by custom ``__getdescriptor__`` methods, the default implementation for ``object`` won't cause problems because that still only uses the class ``__dict__`` and won't cause visible changes to the visible behaviour of the ``object.__getattribute__``.
* ``dir`` might not show all attributes
As with a custom ``__getattribute__`` method `dir()`_ might not see all (instance) attributes when using the ``__getdescriptor__()`` method to dynamicly resolve attributes.
The solution for that is quite simple: classes using ``__getdescriptor__`` should also implement `__dir__()`_ if they want full support for the builtin `dir()`_ function.
* ``inspect.getattr_static`` might not show all attributes
The function ``inspect.getattr_static`` intentionally does not invoke ``__getattribute__`` and descriptors to avoid invoking user code during introspection with this function. The ``__getdescriptor__`` method will also be ignored and is another way in which the result of ``inspect.getattr_static`` can be different from that of ``builtin.getattr``.
* ``inspect.getmembers`` and ``inspect.get_class_attrs``
Both of these functions directly access the class __dict__ of classes along the MRO, and hence can be affected by a custom ``__getdescriptor__`` method.
**TODO**: I haven't fully worked out what the impact of this is, and if there are mitigations for those using either updates to these functions, or additional methods that users should implement to be fully compatible with these functions.
One possible mitigation is to have a custom ``__getattribute__`` for these classes that fills ``__dict__`` before returning and and defers to the default implementation for other attributes.
* Direct introspection of the class ``__dict__``
Any code that directly access the class ``__dict__`` for introspection can be affected by a custom ``__getdescriptor__`` method.
Performance impact ------------------
**WARNING**: The benchmark results in this section are old, and will be updated when I've ported the patch to the current trunk. I don't expect significant changes to the results in this section.
[snipped]
Alternative proposals ---------------------
``__getattribute_super__`` ..........................
An earlier version of this PEP used the following static method on classes::
def __getattribute_super__(cls, name, object, owner): pass
This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``.
Reuse ``tp_getattro`` .....................
It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO.
That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_.
Alternate placement of the new method .....................................
This PEP proposes to add ``__getdescriptor__`` as a method on the metaclass. An alternative would be to add it as a class method on the class itself (simular to how ``__new__`` is a `staticmethod`_ of the class and not a method of the metaclass).
The two are functionally equivalent, and there's something to be said about not requiring the use of a meta class.
References ==========
* `Issue 18181`_ contains an out of date prototype implementation
Copyright =========
This document has been placed in the public domain.
.. _`Issue 18181`: http://bugs.python.org/issue18181
.. _`super class`: http://docs.python.org/3/library/functions.html#super
.. _`super proxy`: http://docs.python.org/3/library/functions.html#super
.. _`super`: http://docs.python.org/3/library/functions.html#super
.. _`dir()`: http://docs.python.org/3/library/functions.html#dir
.. _`staticmethod`: http://docs.python.org/3/library/functions.html#staticmethod
.. _`__dir__()`: https://docs.python.org/3/reference/datamodel.html#object.__dir__
.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented
.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr
.. _`type`: http://docs.python.org/3/library/functions.html#type
.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError
.. _`PyObjC`: http://pyobjc.sourceforge.net/
.. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dimaqq%40gmail.com
I wonder whether XML RPC might be a good example? After all, it's already in the stdlib and presumably suffers from the same issue. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Ronald Oussoren<mailto:ronaldoussoren@mac.com> Sent: 7/23/2015 3:07 To: Dima Tisnek<mailto:dimaqq@gmail.com> Cc: Python Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] PEP 447 (type.__getdescriptor__)
On 23 Jul 2015, at 11:29, Dima Tisnek <dimaqq@gmail.com> wrote:
Hey I've taken time to read the PEP, my 2c... actually 1c:
Most important is to explain how this changes behaviour of Python programs.
A comprehensive set of Python examples where behaviour is changed (for better or worse) please.
The behaviour of existing python code is not changed at all. Code that directly looks in the class __dict__ might be impacted, but only when running into classes with a custom __getdescriptor__ method. I’ve listed the code in the stdlib that could be affected, but have to do a new pass of the stdlib to check if anything relevant changed since I wrote the section. I general you run into the same issues when adding a custom __getattribute__ or __getattr__ method, both of which will confuse introspection tools that assume regular attribute lookup semantics.
While I understand the concern of "superclasses of objects that gain or lose attributes at runtime" on the theoretical level, please translate that into actual Python examples.
The primary use case I have for this are classes that are proxies for external systems. There may be other uses as well, but I don’t have examples of that (other than the contrived example in the PEP). The reason I wrote the PEP in the first place is PyObjC: this project defines a proxy later between Python and Objective-C, with the goal to making it possible to write programs for Mac OS X in Python while being able to make full use of Apple’s high-level APIs. The proxy layer is mostly completely dynamic: proxies for Objective-C classes and their methods are created at runtime by inspecting the Objective-C runtime with optionally extra annotations (provided by the project) for stuff that cannot be extracted from the runtime. That is, at runtime PyObjC creates a Python class “NSObject” that corresponds to the Objective-C class “NSObject” as defined by Apple. Every method of the Objective-C class is make available as a method on the Python proxy class as well. It is not possible to 100% reliably set up the Python proxy class for “NSObject” with all methods because Objective-C classes can grow new methods at runtime, and the introspection API that Apple provides does not have a way to detect this other than by polling. Older versions of PyObjC did poll, but even that was not relialble enough and left a race condition: def someAction_(self, sender): self.someMethod() self.button.setTitle_(“click me”) super().someOtherMethod() The call to “someMethod” used to poll the Objective-C runtime for changes. The call through super() of someOtherMethod() does not do so because of the current semantics of super (which PEP 447 tries to change). That’s a problem because “self.button.setTitle_” might load a bundle that adds “someOtherMethod” to one of our super classes. That sadly enough is not a theoretical concern, I’ve seen something like this in the past. Because of this PyObjC contains its own version of builtins.super which must be used with it (and is fully compatible with builtin.super for other classes). Recent versions of PyObjC no longer poll, primarily because polling is costly and because Objective-C classes tend to have fat APIs most of which is never used by any one program. What bothers me with PyObjC’s current approach is one the one hand that a custom super is inherently incompatible with any other library that might have a simular need, and on there other hand that I have to reimplement all logic in both object.__getattribute__ and super.__getattribute__ to be able to have a customisation of one small aspect of attribute lookup. Ronald
d.
On 22 July 2015 at 09:25, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
The PEP proposes to to replace direct access to the class __dict__ in object.__getattribute__ and super.__getattribute__ by calls to a new special method to give the metaclass more control over attribute lookup, especially for access using a super() object. This is needed for classes that cannot store (all) descriptors in the class dict for some reason, see the PEP text for a real-world example of that.
Regards,
Ronald
The PEP text (with an outdated section with benchmarks removed):
PEP: 447 Title: Add __getdescriptor__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
Abstract ========
Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__getdescriptor__`` method to a metaclass that replaces this behavior and gives more control over attribute lookup, especially when using a `super`_ object.
That is, the MRO walking loop in ``_PyType_Lookup`` and ``super.__getattribute__`` gets changed from::
def lookup(mro_list, name): for cls in mro_list: if name in cls.__dict__: return cls.__dict__
return NotFound
to::
def lookup(mro_list, name): for cls in mro_list: try: return cls.__getdescriptor__(name) except AttributeError: pass
return NotFound
The default implemention of ``__getdescriptor__`` looks in the class dictionary::
class type: def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
Rationale =========
It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand.
The ``__getdescriptor__`` method makes it possible to dynamically add attributes even when looking them up using the `super class`_.
The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency and to have a single place to implement dynamic attribute resolution for classes.
Background ----------
The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with `super`_ objects. Because of this PyObjC currently includes a custom `super`_ that must be used with its classes, as well as completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute access.
The API in this PEP makes it possible to remove the custom `super`_ and simplifies the implementation because the custom lookup behavior can be added in a central location.
.. note::
`PyObjC`_ cannot precalculate the contents of the class ``__dict__`` because Objective-C classes can grow new methods at runtime. Furthermore Objective-C classes tend to contain a lot of methods while most Python code will only use a small subset of them, this makes precalculating unnecessarily expensive.
The superclass attribute lookup hook ====================================
Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code) walk an object's MRO and currently peek in the class' ``__dict__`` to look up attributes.
With this proposal both lookup methods no longer peek in the class ``__dict__`` but call the special method ``__getdescriptor__``, which is a slot defined on the metaclass. The default implementation of that method looks up the name the class ``__dict__``, which means that attribute lookup is unchanged unless a metatype actually defines the new special method.
Aside: Attribute resolution algorithm in Python -----------------------------------------------
The attribute resolution proces as implemented by ``object.__getattribute__`` (or PyObject_GenericGetAttr`` in CPython's implementation) is fairly straightforward, but not entirely so without reading C code.
The current CPython implementation of object.__getattribute__ is basicly equivalent to the following (pseudo-) Python code (excluding some house keeping and speed tricks)::
def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple)
for base in mro: assert isinstance(base, type)
# PEP 447 will change these lines: try: return base.__dict__[name] except KeyError: pass
return None
class object: def __getattribute__(self, name): assert isinstance(name, str)
tp = type(self) descr = _PyType_Lookup(tp, name)
f = None if descr is not None: f = descr.__get__ if f is not None and descr.__set__ is not None: # Data descriptor return f(descr, self, type(self))
dict = self.__dict__ if dict is not None: try: return self.__dict__[name] except KeyError: pass
if f is not None: # Non-data descriptor return f(descr, self, type(self))
if descr is not None: # Regular class attribute return descr
raise AttributeError(name)
class super: def __getattribute__(self, name): assert isinstance(name, unicode)
if name != '__class__': starttype = self.__self_type__ mro = startype.mro()
try: idx = mro.index(self.__thisclass__)
except ValueError: pass
else: for base in mro[idx+1:]: # PEP 447 will change these lines: try: descr = base.__dict__[name] except KeyError: continue
f = descr.__get__ if f is not None: return f(descr, None if (self.__self__ is self.__self_type__) else self.__self__, starttype)
else: return descr
return object.__getattribute__(self, name)
This PEP should change the dict lookup at the lines starting at "# PEP 447" with a method call to perform the actual lookup, making is possible to affect that lookup both for normal attribute access and access through the `super proxy`_.
Note that specific classes can already completely override the default behaviour by implementing their own ``__getattribute__`` slot (with or without calling the super class implementation).
In Python code --------------
A meta type can define a method ``__getdescriptor__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::
class MetaType(type): def __getdescriptor__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None
The ``__getdescriptor__`` method has as its arguments a class (which is an instance of the meta type) and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, and should raise `AttributeError`_ when the name cannot be found.
The `type`_ class provides a default implementation for ``__getdescriptor__``, that looks up the name in the class dictionary.
Example usage .............
The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names::
class UpperCaseAccess (type): def __getdescriptor__(cls, name): try: return cls.__dict__[name.upper()] except KeyError: raise AttributeError(name) from None
class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42
def M(self): return "fourtytwo"
obj = SillyObject() assert obj.m() == "fortytwo"
As mentioned earlier in this PEP a more realistic use case of this functionallity is a ``__getdescriptor__`` method that dynamicly populates the class ``__dict__`` based on attribute access, primarily when it is not possible to reliably keep the class dict in sync with its source, for example because the source used to populate ``__dict__`` is dynamic as well and does not have triggers that can be used to detect changes to that source.
An example of that are the class bridges in PyObjC: the class bridge is a Python object (class) that represents an Objective-C class and conceptually has a Python method for every Objective-C method in the Objective-C class. As with Python it is possible to add new methods to an Objective-C class, or replace existing ones, and there are no callbacks that can be used to detect this.
In C code ---------
A new slot ``tp_getdescriptor`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__getdescriptor__`` method on `type`_.
The slot has the following prototype::
PyObject* (*getdescriptorfunc)(PyTypeObject* cls, PyObject* name);
This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).
Use of this hook by the interpreter -----------------------------------
The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__getdescriptor__`` method when walking the MRO.
Other changes to the implementation -----------------------------------
The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__getdescriptor__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API.
The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__getdescriptor__``, because using the cache might not be valid for such classes.
Impact of this PEP on introspection -----------------------------------
Use of the method introduced in this PEP can affect introspection of classes with a metaclass that uses a custom ``__getdescriptor__`` method. This section lists those changes.
The items listed below are only affected by custom ``__getdescriptor__`` methods, the default implementation for ``object`` won't cause problems because that still only uses the class ``__dict__`` and won't cause visible changes to the visible behaviour of the ``object.__getattribute__``.
* ``dir`` might not show all attributes
As with a custom ``__getattribute__`` method `dir()`_ might not see all (instance) attributes when using the ``__getdescriptor__()`` method to dynamicly resolve attributes.
The solution for that is quite simple: classes using ``__getdescriptor__`` should also implement `__dir__()`_ if they want full support for the builtin `dir()`_ function.
* ``inspect.getattr_static`` might not show all attributes
The function ``inspect.getattr_static`` intentionally does not invoke ``__getattribute__`` and descriptors to avoid invoking user code during introspection with this function. The ``__getdescriptor__`` method will also be ignored and is another way in which the result of ``inspect.getattr_static`` can be different from that of ``builtin.getattr``.
* ``inspect.getmembers`` and ``inspect.get_class_attrs``
Both of these functions directly access the class __dict__ of classes along the MRO, and hence can be affected by a custom ``__getdescriptor__`` method.
**TODO**: I haven't fully worked out what the impact of this is, and if there are mitigations for those using either updates to these functions, or additional methods that users should implement to be fully compatible with these functions.
One possible mitigation is to have a custom ``__getattribute__`` for these classes that fills ``__dict__`` before returning and and defers to the default implementation for other attributes.
* Direct introspection of the class ``__dict__``
Any code that directly access the class ``__dict__`` for introspection can be affected by a custom ``__getdescriptor__`` method.
Performance impact ------------------
**WARNING**: The benchmark results in this section are old, and will be updated when I've ported the patch to the current trunk. I don't expect significant changes to the results in this section.
[snipped]
Alternative proposals ---------------------
``__getattribute_super__`` ..........................
An earlier version of this PEP used the following static method on classes::
def __getattribute_super__(cls, name, object, owner): pass
This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``.
Reuse ``tp_getattro`` .....................
It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO.
That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_.
Alternate placement of the new method .....................................
This PEP proposes to add ``__getdescriptor__`` as a method on the metaclass. An alternative would be to add it as a class method on the class itself (simular to how ``__new__`` is a `staticmethod`_ of the class and not a method of the metaclass).
The two are functionally equivalent, and there's something to be said about not requiring the use of a meta class.
References ==========
* `Issue 18181`_ contains an out of date prototype implementation
Copyright =========
This document has been placed in the public domain.
.. _`Issue 18181`: http://bugs.python.org/issue18181
.. _`super class`: http://docs.python.org/3/library/functions.html#super
.. _`super proxy`: http://docs.python.org/3/library/functions.html#super
.. _`super`: http://docs.python.org/3/library/functions.html#super
.. _`dir()`: http://docs.python.org/3/library/functions.html#dir
.. _`staticmethod`: http://docs.python.org/3/library/functions.html#staticmethod
.. _`__dir__()`: https://docs.python.org/3/reference/datamodel.html#object.__dir__
.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented
.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr
.. _`type`: http://docs.python.org/3/library/functions.html#type
.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError
.. _`PyObjC`: http://pyobjc.sourceforge.net/
.. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dimaqq%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
Hi, On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
IMO, there are two main issues with the PEP and implementation. 1. The implementation as outlined in the PEP is infinitely recursive, since the lookup of "__getdescriptor__" on type must necessarily call type.__getdescriptor__. The implementation (in C) special cases classes that inherit "__getdescriptor__" from type. This special casing should be mentioned in the PEP. 2. The actual implementation in C does not account for the case where the class of a metaclass implements __getdescriptor__ and that method returns a value when called with "__getdescriptor__" as the argument. Why was "__getattribute_super__" rejected as an alternative? No reason is given. "__getattribute_super__" has none of the problems listed above. Making super(t, obj) delegate to t.__super__(obj) seems consistent with other builtin method/classes and doesn't add corner cases to the already complex implementation of PyType_Lookup(). Cheers, Mark
On 25 Jul 2015, at 17:39, Mark Shannon <mark@hotpy.org> wrote:
Hi,
On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
IMO, there are two main issues with the PEP and implementation.
1. The implementation as outlined in the PEP is infinitely recursive, since the lookup of "__getdescriptor__" on type must necessarily call type.__getdescriptor__. The implementation (in C) special cases classes that inherit "__getdescriptor__" from type. This special casing should be mentioned in the PEP.
Sure. An alternative is to slightly change the the PEP: use __getdescriptor__ when present and directly peek into __dict__ when it is not, and then remove the default __getdescriptor__. The reason I didn’t do this in the PEP is that I prefer a programming model where I can explicitly call the default behaviour.
2. The actual implementation in C does not account for the case where the class of a metaclass implements __getdescriptor__ and that method returns a value when called with "__getdescriptor__" as the argument.
Isn’t that the same problem as with all slots, even when using __getattribute__? That is, a meta class that implements __getattribute__ to return implementations for (say) __getitem__ won’t work because the interpreter won’t call __getattribute__ to get that implementation unless it already knows that the attribute is present. Class creation, and __setattr__ on type will not only fill __dict__, but also set slots in the type structure as appropriate. The interpreter than uses those slots to determine if a special method is present. In code: class Meta1 (type): def __getitem__(self, key): return "<{} {}>".format(self.__name__, key) class Class1 (metaclass=Meta1): pass class Meta2 (type): def __getattribute__(self, name): if name == "__getitem__": return lambda key: "<{} {}>".format(self.__name__, key) return super().__getattribute__(name) class Class2 (metaclass=Meta2): pass print(Class1.__getitem__("hello")) print(Class1["hello"]) print(Class2.__getitem__("hello")) print(Class2["hello"]) The last line causes an exception: Traceback (most recent call last): File "demo-getattr.py", line 24, in <module> print(Class2["hello"]) TypeError: 'Meta2' object is not subscriptable I agree that this should be mentioned in the PEP as it can be confusing.
Why was "__getattribute_super__" rejected as an alternative? No reason is given.
"__getattribute_super__" has none of the problems listed above.
Not really. I initially used __getattribute_super__ as the name, but IIRC with the same semantics.
Making super(t, obj) delegate to t.__super__(obj) seems consistent with other builtin method/classes and doesn't add corner cases to the already complex implementation of PyType_Lookup().
A disadvantage of delegation is t.__super__ then reproduce the logic dealing with the MRO, while my proposal allows the metaclass to just deal with lookup in a specific class object. Implementation complexity is an issue, but it seems to be acceptable so far. The main problem w.r.t. additional complexity is that PyType_Lookup can now fail with an exception other than an implied AttributeError and that causes changes elsewhere in the implementation. BTW. The patch for that part is slightly uglier than it needs to be, I currently test for PyErr_Occurred() instead of using return codes in a number of places to minimise the number of lines changes to make code review easier. That needs to be changed before the code would actually be committed. Ronald P.S. Are you at the EP sprints? I’ll be there until early in the afternoon.
Cheers, Mark _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
On 26 Jul 2015, at 09:14, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 25 Jul 2015, at 17:39, Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:
Hi,
On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
IMO, there are two main issues with the PEP and implementation.
1. The implementation as outlined in the PEP is infinitely recursive, since the lookup of "__getdescriptor__" on type must necessarily call type.__getdescriptor__. The implementation (in C) special cases classes that inherit "__getdescriptor__" from type. This special casing should be mentioned in the PEP.
Sure. An alternative is to slightly change the the PEP: use __getdescriptor__ when present and directly peek into __dict__ when it is not, and then remove the default __getdescriptor__.
The reason I didn’t do this in the PEP is that I prefer a programming model where I can explicitly call the default behaviour.
I’m not sure there is a problem after all (but am willing to use the alternative I describe above), although that might be because I’m too much focussed on CPython semantics. The __getdescriptor__ method is a slot in the type object and because of that the normal attribute lookup mechanism is side-stepped for methods implemented in C. A __getdescriptor__ that is implemented on Python is looked up the normal way by the C function that gets added to the type struct for such methods, but that’s not a problem for type itself. That’s not new for __getdescriptor__ but happens for most other special methods as well, as I noted in my previous mail, and also happens for the __dict__ lookup that’s currently used (t.__dict__ is an attribute and should be lookup up using __getattribute__, …) Ronald
On 26 July 2015 at 10:41 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Jul 2015, at 09:14, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 25 Jul 2015, at 17:39, Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:
Hi,
On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
IMO, there are two main issues with the PEP and implementation.
1. The implementation as outlined in the PEP is infinitely recursive, since the lookup of "__getdescriptor__" on type must necessarily call type.__getdescriptor__. The implementation (in C) special cases classes that inherit "__getdescriptor__" from type. This special casing should be mentioned in the PEP.
Sure. An alternative is to slightly change the the PEP: use __getdescriptor__ when present and directly peek into __dict__ when it is not, and then remove the default __getdescriptor__.
The reason I didn’t do this in the PEP is that I prefer a programming model where I can explicitly call the default behaviour.
I’m not sure there is a problem after all (but am willing to use the alternative I describe above), although that might be because I’m too much focussed on CPython semantics.
The __getdescriptor__ method is a slot in the type object and because of that the normal attribute lookup mechanism is side-stepped for methods implemented in C. A __getdescriptor__ that is implemented on Python is looked up the normal way by the C function that gets added to the type struct for such methods, but that’s not a problem for type itself.
That’s not new for __getdescriptor__ but happens for most other special methods as well, as I noted in my previous mail, and also happens for the __dict__ lookup that’s currently used (t.__dict__ is an attribute and should be lookup up using __getattribute__, …)
"__getdescriptor__" is fundamentally different from "__getattribute__" in that is defined in terms of itself. object.__getattribute__ is defined in terms of type.__getattribute__, but type.__getattribute__ just does dictionary lookups. However defining type.__getattribute__ in terms of __descriptor__ causes a circularity as __descriptor__ has to be looked up on a type. So, not only must the cycle be broken by special casing "type", but that "__getdescriptor__" can be defined not only by a subclass, but also a metaclass that uses "__getdescriptor__" to define "__getdescriptor__" on the class. (and so on for meta-meta classes, etc.) Cheers, Mark
On 26 Jul 2015, at 14:18, Mark Shannon <mark@hotpy.org> wrote:
On 26 July 2015 at 10:41 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 26 Jul 2015, at 09:14, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 25 Jul 2015, at 17:39, Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org>> wrote:
Hi,
On 22/07/15 09:25, Ronald Oussoren wrote:> Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
IMO, there are two main issues with the PEP and implementation.
1. The implementation as outlined in the PEP is infinitely recursive, since the lookup of "__getdescriptor__" on type must necessarily call type.__getdescriptor__. The implementation (in C) special cases classes that inherit "__getdescriptor__" from type. This special casing should be mentioned in the PEP.
Sure. An alternative is to slightly change the the PEP: use __getdescriptor__ when present and directly peek into __dict__ when it is not, and then remove the default __getdescriptor__.
The reason I didn’t do this in the PEP is that I prefer a programming model where I can explicitly call the default behaviour.
I’m not sure there is a problem after all (but am willing to use the alternative I describe above), although that might be because I’m too much focussed on CPython semantics.
The __getdescriptor__ method is a slot in the type object and because of that the normal attribute lookup mechanism is side-stepped for methods implemented in C. A __getdescriptor__ that is implemented on Python is looked up the normal way by the C function that gets added to the type struct for such methods, but that’s not a problem for type itself.
That’s not new for __getdescriptor__ but happens for most other special methods as well, as I noted in my previous mail, and also happens for the __dict__ lookup that’s currently used (t.__dict__ is an attribute and should be lookup up using __getattribute__, …)
"__getdescriptor__" is fundamentally different from "__getattribute__" in that is defined in terms of itself.
object.__getattribute__ is defined in terms of type.__getattribute__, but type.__getattribute__ just does dictionary lookups.
object.__getattribute__ is actually defined in terms of type.__dict__ and object.__dict__. Type.__getattribute__ is at best used to to find type.__dict__.
However defining type.__getattribute__ in terms of __descriptor__ causes a circularity as __descriptor__ has to be looked up on a type.
So, not only must the cycle be broken by special casing "type", but that "__getdescriptor__" can be defined not only by a subclass, but also a metaclass that uses "__getdescriptor__" to define "__getdescriptor__" on the class. (and so on for meta-meta classes, etc.)
Are the semantics of special methods backed by a member in PyTypeObject part of Python’s semantics, or are those CPython implementation details/warts? In particular that such methods are access directly without using __getattribute__ at all (or at least only indirectly when the method is implemented in Python). That is:
class Dict (dict): ... def __getattribute__(self, nm): ... print("Get", nm) ... return dict.__getattribute__(self, nm) ...
d = Dict(a=4) d.__getitem__('a') Get __getitem__ 4 d['a'] 4
(And likewise for other special methods, which amongst others means that neither __getattribute__ nor __getdescriptor__ can be used to dynamicly add such methods to a class) In my proposed patch I do special case “type”, but that’s only intended as a (for now unbenchmarked) speed hack. The code would work just as well without the hack because the metatype’s __getdescriptor__ is looked up directly in the PyTypeObject on the C level, without using __getattribute__ and hence without having to use recursion. BTW. I wouldn’t mind dropping the default “type.__getdescriptor__” completely and reword my proposal to state that __getdescriptor__ is used when present, and otherwise __dict__ is accessed directly. That would remove the infinite recursion, as all metaclass chains should at some end up at “type” which then wouldn’t have a “__getdescriptor__”. The reason I added “type.__getdescriptor__” is that IMHO gives a nicer programming model where you can call the superclass implementation in the implementation of __getdescriptor__ in a subclass. Given the minimal semantics of “type.__getdescriptor__” loosing that wouldn’t be too bad to get a better object model. Ronald
Cheers, Mark _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
participants (8)
-
Brett Cannon -
Dima Tisnek -
Joao S. O. Bueno -
Mark Shannon -
Nick Coghlan -
Ronald Oussoren -
Steve Dower -
Terry Reedy