PEP 447: Add __getdescriptor__ to metaclasses
Hi, It’s getting a tradition for me to work on PEP 447 during the EuroPython sprints and disappear afterwards. Hopefully I can manage to avoid the latter step this year… Last year the conclusion appeared to be that this is an acceptable PEP, but Mark Shannon had a concern about a default implementation for __getdescriptor__ on type in <https://mail.python.org/pipermail/python-dev/2015-July/140938.html> (follow the link for more context):
"__getdescriptor__" is fundamentally different from "__getattribute__" in that is defined in terms of itself.
object.__getattribute__ is defined in terms of type.__getattribute__, but type.__getattribute__ just does dictionary lookups. However defining type.__getattribute__ in terms of __descriptor__ causes a circularity as __descriptor__ has to be looked up on a type.
So, not only must the cycle be broken by special casing "type", but that "__getdescriptor__" can be defined not only by a subclass, but also a metaclass that uses "__getdescriptor__" to define "__getdescriptor__" on the class. (and so on for meta-meta classes, etc.) My reaction that year is in <https://mail.python.org/pipermail/python-dev/2015-August/141114.html>. As I wrote there I did not fully understand the concerns Mark has, probably because I’m focussed too much on the implementation in CPython. If removing type.__getdescriptor__ and leaving this special method as an optional hook for subclasses of type fixes the conceptual concerns then that’s fine by me. I used type.__getdescriptor__ as the default implementation both because it appears to be cleaner to me and because this gives subclasses an easy way to access the default implementation.
The implementation of the PEP in issue 18181 <http://bugs.python.org/issue18181> does special-case type.__getdescriptor__ but only as an optimisation, the code would work just as well without that special casing because the normal attribute lookup machinery is not used when accessing special methods written in C. That is, the implementation of object.__getattribute__ directly accesses fields of the type struct at the C level. Some magic behavior appears to be necessary even without the addition of __getdescriptor__ (type is a subclass of itself, object.__getattribute__ has direct access to dict.__getitem__, …). I’m currently working on getting the patch in 18181 up-to-date w.r.t. the current trunk, the patch in the issue no longer applies cleanly. After that I’ll try to think up some tests that seriously try to break the new behaviour, and I want to update a patch I have for PyObjC to make use of the new functionality to make sure that the PEP actually fixes the issues I had w.r.t. builtin.super’s behavior. What is the best way forward after that? As before this is a change in behavior that, unsurprisingly, few core devs appear to be comfortable with evaluating, combined with new functionality that will likely see little use beyond PyObjC (although my opinions of that shouldn’t carry much weight, I thought that decorators would have limited appeal when those where introduced and couldn’t have been more wrong about that). Ronald P.S. The PEP itself: <https://www.python.org/dev/peps/pep-0447>
On Sat, 23 Jul 2016 at 05:27 Ronald Oussoren <ronaldoussoren@mac.com> wrote:
[SNIP]
What is the best way forward after that? As before this is a change in behavior that, unsurprisingly, few core devs appear to be comfortable with evaluating, combined with new functionality that will likely see little use beyond PyObjC (although my opinions of that shouldn’t carry much weight, I thought that decorators would have limited appeal when those where introduced and couldn’t have been more wrong about that).
Ronald
P.S. The PEP itself: <https://www.python.org/dev/peps/pep-0447
If the PEP is ready to be reviewed after that then either getting Guido to pronounce or for him to assign a BDFL delegate.
On 23 July 2016 at 22:26, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I’m currently working on getting the patch in 18181 up-to-date w.r.t. the current trunk, the patch in the issue no longer applies cleanly. After that I’ll try to think up some tests that seriously try to break the new behaviour, and I want to update a patch I have for PyObjC to make use of the new functionality to make sure that the PEP actually fixes the issues I had w.r.t. builtin.super’s behavior.
You may also want to check compatibility with Martin's patch for PEP 487 (__init_subclass__ and __set_name__) at http://bugs.python.org/issue27366 I don't *think* it will conflict, but "try it and see what happens" is generally a better idea for the descriptor machinery than assuming changes are going to be non-conflicting :)
What is the best way forward after that? As before this is a change in behavior that, unsurprisingly, few core devs appear to be comfortable with evaluating, combined with new functionality that will likely see little use beyond PyObjC.
You may want to explicitly ping the https://github.com/ipython/traitlets developers to see if this change would let them do anything they currently find impractical or impossible. As far as Mark's concern about a non-terminating method definition goes, I do think you need to double check how the semantics of object.__getattribute__ are formally defined. >>> class Meta(type): ... def __getattribute__(self, attr): ... print("Via metaclass!") ... return super().__getattribute__(attr) ... >>> class Example(metaclass=Meta): pass ... >>> Example.mro() Via metaclass! [<class '__main__.Example'>, <class 'object'>] Where the current PEP risks falling into unbounded recursion is that it appears to propose that the default type.__getdescriptor__ implementation be defined in terms of accessing cls.__dict__, but a normal Python level access to "cls.__dict__" would go through the descriptor machinery, triggering an infinite regress. The PEP needs to be explicit that where "cls.__dict__" is written in the definitions of both the old and new lookup semantics, it is *not* referring to a normal class attribute lookup, but rather to the interpreter's privileged access to the class namespace (e.g. direct 'tp_dict' access in CPython). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 Jul 2016, at 12:37, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 23 July 2016 at 22:26, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I’m currently working on getting the patch in 18181 up-to-date w.r.t. the current trunk, the patch in the issue no longer applies cleanly. After that I’ll try to think up some tests that seriously try to break the new behaviour, and I want to update a patch I have for PyObjC to make use of the new functionality to make sure that the PEP actually fixes the issues I had w.r.t. builtin.super’s behavior.
You may also want to check compatibility with Martin's patch for PEP 487 (__init_subclass__ and __set_name__) at http://bugs.python.org/issue27366
I don't *think* it will conflict, but "try it and see what happens" is generally a better idea for the descriptor machinery than assuming changes are going to be non-conflicting :)
I also don’t think the two will conflict, but that’s based on a superficial read of that PEP the last time it was posted on python-dev. PEP 487 and 447 affect different parts of the object model, in particular PEP 487 doesn’t affect attribute lookup.
What is the best way forward after that? As before this is a change in behavior that, unsurprisingly, few core devs appear to be comfortable with evaluating, combined with new functionality that will likely see little use beyond PyObjC.
You may want to explicitly ping the https://github.com/ipython/traitlets developers to see if this change would let them do anything they currently find impractical or impossible.
I’ll ask them.
As far as Mark's concern about a non-terminating method definition goes, I do think you need to double check how the semantics of object.__getattribute__ are formally defined.
class Meta(type): ... def __getattribute__(self, attr): ... print("Via metaclass!") ... return super().__getattribute__(attr) ... class Example(metaclass=Meta): pass ... Example.mro() Via metaclass! [<class '__main__.Example'>, <class 'object'>]
Where the current PEP risks falling into unbounded recursion is that it appears to propose that the default type.__getdescriptor__ implementation be defined in terms of accessing cls.__dict__, but a normal Python level access to "cls.__dict__" would go through the descriptor machinery, triggering an infinite regress.
The PEP needs to be explicit that where "cls.__dict__" is written in the definitions of both the old and new lookup semantics, it is *not* referring to a normal class attribute lookup, but rather to the interpreter's privileged access to the class namespace (e.g. direct 'tp_dict' access in CPython).
On first glance the same is true for all access to dunder attributes in sample code for the PEP, a similar example could be written for __get__ or __set__. I have to think a bit more about how to clearly describe this. I’m currently coaxing PyObjC into using PEP 447 when that’s available and that involves several layers of metaclasses in C and that’s annoyingly hard to debug when the code doesn’t do what I want like it does now. But on the other hand, that’s why wanted to use PyObjC to validate the PEP in the first place. Back to wrangling C code, Ronald
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 Jul 2016, at 13:06, Ronald Oussoren <ronaldoussoren@mac.com <mailto:ronaldoussoren@mac.com>> wrote:
…
But on the other hand, that’s why wanted to use PyObjC to validate the PEP in the first place.
I’ve hit a fairly significant issue with this, PyObjC’s super contains more magic than just this magic that would be fixed by PEP 447. I don’t think I’ll be able to finish work on PEP 447 this week because of that, and in the worst case will have to retire the PEP. The problem is as follows: to be able to map all of Cocoa’s methods to Python PyObjC creates two proxy classes for every Cocoa class: the regular class and its metaclass. The latter is used to store class methods. This is needed because Objective-C classes can have instance and class methods with the same name, as an example: @interface NSObject -(NSString*)description; +(NSString*)description @end The first declaration for “description” is an instance method, the second is a class method. The Python metaclass is mostly a hidden detail, users don’t explicitly interact with these classes and use the normal Python convention for defining class methods. This works fine, problems starts when you want to subclass in Python and override the class method: class MyClass (NSObject): @classmethod def description(cls): return “hello there from %r” % (super(MyClass, cls).description()) If you’re used to normal Python code there’s nothing wrong here, but getting this to work required some magic in objc.super to ensure that its __getattribute__ looks in the metaclass in this case and not the regular class. The current PEP447-ised version of PyObjC has a number of test failures because builtin.super obviously doesn’t contain this hack (and shouldn’t). I think I can fix this for modern code that uses an argumentless call to super by replacing the cell containing the __class__ reference when moving the method from the regular class to the instance class. That would obviously not work for the code I showed earlier, but that at least won’t fail silently and the error message is specific enough that I can include it in PyObjC’s documentation. Ronald
Back to wrangling C code,
Ronald
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia
_______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> https://mail.python.org/mailman/listinfo/python-dev <https://mail.python.org/mailman/listinfo/python-dev> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com <https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com>
Hi Ronald, The feature freeze for 3.6 is closing in a few days; 3.6b1 will go out this weekend. Did you overcome the issue, or does your PEP need to be postponed until 3.7? --Guido On Sun, Jul 24, 2016 at 9:58 PM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 24 Jul 2016, at 13:06, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
…
But on the other hand, that’s why wanted to use PyObjC to validate the PEP in the first place.
I’ve hit a fairly significant issue with this, PyObjC’s super contains more magic than just this magic that would be fixed by PEP 447. I don’t think I’ll be able to finish work on PEP 447 this week because of that, and in the worst case will have to retire the PEP.
The problem is as follows: to be able to map all of Cocoa’s methods to Python PyObjC creates two proxy classes for every Cocoa class: the regular class and its metaclass. The latter is used to store class methods. This is needed because Objective-C classes can have instance and class methods with the same name, as an example:
@interface NSObject -(NSString*)description; +(NSString*)description @end
The first declaration for “description” is an instance method, the second is a class method. The Python metaclass is mostly a hidden detail, users don’t explicitly interact with these classes and use the normal Python convention for defining class methods.
This works fine, problems starts when you want to subclass in Python and override the class method:
class MyClass (NSObject): @classmethod def description(cls): return “hello there from %r” % (super(MyClass, cls).description())
If you’re used to normal Python code there’s nothing wrong here, but getting this to work required some magic in objc.super to ensure that its __getattribute__ looks in the metaclass in this case and not the regular class. The current PEP447-ised version of PyObjC has a number of test failures because builtin.super obviously doesn’t contain this hack (and shouldn’t).
I think I can fix this for modern code that uses an argumentless call to super by replacing the cell containing the __class__ reference when moving the method from the regular class to the instance class. That would obviously not work for the code I showed earlier, but that at least won’t fail silently and the error message is specific enough that I can include it in PyObjC’s documentation.
Ronald
Back to wrangling C code,
Ronald
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
participants (4)
-
Brett Cannon -
Guido van Rossum -
Nick Coghlan -
Ronald Oussoren