
Hi, After a long hiatus I’ve done some updates to PEP 447 which proposes a new metaclass method that’s used in attribute resolution for normal and super instances. There have been two updates, the first one is trivial, the proposed method has a new name (__getdescriptor__). The second change to the PEP is to add a Python pseudo implementation of object.__getattribute__ and super.__getattribute__ to make it easier to reason about the impact of the proposal. I’d like to move forward with this PEP, either to rejection or (preferable) to acceptance of the feature in some form. That said, I’m not too attached to the exact proposal, it just seems to be the minmal clean change that can be used to implement my use case for this. My use case is fairly obscure, but hopefully it is not too obscure :-). The problem I have at the moment is basically that it is not possible to hook into the attribute resolution algorithm used by super.__getattribute__ and this PEP would solve that. My use case for this PEP is PyObjC, the PEP would make it possible to remove a custom “super” class used in that project. I’ll try to sketch what PyObjC does and why the current super is a problem in the paragraphs below. PyObjC is a bridge between Python and Objective-C. The bit that’s important for this discussion is that every Objective-C object and class can be proxied into Python code. That’s done completely dynamically: the PyObjC bridge reads information from the Objective-C runtime (using a public API for that) to determine which classes are present there and which methods those classes have. Accessing the information on methods is done on demand, the bridge only looks for a method when Python code tries to access it. There are two reasons for that, the first one is performance: extracting method information eagerly is too expensive because there are a lot of them and Python code typically uses only a fraction of them. The second reason is more important than that: Objective-C classes are almost as dynamic Python classes and it is possible to add new methods at runtime either by loading add-on bundles (“Categories”) or by interacting with the Objective-C runtime. Both are actually used by Apple’s frameworks. There are no hooks that can be used to detect there modification, the only option I’ve found that can be used to keep the Python representation of a class in sync with the Objective-C representation is to eagerly scan classes every time they might be accessed, for example in the __getattribute__ of the proxies for Objective-C classes and instances. That’s terribly expensive, and still leaves a race condition when using super, in code like the code below the superclass might grow a new method between the call to the python method and using the superclass method: def myMethod(self): self.objectiveCMethod() super().otherMethod() Because of this the current PyObjC release doesn’t even try to keep the Python representation in sync, but always lazily looks for methods (but with a cache for all found methods to avoid the overhead of looking for them when methods are used multiple times). As that definitely will break builtin.super PyObjC also includes a custom super implementation that must be used. That works, but can lead to confusing errors when users forget to add “from objc import super” to modules that use super in subclasses from Objective-C classes. The performance impact on CPython seemed to be minimal according to the testing I performed last year, but I have no idea what the impact would be on other implementation (in particular PyPy’s JIT). A link to the PEP: http://legacy.python.org/dev/peps/pep-0447/ I’d really appreciate further feedback on this PEP. Regards, Ronald

On 26 July 2014 18:03, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
The use case seems reasonable to me, and the new slot name seems much easier to document and explain than the previous iteration. I'd like to see the PEP look into the inspect module and consider the consequences for the functions there (e.g. another way for getattr_static to miss methods), as well as any possible implications for dir(). We had a few issues there with the enum changes for 3.4 (and some more serious ones with Argument Clinic) - it's not a blocker, it's just nice going in to have some idea of the impact going in :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 Jul 2014, at 13:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
Some Australian guy you may know suggested the name the last time I posted the PEP for review, and I liked the name. Naming is hard...
I agree that it is useful to explain those consequences. The consequences for dir() should be similar to those for __getattribute__ itself: if you override the default implementation you should implement __dir__ to match, or live with the inconsistency. There should be little or no impact on inspect, other then that getattr_static may not work as expected when using a custom implemention of __getdescriptor__ because the class __dict__ may not contain the values you need. There’s nothing that can be done about that, the entire point of getattr_static is to avoid triggering custom attribute lookup code. inspect.getmembers and inspect.get_class_attrs, look directly at the class __dict__, and hence might not show everything that’s available through the class when using a custom __getdescriptor__ method. I have to think about the consequences and possible mitigation of those consequences a bit, not just for this PEP but for the current PyObjC implementation as well. Anyways, I’ll add a section about introspection to the PEP that describes these issues and their consequences. Ronald

On 26 July 2014 18:03, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
The use case seems reasonable to me, and the new slot name seems much easier to document and explain than the previous iteration. I'd like to see the PEP look into the inspect module and consider the consequences for the functions there (e.g. another way for getattr_static to miss methods), as well as any possible implications for dir(). We had a few issues there with the enum changes for 3.4 (and some more serious ones with Argument Clinic) - it's not a blocker, it's just nice going in to have some idea of the impact going in :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 Jul 2014, at 13:59, Nick Coghlan <ncoghlan@gmail.com> wrote:
Some Australian guy you may know suggested the name the last time I posted the PEP for review, and I liked the name. Naming is hard...
I agree that it is useful to explain those consequences. The consequences for dir() should be similar to those for __getattribute__ itself: if you override the default implementation you should implement __dir__ to match, or live with the inconsistency. There should be little or no impact on inspect, other then that getattr_static may not work as expected when using a custom implemention of __getdescriptor__ because the class __dict__ may not contain the values you need. There’s nothing that can be done about that, the entire point of getattr_static is to avoid triggering custom attribute lookup code. inspect.getmembers and inspect.get_class_attrs, look directly at the class __dict__, and hence might not show everything that’s available through the class when using a custom __getdescriptor__ method. I have to think about the consequences and possible mitigation of those consequences a bit, not just for this PEP but for the current PyObjC implementation as well. Anyways, I’ll add a section about introspection to the PEP that describes these issues and their consequences. Ronald
participants (2)
-
Nick Coghlan
-
Ronald Oussoren