[Python-ideas] PEP 447 revisited

[Ronald Oussoren]

Hi,

After a long hiatus I’ve done some updates to PEP 447 which proposes a new metaclass method that’s used in attribute resolution for normal and super instances. There have been two updates, the first one is trivial, the proposed method has a new name (__getdescriptor__).  The second change to the PEP is to add a Python pseudo implementation of object.__getattribute__ and super.__getattribute__ to make it easier to reason about the impact of the proposal.

I’d like to move forward with this PEP, either to rejection or (preferable) to acceptance of the feature in some form. That said, I’m not too attached to the exact proposal, it just seems to be the minmal clean change that can be used to implement my use case for this.

My use case is fairly obscure, but hopefully it is not too obscure :-).  The problem I have at the moment is basically that it is not possible to hook into the attribute resolution algorithm used by super.__getattribute__ and this PEP would solve that.

My use case for this PEP is PyObjC, the PEP would make it possible to remove a custom “super” class used in that project. I’ll try to sketch what PyObjC does and why the current super is a problem in the paragraphs below.

PyObjC is a bridge between Python and Objective-C.  The bit that’s important for this discussion is that every Objective-C object and class can be proxied into Python code. That’s done completely dynamically: the PyObjC bridge reads information from the Objective-C runtime (using a public API for that) to determine which classes are present there and which methods those classes have.  

Accessing the information on methods is done on demand, the bridge only looks for a method when Python code tries to access it. There are two reasons for that, the first one is performance: extracting method information eagerly is too expensive because there are a lot of them and Python code typically uses only a fraction of them.  The second reason is more important than that: Objective-C classes are almost as dynamic Python classes and it is possible to add new methods at runtime either by loading add-on bundles (“Categories”) or by interacting with the Objective-C runtime. Both are actually used by Apple’s frameworks.   There are no hooks that can be used to detect there modification, the only option I’ve found that can be used to keep the Python representation of a class in sync with the Objective-C representation is to eagerly scan classes every time they might be accessed, for example in the __getattribute__ of the proxies for Objective-C classes and instances.

That’s terribly expensive, and still leaves a race condition when using super, in code like the code below the superclass might grow a new method between the call to the python method and using the superclass method:

     def myMethod(self):
          self.objectiveCMethod()
          super().otherMethod()

Because of this the current PyObjC release doesn’t even try to keep the Python representation in sync, but always lazily looks for methods (but with a cache for all found methods to avoid the overhead of looking for them when methods are used multiple times). As that definitely will break builtin.super PyObjC also includes a custom super implementation that must be used.  That works, but can lead to confusing errors when users forget to add “from objc import super” to modules that use super in subclasses from Objective-C classes.

The performance impact on CPython seemed to be minimal according to the testing I performed last year, but I have no idea what the impact would be on other implementation (in particular PyPy’s JIT).

A link to the PEP: http://legacy.python.org/dev/peps/pep-0447/

I’d really appreciate further feedback on this PEP. 

Regards,

   Ronald