Hi, This PEP proposed to add a __locallookup__ slot to type objects, which is used by _PyType_Lookup and super_getattro instead of peeking in the tp_dict of classes. The PEP text explains why this is needed. Differences with the previous version: * Better explanation of why this is a useful addition * type.__locallookup__ is no longer optional. * I've added benchmarking results using pybench. (using the patch attached to issue 18181) Ronald PEP: 447 Title: Add __locallookup__ method to metaclass Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren <ronaldoussoren@mac.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013 Abstract ======== Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek in the ``__dict__`` of classes on the MRO for a class when looking for an attribute. This PEP adds an optional ``__locallookup__`` method to a metaclass that can be used to override this behavior. Rationale ========= It is currently not possible to influence how the `super class`_ looks up attributes (that is, ``super.__getattribute__`` unconditionally peeks in the class ``__dict__``), and that can be problematic for dynamic classes that can grow new methods on demand. The ``__locallookup__`` method makes it possible to dynamicly add attributes even when looking them up using the `super class`_. The new method affects ``object.__getattribute__`` (and `PyObject_GenericGetAttr`_) as well for consistency. Background ---------- The current behavior of ``super.__getattribute__`` causes problems for classes that are dynamic proxies for other (non-Python) classes or types, an example of which is `PyObjC`_. PyObjC creates a Python class for every class in the Objective-C runtime, and looks up methods in the Objective-C runtime when they are used. This works fine for normal access, but doesn't work for access with ``super`` objects. Because of this PyObjC currently includes a custom ``super`` that must be used with its classes. The API in this PEP makes it possible to remove the custom ``super`` and simplifies the implementation because the custom lookup behavior can be added in a central location. The superclass attribute lookup hook ==================================== Both ``super.__getattribute__`` and ``object.__getattribute__`` (or `PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the class' ``__dict__`` to look up attributes. A way to affect this lookup is using a method on the meta class for the type, that by default looks up the name in the class ``__dict__``. In Python code -------------- A meta type can define a method ``__locallookup__`` that is called during attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``:: class MetaType(type): def __locallookup__(cls, name): try: return cls.__dict__[name] except KeyError: raise AttributeError(name) from None The ``__locallookup__`` method has as its arguments a class and the name of the attribute that is looked up. It should return the value of the attribute without invoking descriptors, or raise `AttributeError`_ when the name cannot be found. The `type`_ class provides a default implementation for ``__locallookup__``, that looks up the name in the class dictionary. Example usage ............. The code below implements a silly metaclass that redirects attribute lookup to uppercase versions of names:: class UpperCaseAccess (type): def __locallookup__(cls, name): return cls.__dict__[name.upper()] class SillyObject (metaclass=UpperCaseAccess): def m(self): return 42 def M(self): return "fourtytwo" obj = SillyObject() assert obj.m() == "fortytwo" In C code --------- A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot corresponds to the ``__locallookup__`` method on `type`_. The slot has the following prototype:: PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name); This method should lookup *name* in the namespace of *cls*, without looking at superclasses, and should not invoke descriptors. The method returns ``NULL`` without setting an exception when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference). Use of this hook by the interpreter ----------------------------------- The new method is required for metatypes and as such is defined on `type_`. Both ``super.__getattribute__`` and ``object.__getattribute__``/`PyObject_GenericGetAttr`_ (through ``_PyType_Lookup``) use the this ``__locallookup__`` method when walking the MRO. Other changes to the implementation ----------------------------------- The change for `PyObject_GenericGetAttr`_ will be done by changing the private function ``_PyType_Lookup``. This currently returns a borrowed reference, but must return a new reference when the ``__locallookup__`` method is present. Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``, this will cause compile-time errors for all out-of-tree users of this private API. The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes that have a metaclass that overrides ``__locallookup__``, because using the cache might not be valid for such classes. Performance impact ------------------ The pybench output below compares an implementation of this PEP with the regular source tree, both based on changeset a5681f50bae2, run on an idle machine an Core i7 processor running Centos 6.4. Even though the machine was idle there were clear differences between runs, I've seen difference in "minimum time" vary from -0.1% to +1.5%, with simular (but slightly smaller) differences in the "average time" difference. .. :: ------------------------------------------------------------------------------- PYBENCH 2.1 ------------------------------------------------------------------------------- * using CPython 3.4.0a0 (default, Jul 29 2013, 13:01:34) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] * disabled garbage collection * system check interval set to maximum: 2147483647 * using timer: time.perf_counter * timer: resolution=1e-09, implementation=clock_gettime(CLOCK_MONOTONIC) ------------------------------------------------------------------------------- Benchmark: pep447.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default-pep447/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 14:09:12 (#default) Unicode: UCS4 ------------------------------------------------------------------------------- Comparing with: default.pybench ------------------------------------------------------------------------------- Rounds: 10 Warp: 10 Timer: time.perf_counter Machine Details: Platform ID: Linux-2.6.32-358.114.1.openstack.el6.x86_64-x86_64-with-centos-6.4-Final Processor: x86_64 Python: Implementation: CPython Executable: /tmp/default/bin/python3 Version: 3.4.0a0 Compiler: GCC 4.4.7 20120313 (Red Hat 4.4.7-3) Bits: 64bit Build: Jul 29 2013 13:01:34 (#default) Unicode: UCS4 Test minimum run-time average run-time this other diff this other diff ------------------------------------------------------------------------------- BuiltinFunctionCalls: 45ms 44ms +1.3% 45ms 44ms +1.3% BuiltinMethodLookup: 26ms 27ms -2.4% 27ms 27ms -2.2% CompareFloats: 33ms 34ms -0.7% 33ms 34ms -1.1% CompareFloatsIntegers: 66ms 67ms -0.9% 66ms 67ms -0.8% CompareIntegers: 51ms 50ms +0.9% 51ms 50ms +0.8% CompareInternedStrings: 34ms 33ms +0.4% 34ms 34ms -0.4% CompareLongs: 29ms 29ms -0.1% 29ms 29ms -0.0% CompareStrings: 43ms 44ms -1.8% 44ms 44ms -1.8% ComplexPythonFunctionCalls: 44ms 42ms +3.9% 44ms 42ms +4.1% ConcatStrings: 33ms 33ms -0.4% 33ms 33ms -1.0% CreateInstances: 47ms 48ms -2.9% 47ms 49ms -3.4% CreateNewInstances: 35ms 36ms -2.5% 36ms 36ms -2.5% CreateStringsWithConcat: 69ms 70ms -0.7% 69ms 70ms -0.9% DictCreation: 52ms 50ms +3.1% 52ms 50ms +3.0% DictWithFloatKeys: 40ms 44ms -10.1% 43ms 45ms -5.8% DictWithIntegerKeys: 32ms 36ms -11.2% 35ms 37ms -4.6% DictWithStringKeys: 29ms 34ms -15.7% 35ms 40ms -11.0% ForLoops: 30ms 29ms +2.2% 30ms 29ms +2.2% IfThenElse: 38ms 41ms -6.7% 38ms 41ms -6.9% ListSlicing: 36ms 36ms -0.7% 36ms 37ms -1.3% NestedForLoops: 43ms 45ms -3.1% 43ms 45ms -3.2% NestedListComprehensions: 39ms 40ms -1.7% 39ms 40ms -2.1% NormalClassAttribute: 86ms 82ms +5.1% 86ms 82ms +5.0% NormalInstanceAttribute: 42ms 42ms +0.3% 42ms 42ms +0.0% PythonFunctionCalls: 39ms 38ms +3.5% 39ms 38ms +2.8% PythonMethodCalls: 51ms 49ms +3.0% 51ms 50ms +2.8% Recursion: 67ms 68ms -1.4% 67ms 68ms -1.4% SecondImport: 41ms 36ms +12.5% 41ms 36ms +12.6% SecondPackageImport: 45ms 40ms +13.1% 45ms 40ms +13.2% SecondSubmoduleImport: 92ms 95ms -2.4% 95ms 98ms -3.6% SimpleComplexArithmetic: 28ms 28ms -0.1% 28ms 28ms -0.2% SimpleDictManipulation: 57ms 57ms -1.0% 57ms 58ms -1.0% SimpleFloatArithmetic: 29ms 28ms +4.7% 29ms 28ms +4.9% SimpleIntFloatArithmetic: 37ms 41ms -8.5% 37ms 41ms -8.7% SimpleIntegerArithmetic: 37ms 41ms -9.4% 37ms 42ms -10.2% SimpleListComprehensions: 33ms 33ms -1.9% 33ms 34ms -2.9% SimpleListManipulation: 28ms 30ms -4.3% 29ms 30ms -4.1% SimpleLongArithmetic: 26ms 26ms +0.5% 26ms 26ms +0.5% SmallLists: 40ms 40ms +0.1% 40ms 40ms +0.1% SmallTuples: 46ms 47ms -2.4% 46ms 48ms -3.0% SpecialClassAttribute: 126ms 120ms +4.7% 126ms 121ms +4.4% SpecialInstanceAttribute: 42ms 42ms +0.6% 42ms 42ms +0.8% StringMappings: 94ms 91ms +3.9% 94ms 91ms +3.8% StringPredicates: 48ms 49ms -1.7% 48ms 49ms -2.1% StringSlicing: 45ms 45ms +1.4% 46ms 45ms +1.5% TryExcept: 23ms 22ms +4.9% 23ms 22ms +4.8% TryFinally: 32ms 32ms -0.1% 32ms 32ms +0.1% TryRaiseExcept: 17ms 17ms +0.9% 17ms 17ms +0.5% TupleSlicing: 49ms 48ms +1.1% 49ms 49ms +1.0% WithFinally: 48ms 47ms +2.3% 48ms 47ms +2.4% WithRaiseExcept: 45ms 44ms +0.8% 45ms 45ms +0.5% ------------------------------------------------------------------------------- Totals: 2284ms 2287ms -0.1% 2306ms 2308ms -0.1% (this=pep447.pybench, other=default.pybench) Alternative proposals --------------------- ``__getattribute_super__`` .......................... An earlier version of this PEP used the following static method on classes:: def __getattribute_super__(cls, name, object, owner): pass This method performed name lookup as well as invoking descriptors and was necessarily limited to working only with ``super.__getattribute__``. Reuse ``tp_getattro`` ..................... It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. That won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. References ========== * `Issue 18181`_ contains a prototype implementation Copyright ========= This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html#super .. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented .. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr .. _`type`: http://docs.python.org/3/library/functions.html#type .. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError .. _`PyObjC`: http://pyobjc.sourceforge.net/ .. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod