[Python-Dev] PEP 447: hooking into super() attribute resolution

[Ronald Oussoren]

Hi,

I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower.

In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo:

    class meta (type):
        def __locallookup__(cls, name):
            return type.__locallookup__(cls, name.upper())

    class S (metaclass=meta):
        def m(self):
            return 42

        def M(self):
             return "fortytwo"

    class SS (S):
        def m(self):
            return 21

        def M(self):
           return "twentyone"

    o = SS()
    print("self", o.m())
    print("super", super(SS, o).m())

The last line currently prints "super 42" and would print "super fortytwo" with my proposal.

A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details).  I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex.

Ronald



PEP: 447
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren at mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, ?


Abstract
========

In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.


Rationale
=========

Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.

This new hook makes it possible to affect attribute lookup for both normal
attribute lookup and lookup through the `super class`_.


The superclass attribute lookup hook
====================================

Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the
class' ``__dict__`` to look up attributes. A way to affect this lookup is
using a method on the meta class for the type, that by default looks up
the name in the class ``__dict__``.

In Python code
--------------

A meta type can define a method ``__locallookup__`` that is called during
attribute resolution by both ``super.__getattribute__`` and ``object.__getattribute``::

    class MetaType (type):

        def __locallookup__(cls, name):
            try:
                return cls.__dict__[name]
            except KeyError:
                raise AttributeError(name) from None

The example method above is pseudo code for the implementation of this method on
`type`_ (the actual implementation is in C, and doesn't suffer from the recursion
problem in this example).

The ``__locallookup__`` method has as its arguments a class and the name of the attribute
that is looked up. It should return the value of the attribute without invoking descriptors,
or raise `AttributeError`_ when the name cannot be found.


In C code
---------

A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot
corresponds to the ``__locallookup__`` method on `type`_.

The slot has the following prototype::

    PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name);

This method should lookup *name* in the namespace of *cls*, without looking at superclasses,
and should not invoke descriptors. The method returns ``NULL`` without setting an exception
when the *name* cannot be found, and returns a new reference otherwise (not a borrowed reference).


Usage of this hook
------------------

The new method will be defined for `type`_, and will peek in the class dictionary::

    static PyObject*
    type_locallookup(PyTypeObject* cls, PyObject* name)
    {
        PyObject* res;
        if (!cls->tp_dict) {
	    return NULL;
	}

	res = PyDict_GetItem(cls, name);
	Py_XINCREF(res);
	return res;
    }

The new method will be used by both `PyObject_GenericGetAttr`_ and
``super.__getattribute__`` instead of peeking in a type's ``tp_dict``.


Alternative proposals
---------------------

``__getattribute_super__``
..........................

An earlier version of this PEP used the following static method on classes::

    def __getattribute_super__(cls, name, object, owner): pass

This method performed name lookup as well as invoking descriptors and was necessarily
limited to working only with ``super.__getattribute__``.


Reuse ``tp_getattro``
.....................

It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand.  A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.

That won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.


Open Issues
===========

* The names of the new slot and magic method are far from settled.

* Should the python method raise an exception or return a magic value (such as the
  `NotImplemented`_ return value used by comparison operators). The latter could be
  slightly faster because it doesn't have to overhead of setting up exception state, but
  makes it impossible to use that value as an attribute on a class.

* The proposed change to `PyObject_GenericGetAttr`_ will probably cause problems with the
  attribute lookup cache (MCACHE):

  1. That code stores borrowed references, which won't work when the hook is present. That
     is mostly fixable, but at the cost of possibly keeping garbage alive.

  2. Caching isn't an option when a hook might execute arbitrary code (and there hence is
     no reason to assume that the hooks return value won't change later on).

     The only workaround I could find for this is to make the hook a fallback (that is,
     more like ``__getattr__`` than ``__getattribute__``).


References
==========

* `Issue 18181`_ contains a prototype implementation


Copyright
=========

This document has been placed in the public domain.

.. _`Issue 18181`: http://bugs.python.org/issue18181

.. _`super class`: http://docs.python.org/3/library/functions.html#super

.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented

.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr

.. _`type`: http://docs.python.org/3/library/functions.html#type

.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError