[Python-Dev] PEP 487: Simpler customization of class creation

Martin Teichmann lkb.teichmann at gmail.com
Fri Jun 24 03:41:36 EDT 2016


Hi list,

just recently, I posted about the implementation of PEP 487. The
discussion quickly diverted to PEP 520, which happened to be
strongly related.

Hoping to get some comments about the rest of PEP 487, I took
out the part that is also in PEP 520. I attached the new version of
the PEP. The implementation can be found on the Python issue tracker:
http://bugs.python.org/issue27366

So PEP 487 is about simplifying the customization of class creation.
Currently, this is done via metaclasses, which are very powerful, but
often inflexible, as it is hard to combine two metaclasses. PEP 487
proposes a new metaclass which calls a method on all newly created
subclasses. This way, in order to customize the creation of subclasses,
one just needs to write a simple method.

An absolutely classic example for metaclasses is the need to tell descriptors
who they belong to. There are many large frameworks out there, e.g.
enthought's traits, IPython's traitlets, Django's forms and many more.
Their problem is: they're all fully incompatible. It's really hard to inherit
from two classes which have different metaclasses.

PEP 487 proposes to have one simple metaclass, which can do all those
frameworks need, making them all compatible. As an example, imagine
the framework has a generic descriptor called Integer, which describes,
well, an integer. Typically you use it like that:

    class MyClass(FrameworkBaseClass):
        my_value = Integer()

how does my_value know how it's called, how it should put its data into the
object's __dict__? Well, this is what the framework's metaclass is for.
With PEP 487, a framework doesn't need to declare an own metaclass
anymore, but simply uses types.Object of PEP 487 as a base class:

    class FrameworkBaseClass(types.Object):
        def __init_subclass__(cls):
            super().__init_subclass__()
            for k, v in cls.__dict__.items():
                if isinstance(v, FrameworkDescriptorBase):
                    v.__set_owner__(cls, name)

and all the framework's descriptors know their name. And if another framework
should be used as well: no problem, they just work together easily.

Actually, the above example is just that common, that PEP 487 includes
it directly:
a method __set_owner__ is called for every descriptor. That could make most
descriptors in frameworks work out of the box.

So now I am hoping for comments!

Greetings

Martin

New version of the PEP follows:

PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichmann at gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016
Replaces: 422


Abstract
========

Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
a hook to initialize attributes.

Those hooks should at first be defined in a metaclass in the standard
library, with the option that this metaclass eventually becomes the
default ``type`` metaclass.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.


Background
==========

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.

Proposal
========

While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.

Those three use cases can easily be performed by just one metaclass. If
this metaclass is put into the standard library, and all libraries that
wish to customize class creation use this very metaclass, no combination
of metaclasses is necessary anymore. Said metaclass should live in the
``types`` module under the name ``Type``. This should hint the user that
in the future, this metaclass may become the default metaclass ``type``.

The three use cases are achieved as follows:

1. The metaclass contains an ``__init_subclass__`` hook that initializes
   all subclasses of a given class,
2. the metaclass calls a ``__set_owner__`` hook on all the attribute
   (descriptors) defined in the class, and

For ease of use, a base class ``types.Object`` is defined, which uses said
metaclass and contains an empty stub for the hook described for use case 1.
It will eventually become the new replacement for the standard ``object``.

As an example, the first use case looks as follows::

   >>> class SpamBase(types.Object):
   ...    # this is implicitly a @classmethod
   ...    def __init_subclass__(cls, **kwargs):
   ...        cls.class_args = kwargs
   ...        super().__init_subclass__(cls, **kwargs)

   >>> class Spam(SpamBase, a=1, b="b"):
   ...    pass

   >>> Spam.class_args
   {'a': 1, 'b': 'b'}

The base class ``types.Object`` contains an empty ``__init_subclass__``
method which serves as an endpoint for cooperative multiple inheritance.
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.

The second part of the proposal adds an ``__set_owner__``
initializer for class attributes, especially if they are descriptors.
Descriptors are defined in the body of a
class, but they do not know anything about that class, they do not
even know the name they are accessed with. They do get to know their
owner once ``__get__`` is called, but still they do not know their
name. This is unfortunate, for example they cannot put their
associated value into their object's ``__dict__`` under their name,
since they do not know that name.  This problem has been solved many
times, and is one of the most important reasons to have a metaclass in
a library. While it would be easy to implement such a mechanism using
the first part of the proposal, it makes sense to have one solution
for this problem for everyone.

To give an example of its usage, imagine a descriptor representing weak
referenced values::

    import weakref

    class WeakAttribute:
        def __get__(self, instance, owner):
            return instance.__dict__[self.name]

        def __set__(self, instance, value):
            instance.__dict__[self.name] = weakref.ref(value)

        # this is the new initializer:
        def __set_owner__(self, owner, name):
            self.name = name

While this example looks very trivial, it should be noted that until
now such an attribute cannot be defined without the use of a metaclass.
And given that such a metaclass can make life very hard, this kind of
attribute does not exist yet.


Key Benefits
============


Easier inheritance of definition time behaviour
-----------------------------------------------

Understanding Python's metaclasses requires a deep understanding of
the type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the code,
the metaclass hint, the actual metaclass, the class object, instances of the
class object) clearly distinct in your mind. Even when you know the rules,
it's still easy to make a mistake if you're not being extremely careful.

Understanding the proposed implicit class initialization hook only requires
ordinary method inheritance, which isn't quite as daunting a task. The new
hook provides a more gradual path towards understanding all of the phases
involved in the class definition process.


Reduced chance of metaclass conflicts
-------------------------------------

One of the big issues that makes library authors reluctant to use metaclasses
(even when they would be appropriate) is the risk of metaclass conflicts.
These occur whenever two unrelated metaclasses are used by the desired
parents of a class definition. This risk also makes it very difficult to
*add* a metaclass to a class that has previously been published without one.

By contrast, adding an ``__init_subclass__`` method to an existing type poses
a similar level of risk to adding an ``__init__`` method: technically, there
is a risk of breaking poorly implemented subclasses, but when that occurs,
it is recognised as a bug in the subclass rather than the library author
breaching backwards compatibility guarantees.


A path of introduction into Python
==================================

Most of the benefits of this PEP can already be implemented using
a simple metaclass. For the ``__init_subclass__`` hook this works
all the way down to Python 2.7, while the attribute order needs Python 3.0
to work. Such a class has been `uploaded to PyPI`_.

The only drawback of such a metaclass are the mentioned problems with
metaclasses and multiple inheritance. Two classes using such a
metaclass can only be combined, if they use exactly the same such
metaclass. This fact calls for the inclusion of such a class into the
standard library, as ``types.Type``, with a ``types.Object`` base class
using it. Once all users use this standard
library metaclass, classes from different packages can easily be
combined.

But still such classes cannot be easily combined with other classes
using other metaclasses. Authors of metaclasses should bear that in
mind and inherit from the standard metaclass if it seems useful
for users of the metaclass to add more functionality. Ultimately,
if the need for combining with other metaclasses is strong enough,
the proposed functionality may be introduced into Python's ``type``.

Those arguments strongly hint to the following procedure to include
the proposed functionality into Python:

1. The metaclass implementing this proposal is put onto PyPI, so that
   it can be used and scrutinized.
2. Introduce this class into the Python 3.6 standard library.
3. Consider this as the default behavior for Python 3.7.

Steps 2 and 3 would be similar to how the ``set`` datatype was first
introduced as ``sets.Set``, and only later made a builtin type (with a
slightly different API) based on wider experiences with the ``sets``
module.

While the metaclass is still in the standard library and not in the
language, it may still clash with other metaclasses.  The most
prominent metaclass in use is probably ABCMeta.  It is also a
particularly good example for the need of combining metaclasses. For
users who want to define a ABC with subclass initialization, we should
support a ``types.ABCMeta`` class, or let ``abc.ABCMeta`` inherit from this
PEP's metaclass. As it turns out, most of the behavior of ``abc.ABCMeta``
can be done achieved with our ``types.Type``, except its core behavior,
``__instancecheck__`` and ``__subclasscheck__`` which can be supplied,
as per the definition of the Python language, exclusively in a metaclass.

Extensions written in C or C++ also often define their own metaclass.
It would be very useful if those could also inherit from the metaclass
defined here, but this is probably not possible.

New Ways of Using Classes
=========================

This proposal has many usecases like the following. In the examples,
we still inherit from the ``SubclassInit`` base class. This would
become unnecessary once this PEP is included in Python directly.

Subclass registration
---------------------

Especially when writing a plugin system, one likes to register new
subclasses of a plugin baseclass. This can be done as follows::

   class PluginBase(Object):
       subclasses = []

       def __init_subclass__(cls, **kwargs):
           super().__init_subclass__(**kwargs)
           cls.subclasses.append(cls)

In this example, ``PluginBase.subclasses`` will contain a plain list of all
subclasses in the entire inheritance tree.  One should note that this also
works nicely as a mixin class.

Trait descriptors
-----------------

There are many designs of Python descriptors in the wild which, for
example, check boundaries of values. Often those "traits" need some support
of a metaclass to work. This is how this would look like with this
PEP::

   class Trait:
       def __get__(self, instance, owner):
           return instance.__dict__[self.key]

       def __set__(self, instance, value):
           instance.__dict__[self.key] = value

       def __set_owner__(self, owner, name):
           self.key = name


Rejected Design Options
=======================


Calling the hook on the class itself
------------------------------------

Adding an ``__autodecorate__`` hook that would be called on the class
itself was the proposed idea of PEP 422.  Most examples work the same
way or even better if the hook is called on the subclass. In general,
it is much easier to explicitly call the hook on the class in which it
is defined (to opt-in to such a behavior) than to opt-out, meaning
that one does not want the hook to be called on the class it is
defined in.

This becomes most evident if the class in question is designed as a
mixin: it is very unlikely that the code of the mixin is to be
executed for the mixin class itself, as it is not supposed to be a
complete class on its own.

The original proposal also made major changes in the class
initialization process, rendering it impossible to back-port the
proposal to older Python versions.

More importantly, having a pure Python implementation allows us to
take two preliminary steps before before we actually change the
interpreter, giving us the chance to iron out all possible wrinkles
in the API.


Other variants of calling the hook
----------------------------------

Other names for the hook were presented, namely ``__decorate__`` or
``__autodecorate__``. This proposal opts for ``__init_subclass__`` as
it is very close to the ``__init__`` method, just for the subclass,
while it is not very close to decorators, as it does not return the
class.


Requiring an explicit decorator on ``__init_subclass__``
--------------------------------------------------------

One could require the explicit use of ``@classmethod`` on the
``__init_subclass__`` decorator. It was made implicit since there's no
sensible interpretation for leaving it out, and that case would need
to be detected anyway in order to give a useful error message.

This decision was reinforced after noticing that the user experience of
defining ``__prepare__`` and forgetting the ``@classmethod`` method
decorator is singularly incomprehensible (particularly since PEP 3115
documents it as an ordinary method, and the current documentation doesn't
explicitly say anything one way or the other).


Defining arbitrary namespaces
-----------------------------

PEP 422 defined a generic way to add arbitrary namespaces for class
definitions. This approach is much more flexible than just leaving
the definition order in a tuple. The ``__prepare__`` method in a metaclass
supports exactly this behavior. But given that effectively
the only use cases that could be found out in the wild were the
``OrderedDict`` way of determining the attribute order, it seemed
reasonable to only support this special case.

The metaclass described in this PEP has been designed to be very simple
such that it could be reasonably made the default metaclass. This was
especially important when designing the attribute order functionality:
This was a highly demanded feature and has been enabled through the
``__prepare__`` method of metaclasses. This method can be abused in
very weird ways, making it hard to correctly maintain this feature in
CPython. This is why it has been proposed to deprecated this feature,
and instead use ``OrderedDict`` as the standard namespace, supporting
the most important feature while dropping most of the complexity. But
this would have meant that ``OrderedDict`` becomes a language builtin
like dict and set, and not just a standard library class. The choice
of the ``__attribute_order__`` tuple is a much simpler solution to the
problem.

A more ``__new__``-like hook
----------------------------

In PEP 422 the hook worked more like the ``__new__`` method than the
``__init__`` method, meaning that it returned a class instead of
modifying one. This allows a bit more flexibility, but at the cost
of much harder implementation and undesired side effects.

Adding a class attribute with the attribute order
-------------------------------------------------

This got its own PEP 520.


History
=======

This used to be a competing proposal to PEP 422 by Nick Coghlan and Daniel
Urban. PEP 422 intended to achieve the same goals as this PEP, but with a
different way of implementation.  In the meantime, PEP 422 has been withdrawn
favouring this approach.

References
==========

.. _published code:
   http://mail.python.org/pipermail/python-dev/2012-June/119878.html

.. _more than 10 years ago:
   http://mail.python.org/pipermail/python-dev/2001-November/018651.html

.. _Zope's ExtensionClass:
   http://docs.zope.org/zope_secrets/extensionclass.html

.. _uploaded to PyPI:
   https://pypi.python.org/pypi/metaclass

Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


More information about the Python-Dev mailing list