[Python-Dev] PEP487: Simpler customization of class creation

[Guido van Rossum]

On Wed, Jul 13, 2016 at 7:15 AM, Martin Teichmann
<lkb.teichmann at gmail.com> wrote:
> Hi list,
>
> another round for PEP 487, is there any chance it still makes it into
> Python 3.6?

Sure, feature freeze isn't until September
(https://www.python.org/dev/peps/pep-0494/).

> The PEP should be effectively done, I updated the examples in it,
> given that I implemented the PEP I could actually test the examples,
> so now they work.

I am +1 on the idea of the PEP; below I am just asking for some
clarifications and pointing out a typo or two. Please submit the next
version to the github peps project as a PR! Re-review should be much
quicker then.

> The implementation is at http://bugs.python.org/issue27366, including
> documentation and tests. Unfortunately nobody has reviewed the patch
> yet.

Sorry, I don't have time for that part, but I'm sure once the PEP is
approved the review will follow.

> The new version of the PEP is attached.
>
> Greetings
>
> Martin
>
> PEP: 487
> Title: Simpler customisation of class creation
> Version: $Revision$
> Last-Modified: $Date$
> Author: Martin Teichmann <lkb.teichmann at gmail.com>,
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 27-Feb-2015
> Python-Version: 3.6
> Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016, 2-Jul-2016, 13-Jul-2016
> Replaces: 422
>
>
> Abstract
> ========
>
> Currently, customising class creation requires the use of a custom metaclass.
> This custom metaclass then persists for the entire lifecycle of the class,
> creating the potential for spurious metaclass conflicts.
>
> This PEP proposes to instead support a wide range of customisation
> scenarios through a new ``__init_subclass__`` hook in the class body,
> and a hook to initialize attributes.
>
> The new mechanism should be easier to understand and use than
> implementing a custom metaclass, and thus should provide a gentler
> introduction to the full power of Python's metaclass machinery.
>
>
> Background
> ==========
>
> Metaclasses are a powerful tool to customize class creation. They have,
> however, the problem that there is no automatic way to combine metaclasses.
> If one wants to use two metaclasses for a class, a new metaclass combining
> those two needs to be created, typically manually.
>
> This need often occurs as a surprise to a user: inheriting from two base
> classes coming from two different libraries suddenly raises the necessity
> to manually create a combined metaclass, where typically one is not
> interested in those details about the libraries at all. This becomes
> even worse if one library starts to make use of a metaclass which it
> has not done before. While the library itself continues to work perfectly,
> suddenly every code combining those classes with classes from another library
> fails.
>
> Proposal
> ========
>
> While there are many possible ways to use a metaclass, the vast majority
> of use cases falls into just three categories: some initialization code
> running after class creation, the initalization of descriptors and

initialization

> keeping the order in which class attributes were defined.
>
> The first two categories can easily be achieved by having simple hooks
> into the class creation:
>
> 1. An ``__init_subclass__`` hook that initializes
>    all subclasses of a given class.
> 2. upon class creation, a ``__set_owner__`` hook is called on all the
>    attribute (descriptors) defined in the class, and
>
> The third category is the topic of another PEP 520.

PEP, PEP 520.

> As an example, the first use case looks as follows::
>
>    >>> class QuestBase:
>    ...    # this is implicitly a @classmethod

maybe add "(see below for motivation)" ?

>    ...    def __init_subclass__(cls, swallow, **kwargs):
>    ...        cls.swallow = swallow
>    ...        super().__init_subclass__(**kwargs)
>
>    >>> class Quest(QuestBase, swallow="african"):
>    ...    pass
>
>    >>> Quest.swallow
>    'african'
>
> The base class ``object`` contains an empty ``__init_subclass__``
> method which serves as an endpoint for cooperative multiple inheritance.
> Note that this method has no keyword arguments, meaning that all
> methods which are more specialized have to process all keyword
> arguments.
>
> This general proposal is not a new idea (it was first suggested for
> inclusion in the language definition `more than 10 years ago`_, and a
> similar mechanism has long been supported by `Zope's ExtensionClass`_),
> but the situation has changed sufficiently in recent years that
> the idea is worth reconsidering for inclusion.
>
> The second part of the proposal adds an ``__set_owner__``
> initializer for class attributes, especially if they are descriptors.
> Descriptors are defined in the body of a
> class, but they do not know anything about that class, they do not
> even know the name they are accessed with. They do get to know their
> owner once ``__get__`` is called, but still they do not know their
> name. This is unfortunate, for example they cannot put their
> associated value into their object's ``__dict__`` under their name,
> since they do not know that name.  This problem has been solved many
> times, and is one of the most important reasons to have a metaclass in
> a library. While it would be easy to implement such a mechanism using
> the first part of the proposal, it makes sense to have one solution
> for this problem for everyone.
>
> To give an example of its usage, imagine a descriptor representing weak
> referenced values::
>
>     import weakref
>
>     class WeakAttribute:
>         def __get__(self, instance, owner):
>             return instance.__dict__[self.name]()
>
>         def __set__(self, instance, value):
>             instance.__dict__[self.name] = weakref.ref(value)
>
>         # this is the new initializer:
>         def __set_owner__(self, owner, name):
>             self.name = name

This example is missing something -- an example of how the
WeakAttribute class would be *used*. I suppose something like

# We wish we could write
class C:
    foo = WeakAttribute()
x = C()
x.foo = ...
print(x.foo)

> While this example looks very trivial, it should be noted that until
> now such an attribute cannot be defined without the use of a metaclass.
> And given that such a metaclass can make life very hard, this kind of
> attribute does not exist yet.
>
> Initializing descriptors could simply be done in the
> ``__init_subclass__`` hook. But this would mean that descriptors can
> only be used in classes that have the proper hook, the generic version
> like in the example would not work generally. One could also call
> ``__set_owner__`` from whithin the base implementation of
> ``object.__init_subclass__``. But given that it is a common mistake
> to forget to call ``super()``, it would happen too often that suddenly
> descriptors are not initialized.
>
>
> Key Benefits
> ============
>
>
> Easier inheritance of definition time behaviour
> -----------------------------------------------
>
> Understanding Python's metaclasses requires a deep understanding of
> the type system and the class construction process. This is legitimately
> seen as challenging, due to the need to keep multiple moving parts (the code,
> the metaclass hint, the actual metaclass, the class object, instances of the
> class object) clearly distinct in your mind. Even when you know the rules,
> it's still easy to make a mistake if you're not being extremely careful.
>
> Understanding the proposed implicit class initialization hook only requires
> ordinary method inheritance, which isn't quite as daunting a task. The new
> hook provides a more gradual path towards understanding all of the phases
> involved in the class definition process.
>
>
> Reduced chance of metaclass conflicts
> -------------------------------------
>
> One of the big issues that makes library authors reluctant to use metaclasses
> (even when they would be appropriate) is the risk of metaclass conflicts.
> These occur whenever two unrelated metaclasses are used by the desired
> parents of a class definition. This risk also makes it very difficult to
> *add* a metaclass to a class that has previously been published without one.
>
> By contrast, adding an ``__init_subclass__`` method to an existing type poses
> a similar level of risk to adding an ``__init__`` method: technically, there
> is a risk of breaking poorly implemented subclasses, but when that occurs,
> it is recognised as a bug in the subclass rather than the library author
> breaching backwards compatibility guarantees.
>
>
> New Ways of Using Classes
> =========================
>
> Subclass registration
> ---------------------
>
> Especially when writing a plugin system, one likes to register new
> subclasses of a plugin baseclass. This can be done as follows::
>
>    class PluginBase(Object):

What is "Object"? I presume just a typo for "object"?

>        subclasses = []
>
>        def __init_subclass__(cls, **kwargs):
>            super().__init_subclass__(**kwargs)
>            cls.subclasses.append(cls)
>
> In this example, ``PluginBase.subclasses`` will contain a plain list of all
> subclasses in the entire inheritance tree.  One should note that this also
> works nicely as a mixin class.
>
> Trait descriptors
> -----------------
>
> There are many designs of Python descriptors in the wild which, for
> example, check boundaries of values. Often those "traits" need some support
> of a metaclass to work. This is how this would look like with this
> PEP::
>
>    class Trait:
>        def __init__(self, minimum, maximum):
>            self.minimum = minimum
>            self.maximum = maximum
>
>        def __get__(self, instance, owner):
>            return instance.__dict__[self.key]
>
>        def __set__(self, instance, value):
>            if self.minimum < value < self.maximum:
>                instance.__dict__[self.key] = value
>            else:
>                raise ValueError("value not in range")
>
>        def __set_owner__(self, owner, name):

I wonder if this should be renamed to __set_name__ or something else
that clarifies we're passing it the name of the attribute? The method
name __set_owner__ made me assume this is about the owning object
(which is often a useful term in other discussions about objects),
whereas it is really about telling the descriptor the name of the
attribute for which it applies.

>            self.key = name
>
> Implementation Details
> ======================
>
> For those who prefer reading Python over english, the following is a Python
> equivalent of the C API changes proposed in this PEP, where the new ``object``
> and ``type`` defined here inherit from the usual ones::

That (inheriting type from type, and object from object) is very
confusing. Why not just define new classes e.g. NewType and NewObject
here, since it's just pseudo code anyway?

>     import types
>
>     class type(type):
>         def __new__(cls, *args, **kwargs):
>             if len(args) == 1:
>                 return super().__new__(cls, args[0])
>             name, bases, ns = args
>             init = ns.get('__init_subclass__')
>             if isinstance(init, types.FunctionType):
>                 ns['__init_subclass__'] = classmethod(init)
>             self = super().__new__(cls, name, bases, ns)
>             for k, v in self.__dict__.items():
>                 func = getattr(v, '__set_owner__', None)
>                 if func is not None:
>                     func(self, k)
>             super(self, self).__init_subclass__(**kwargs)
>             return self
>
>         def __init__(self, name, bases, ns, **kwargs):
>             super().__init__(name, bases, ns)

What does this definition of __init__ add?

>     class object:
>         @classmethod
>         def __init_subclass__(cls):
>             pass
>
>     class object(object, metaclass=type):
>         pass

Eek! Too many things named object.

> In this code, first the ``__set_owner__`` are called on the descriptors, and
> then the ``__init_subclass__``. This means that subclass initializers already
> see the fully initialized descriptors. This way, ``__init_subclass__`` users
> can fix all descriptors again if this is needed.
>
> Another option would have been to call ``__set_owner__`` in the base
> implementation of ``object.__init_subclass__``. This way it would be possible
> event to prevent ``__set_owner__`` from being called. Most of the times,

event to prevent???

> however, such a prevention would be accidental, as it often happens that a call
> to ``super()`` is forgotten.
>
> Another small change should be noted here: in the current implementation of
> CPython, ``type.__init__`` explicitly forbids the use of keyword arguments,
> while ``type.__new__`` allows for its attributes to be shipped as keyword
> arguments. This is weirdly incoherent, and thus the above code forbids that.
> While it would be possible to retain the current behavior, it would be better
> if this was fixed, as it is probably not used at all: the only use case would
> be that at metaclass calls its ``super().__new__`` with *name*, *bases* and
> *dict* (yes, *dict*, not *namespace* or *ns* as mostly used with modern
> metaclasses) as keyword arguments. This should not be done.
>
> As a second change, the new ``type.__init__`` just ignores keyword
> arguments. Currently, it insists that no keyword arguments are given. This
> leads to a (wanted) error if one gives keyword arguments to a class declaration
> if the metaclass does not process them. Metaclass authors that do want to
> accept keyword arguments must filter them out by overriding ``__init___``.
>
> In the new code, it is not ``__init__`` that complains about keyword arguments,
> but ``__init_subclass__``, whose default implementation takes no arguments. In
> a classical inheritance scheme using the method resolution order, each
> ``__init_subclass__`` may take out it's keyword arguments until none are left,
> which is checked by the default implementation of ``__init_subclass__``.

I called this out previously, and I am still a bit uncomfortable with
the backwards incompatibility here. But I believe what you describe
here is the compromise proposed by Nick, and if that's the case I have
peace with it.

> Rejected Design Options
> =======================
>
>
> Calling the hook on the class itself
> ------------------------------------
>
> Adding an ``__autodecorate__`` hook that would be called on the class
> itself was the proposed idea of PEP 422.  Most examples work the same
> way or even better if the hook is called on the subclass. In general,
> it is much easier to explicitly call the hook on the class in which it
> is defined (to opt-in to such a behavior) than to opt-out, meaning
> that one does not want the hook to be called on the class it is
> defined in.
>
> This becomes most evident if the class in question is designed as a
> mixin: it is very unlikely that the code of the mixin is to be
> executed for the mixin class itself, as it is not supposed to be a
> complete class on its own.
>
> The original proposal also made major changes in the class
> initialization process, rendering it impossible to back-port the
> proposal to older Python versions.
>
> More importantly, having a pure Python implementation allows us to
> take two preliminary steps before before we actually change the
> interpreter, giving us the chance to iron out all possible wrinkles
> in the API.
>
>
> Other variants of calling the hook
> ----------------------------------
>
> Other names for the hook were presented, namely ``__decorate__`` or
> ``__autodecorate__``. This proposal opts for ``__init_subclass__`` as
> it is very close to the ``__init__`` method, just for the subclass,
> while it is not very close to decorators, as it does not return the
> class.
>
>
> Requiring an explicit decorator on ``__init_subclass__``
> --------------------------------------------------------
>
> One could require the explicit use of ``@classmethod`` on the
> ``__init_subclass__`` decorator. It was made implicit since there's no
> sensible interpretation for leaving it out, and that case would need
> to be detected anyway in order to give a useful error message.
>
> This decision was reinforced after noticing that the user experience of
> defining ``__prepare__`` and forgetting the ``@classmethod`` method
> decorator is singularly incomprehensible (particularly since PEP 3115
> documents it as an ordinary method, and the current documentation doesn't
> explicitly say anything one way or the other).
>
> A more ``__new__``-like hook
> ----------------------------
>
> In PEP 422 the hook worked more like the ``__new__`` method than the
> ``__init__`` method, meaning that it returned a class instead of
> modifying one. This allows a bit more flexibility, but at the cost
> of much harder implementation and undesired side effects.
>
> Adding a class attribute with the attribute order
> -------------------------------------------------
>
> This got its own PEP 520.
>
>
> History
> =======
>
> This used to be a competing proposal to PEP 422 by Nick Coghlan and Daniel
> Urban. PEP 422 intended to achieve the same goals as this PEP, but with a
> different way of implementation.  In the meantime, PEP 422 has been withdrawn
> favouring this approach.
>
> References
> ==========
>
> .. _more than 10 years ago:
>    http://mail.python.org/pipermail/python-dev/2001-November/018651.html
>
> .. _Zope's ExtensionClass:
>    http://docs.zope.org/zope_secrets/extensionclass.html
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
>
> ..
>    Local Variables:
>    mode: indented-text
>    indent-tabs-mode: nil
>    sentence-end-double-space: t
>    fill-column: 70
>    coding: utf-8
>    End:
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)