[Python-Dev] PEP487: Simpler customization of class creation

[Nick Coghlan]

On 14 July 2016 at 08:46, Guido van Rossum <guido at python.org> wrote:
> On Wed, Jul 13, 2016 at 7:15 AM, Martin Teichmann <lkb.teichmann at gmail.com> wrote:
>> Another small change should be noted here: in the current implementation of
>> CPython, ``type.__init__`` explicitly forbids the use of keyword arguments,
>> while ``type.__new__`` allows for its attributes to be shipped as keyword
>> arguments. This is weirdly incoherent, and thus the above code forbids that.
>> While it would be possible to retain the current behavior, it would be better
>> if this was fixed, as it is probably not used at all: the only use case would
>> be that at metaclass calls its ``super().__new__`` with *name*, *bases* and
>> *dict* (yes, *dict*, not *namespace* or *ns* as mostly used with modern
>> metaclasses) as keyword arguments. This should not be done.
>>
>> As a second change, the new ``type.__init__`` just ignores keyword
>> arguments. Currently, it insists that no keyword arguments are given. This
>> leads to a (wanted) error if one gives keyword arguments to a class declaration
>> if the metaclass does not process them. Metaclass authors that do want to
>> accept keyword arguments must filter them out by overriding ``__init___``.
>>
>> In the new code, it is not ``__init__`` that complains about keyword arguments,
>> but ``__init_subclass__``, whose default implementation takes no arguments. In
>> a classical inheritance scheme using the method resolution order, each
>> ``__init_subclass__`` may take out it's keyword arguments until none are left,
>> which is checked by the default implementation of ``__init_subclass__``.
>
> I called this out previously, and I am still a bit uncomfortable with
> the backwards incompatibility here. But I believe what you describe
> here is the compromise proposed by Nick, and if that's the case I have
> peace with it.

It would be worth spelling out the end result of the new behaviour in
the PEP to make sure it's what we want. Trying to reason about how
that code works is difficult, but looking at some class definition
scenarios and seeing how they behave with the old semantics and the
new semantics should be relatively straightforward (and they can
become test cases for the revised implementation).

The basic scenario to cover would be defining a metaclass which
*doesn't* accept any additional keyword arguments and seeing how it
fails when passed an unsupported parameter:

    class MyMeta(type):
        pass

    class MyClass(metaclass=MyMeta, otherarg=1):
        pass

    MyMeta("MyClass", (), otherargs=1)

    import types
    types.new_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))
    types.prepare_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))

Current behaviour:

>>> class MyMeta(type):
...     pass
...
>>> class MyClass(metaclass=MyMeta, otherarg=1):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type() takes 1 or 3 arguments
>>> MyMeta("MyClass", (), otherargs=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Required argument 'dict' (pos 3) not found
>>> import types
>>> types.new_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/types.py", line 57, in new_class
    return meta(name, bases, ns, **kwds)
TypeError: type() takes 1 or 3 arguments
>>> types.prepare_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))
(<class '__main__.MyMeta'>, {}, {'otherarg': 1})

The error messages may change, but the cases which currently fail
should continue to fail with TypeError

Further scenarios would then cover the changes needed to the
definition of "MyMeta" to make the class creation invocations above
actually work (since the handling of __prepare__ already tolerates
unknown arguments).

First, just defining __new__ (which currently fails):

>>> class MyMeta(type):
...     def __new__(cls, name, bases, namespace, otherarg):
...         self = super().__new__(cls, name, bases, namespace)
...         self.otherarg = otherarg
...         return self
...
>>> class MyClass(metaclass=MyMeta, otherarg=1):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type.__init__() takes no keyword arguments

Making this work would be fine, and that's what I believe will happen
with the PEP's revised semantics.

Then, just defining __init__ (which also fails):

>>> class MyMeta(type):
...     def __init__(self, name, bases, namespace, otherarg):
...         super().__init__(name, bases, namespace)
...         self.otherarg = otherarg
...
>>> class MyClass(metaclass=MyMeta, otherarg=1):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type() takes 1 or 3 arguments

The PEP shouldn't result in any changes in this case.

And finally defining both of them (which succeeds):

>>> class MyMeta(type):
...     def __new__(cls, name, bases, namespace, otherarg):
...         self = super().__new__(cls, name, bases, namespace)
...         self.otherarg = otherarg
...         return self
...     def __init__(self, name, bases, namespace, otherarg):
...         super().__init__(name, bases, namespace)
...
>>> class MyClass(metaclass=MyMeta, otherarg=1):
...     pass
...
>>> MyClass.otherarg
1

That last scenario is the one we need to ensure keeps working (and I
believe it does with Martin's current implementation)

>From a documentation perspective, one subtlety we should highlight is
that the invocation order during subtype creation is:

* mcl.__new__
  - descr.__set_name__
  - cls.__init_subclass__
* mcl.__init__

So if the metaclass defines both __new__ and __init__ methods, the new
hooks will run before the __init__ method does. (I think that's fine,
the docs just need to make it clear that type.__new__ is the operation
doing the heavy lifting)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia