Mailman 3 __init_subclass__ and metaclasses - Python-Dev

__init_subclass__ and metaclasses

Ethan Furman

24 Dec 2020 24 Dec '20

8:24 p.m.

PEP 487 introduced __init_subclass__ and __set_name__, and both of those were wins for the common cases of metaclass usage.

Unfortunately, the implementation of PEP 487 with regards to __init_subclass__ has made the writing of correct metaclasses significantly harder, if not impossible.

The cause is that when a metaclass calls type.__new__ to actually create the class, type.__new__ calls the __init_subclass__ methods of the new class' parents, passing it the newly created, but incomplete, class. In code:

``` class Meta(type): # def __new__(mcls, name, bases, namespace, **kwds): # create new class, which will call __init_subclass__ and __set_name__ new_class = type.__new__(mcls, name, bases, namespace, **kwds) # finish setting up class new_class.some_attr = 9 ```

As you can deduce, when the parent __init_subclass__ is called with the new class, `some_attr` has not been added yet -- the new class is incomplete.

For Enum, this means that __init_subclass__ doesn't have access to the new Enum's members (they haven't beet added yet).

For ABC, this means that __init_subclass__ doesn't have access to __abstract_methods__ (it hasn't been created yet).

Because Enum is pure Python code I was able to work around it: - remove new __init_subclass__ (if it exists) - insert dummy class with a no-op __init_subclass__ - call type.__new__ - save any actual __init_subclass__ - add back any new __init_subclass__ - rewrite the new class' __bases__, removing the no-op dummy class - finish creating the class - call the parent __init_subclass__ with the now complete Enum class

I have not been able to work around the problem for ABC.

Two possible solutions I can think of:

- pass a keyword argument to type.__new__ that suppresses the call to __init_subclass__; and - provide a way to invoke new class' parent's __init_subclass__ before returning it

- instead of type.__new__ doing that work, have type.__init__ do it.

Thoughts?

-- ~Ethan~

Show replies by date

Joao S. O. Bueno

24 Dec 24 Dec

9:40 p.m.

Actually, there are a few steps that `type.__new__` perform that are not customizable in metaclasses.

I had sometimes thought about mailing this here, or Python ideas, but could not come up with a "real world" use case where the customization of those would be meaningful.

Let'me see if I recall all cases - two of them are the calls to `__init_subclass__` and the descriptors `__set_name__` as you put it, I think there is a third behavior that can't be separated from `type.__new__` - but I can't remember it now

Anyway, the "thing to do" that always occurred to me about it is to add "soft" method slots to `type` itself - so that `type.__new__` would call those on the corresponding initialization phases.

Since these are to be run only when classes are created, their impact should be negligible.

In other words, have `type` implement methods like `__run_init_subclass__`, `__run_descriptor_setname__`, (and one for the other task I can't remember now). So, all metaclass code written up to today remains valid, and these behaviors become properly customizable.

Adding keyword parameters to `type.__new__`, IMHO, besides a little bit fishy as we are talking of arguments to change the behavior of the method, would themselves compete and have to be filtered out, or otherwise special-cased in the `__init_subclass__` method itself. I mean - let's suppose we add `__suppress_init_subclass__` as an named parameter to `type.__new__` - what would happen with this argument in `__init_subclass__` ? Would it show up in the kwargs? Otherwise it would be the _only_ kwarg popped out and not passed to __init_subclass__, being an inconvenient exception.

Having an overridable, separate, method in type to run __init_subclass__ and __set_name__ bypass these downsides.

In time, Happy holidays everyone!

js -><-

On Fri, 25 Dec 2020 at 00:38, Ethan Furman ethan@stoneleaf.us wrote:

...

PEP 487 introduced __init_subclass__ and __set_name__, and both of those were wins for the common cases of metaclass usage.

Unfortunately, the implementation of PEP 487 with regards to __init_subclass__ has made the writing of correct metaclasses significantly harder, if not impossible.

The cause is that when a metaclass calls type.__new__ to actually create the class, type.__new__ calls the __init_subclass__ methods of the new class' parents, passing it the newly created, but incomplete, class. In code:
class Meta(type):
     #
     def __new__(mcls, name, bases, namespace, **kwds):
         # create new class, which will call __init_subclass__ and
__set_name__
         new_class = type.__new__(mcls, name, bases, namespace, **kwds)
         # finish setting up class
         new_class.some_attr = 9
As you can deduce, when the parent __init_subclass__ is called with the new class, `some_attr` has not been added yet -- the new class is incomplete.

For Enum, this means that __init_subclass__ doesn't have access to the new Enum's members (they haven't beet added yet).

For ABC, this means that __init_subclass__ doesn't have access to __abstract_methods__ (it hasn't been created yet).

Because Enum is pure Python code I was able to work around it:

remove new __init_subclass__ (if it exists)

insert dummy class with a no-op __init_subclass__

call type.__new__

save any actual __init_subclass__

add back any new __init_subclass__

rewrite the new class' __bases__, removing the no-op dummy class

finish creating the class

call the parent __init_subclass__ with the now complete Enum class

I have not been able to work around the problem for ABC.

Two possible solutions I can think of:

pass a keyword argument to type.__new__ that suppresses the call to

__init_subclass__; and

provide a way to invoke new class' parent's __init_subclass__ before

returning it

or

instead of type.__new__ doing that work, have type.__init__ do it.

Thoughts?

-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFS... Code of Conduct: http://python.org/psf/codeofconduct/

Joao S. O. Bueno

28 Dec 28 Dec

7:41 p.m.

For the record - the 3rd process that is currently un-customizable when creating a class, i.e. things that happen in an opaque way inside `type.__new__`, is the ABC class machinery. I could not recall it when writing the previous e-mail.

Still - I think this might be very little disruptive, and yet provide metaclasses back with full customization power, including being able to address the problem brought up by Etham.

js -><-

On Fri, 25 Dec 2020 at 01:40, Joao S. O. Bueno jsbueno@python.org.br wrote:

...

Actually, there are a few steps that `type.__new__` perform that are not customizable in metaclasses.

I had sometimes thought about mailing this here, or Python ideas, but could not come up with a "real world" use case where the customization of those would be meaningful.

Let'me see if I recall all cases - two of them are the calls to `__init_subclass__` and the descriptors `__set_name__` as you put it, I think there is a third behavior that can't be separated from `type.__new__` - but I can't remember it now

Anyway, the "thing to do" that always occurred to me about it is to add "soft" method slots to `type` itself - so that `type.__new__` would call those on the corresponding initialization phases.

Since these are to be run only when classes are created, their impact should be negligible.

In other words, have `type` implement methods like `__run_init_subclass__`, `__run_descriptor_setname__`, (and one for the other task I can't remember now). So, all metaclass code written up to today remains valid, and these behaviors become properly customizable.

Adding keyword parameters to `type.__new__`, IMHO, besides a little bit fishy as we are talking of arguments to change the behavior of the method, would themselves compete and have to be filtered out, or otherwise special-cased in the `__init_subclass__` method itself. I mean - let's suppose we add `__suppress_init_subclass__` as an named parameter to `type.__new__` - what would happen with this argument in `__init_subclass__` ? Would it show up in the kwargs? Otherwise it would be the _only_ kwarg popped out and not passed to __init_subclass__, being an inconvenient exception.

Having an overridable, separate, method in type to run __init_subclass__ and __set_name__ bypass these downsides.

In time, Happy holidays everyone!

js -><-

On Fri, 25 Dec 2020 at 00:38, Ethan Furman ethan@stoneleaf.us wrote:

...
PEP 487 introduced __init_subclass__ and __set_name__, and both of those were wins for the common cases of metaclass usage.

Unfortunately, the implementation of PEP 487 with regards to __init_subclass__ has made the writing of correct metaclasses significantly harder, if not impossible.

The cause is that when a metaclass calls type.__new__ to actually create the class, type.__new__ calls the __init_subclass__ methods of the new class' parents, passing it the newly created, but incomplete, class. In code:
class Meta(type):
     #
     def __new__(mcls, name, bases, namespace, **kwds):
         # create new class, which will call __init_subclass__ and
__set_name__
         new_class = type.__new__(mcls, name, bases, namespace, **kwds)
         # finish setting up class
         new_class.some_attr = 9
As you can deduce, when the parent __init_subclass__ is called with the new class, `some_attr` has not been added yet -- the new class is incomplete.

For Enum, this means that __init_subclass__ doesn't have access to the new Enum's members (they haven't beet added yet).

For ABC, this means that __init_subclass__ doesn't have access to __abstract_methods__ (it hasn't been created yet).

Because Enum is pure Python code I was able to work around it:

remove new __init_subclass__ (if it exists)

insert dummy class with a no-op __init_subclass__

call type.__new__

save any actual __init_subclass__

add back any new __init_subclass__

rewrite the new class' __bases__, removing the no-op dummy class

finish creating the class

call the parent __init_subclass__ with the now complete Enum class

I have not been able to work around the problem for ABC.

Two possible solutions I can think of:

pass a keyword argument to type.__new__ that suppresses the call to

__init_subclass__; and

provide a way to invoke new class' parent's __init_subclass__ before

returning it

or

instead of type.__new__ doing that work, have type.__init__ do it.

Thoughts?

-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFS... Code of Conduct: http://python.org/psf/codeofconduct/

Guido van Rossum

10:31 p.m.

Let me see if I can unpack this.

I observe that `type.__new__() ` is really the C function `type_new()` in typeobject.c, and hence I will refer to it by that name.

I understand that `type_new()` is the only way to create type objects, and it includes a call to `__init_subclass__()`.

In the source code of `type_new()`, calling `__init_subclass__()` is the last thing it does before returning the newly created class object, so at this point the class object is complete, *except* that any updates made by the caller of `type_new()` after `type_new()` returns have not been made, of course. (For example, `new_class.some_attr = 9` from Ethan's post, or `__abstractmethods__`, which is set by update_abstractmethods() in abc.py.)

Now here's something that Ethan said that I don't follow:

...

For Enum, this means that `__init_subclass__` doesn't have access to the

new Enum's members (they haven't been added yet)

I would presume that in an example like the following, the members *are* set by the time `type_new()` is called. What am I missing? ``` class Color(enum.Enum): RED = 1 GREEN = 2 BLUE = 4 ``` Maybe the problem is that the members are still set to their "naive" initial values (1, 2, 4) rather than to the corresponding enum values (e.g. `<Color.Red: 1>`)?

Without a more elaborate use case I can't give that more than a shrug. This is how the `__init_subclass__()` protocol is designed. If it's not to your liking, you can recommend to your users that they use something else.

Nore that for ABC, if you add or change the abstraction status of some class attributes, you can just call `update_abstractmethods()` and it will update `__abstractmethods__` based on the new contents of the class. This also sounds like no biggie to me.

On Mon, Dec 28, 2020 at 6:47 PM Joao S. O. Bueno jsbueno@python.org.br wrote:

...

For the record - the 3rd process that is currently un-customizable when creating a class, i.e. things that happen in an opaque way inside `type.__new__`, is the ABC class machinery. I could not recall it when writing the previous e-mail.

Still - I think this might be very little disruptive, and yet provide metaclasses back with full customization power, including being able to address the problem brought up by Etham.

js -><-

On Fri, 25 Dec 2020 at 01:40, Joao S. O. Bueno jsbueno@python.org.br wrote:

...
Actually, there are a few steps that `type.__new__` perform that are not customizable in metaclasses.

I had sometimes thought about mailing this here, or Python ideas, but could not come up with a "real world" use case where the customization of those would be meaningful.

Let'me see if I recall all cases - two of them are the calls to `__init_subclass__` and the descriptors `__set_name__` as you put it, I think there is a third behavior that can't be separated from `type.__new__` - but I can't remember it now

Anyway, the "thing to do" that always occurred to me about it is to add "soft" method slots to `type` itself - so that `type.__new__` would call those on the corresponding initialization phases.

Since these are to be run only when classes are created, their impact should be negligible.

In other words, have `type` implement methods like `__run_init_subclass__`, `__run_descriptor_setname__`, (and one for the other task I can't remember now). So, all metaclass code written up to today remains valid, and these behaviors become properly customizable.

Adding keyword parameters to `type.__new__`, IMHO, besides a little bit fishy as we are talking of arguments to change the behavior of the method, would themselves compete and have to be filtered out, or otherwise special-cased in the `__init_subclass__` method itself. I mean - let's suppose we add `__suppress_init_subclass__` as an named parameter to `type.__new__` - what would happen with this argument in `__init_subclass__` ? Would it show up in the kwargs? Otherwise it would be the _only_ kwarg popped out and not passed to __init_subclass__, being an inconvenient exception.

Having an overridable, separate, method in type to run __init_subclass__ and __set_name__ bypass these downsides.

In time, Happy holidays everyone!

js -><-

On Fri, 25 Dec 2020 at 00:38, Ethan Furman ethan@stoneleaf.us wrote:

...
PEP 487 introduced __init_subclass__ and __set_name__, and both of those were wins for the common cases of metaclass usage.

Unfortunately, the implementation of PEP 487 with regards to __init_subclass__ has made the writing of correct metaclasses significantly harder, if not impossible.

The cause is that when a metaclass calls type.__new__ to actually create the class, type.__new__ calls the __init_subclass__ methods of the new class' parents, passing it the newly created, but incomplete, class. In code:
class Meta(type):
     #
     def __new__(mcls, name, bases, namespace, **kwds):
         # create new class, which will call __init_subclass__ and
__set_name__
         new_class = type.__new__(mcls, name, bases, namespace, **kwds)
         # finish setting up class
         new_class.some_attr = 9
As you can deduce, when the parent __init_subclass__ is called with the new class, `some_attr` has not been added yet -- the new class is incomplete.

For Enum, this means that __init_subclass__ doesn't have access to the new Enum's members (they haven't beet added yet).

For ABC, this means that __init_subclass__ doesn't have access to __abstract_methods__ (it hasn't been created yet).

Because Enum is pure Python code I was able to work around it:

remove new __init_subclass__ (if it exists)

insert dummy class with a no-op __init_subclass__

call type.__new__

save any actual __init_subclass__

add back any new __init_subclass__

rewrite the new class' __bases__, removing the no-op dummy class

finish creating the class

call the parent __init_subclass__ with the now complete Enum class

I have not been able to work around the problem for ABC.

Two possible solutions I can think of:

pass a keyword argument to type.__new__ that suppresses the call to

__init_subclass__; and

provide a way to invoke new class' parent's __init_subclass__ before

returning it

or

instead of type.__new__ doing that work, have type.__init__ do it.

Thoughts?

-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZMRRNSFS... Code of Conduct: http://python.org/psf/codeofconduct/
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NUOMIKVC... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/

Ethan Furman

11:18 p.m.

On 12/28/20 9:31 PM, Guido van Rossum wrote:

...

Let me see if I can unpack this.

I observe that `type.__new__() ` is really the C function `type_new()` in typeobject.c, and hence I will refer to it by that name.

I understand that `type_new()` is the only way to create type objects, and it includes a call to `__init_subclass__()`.

In the source code of `type_new()`, calling `__init_subclass__()` is the last thing it does before returning the newly created class object, so at this point the class object is complete, *except* that any updates made by the caller of `type_new()` after `type_new()` returns have not been made, of course. (For example, `new_class.some_attr = 9` from Ethan's post, or `__abstractmethods__`, which is set by update_abstractmethods() in abc.py.)

This is really the heart of the issue. A major reason to write a custom metaclass is to be able to modify the returned class before giving it back to the user -- so even though `type_new` is complete the new class could easily not be complete, and calling `__init_subclass__` and `__set_name__` from `type_new` is premature.

...

Now here's something that Ethan said that I don't follow:

...
For Enum, this means that `__init_subclass__` doesn't have access to the new Enum's members (they haven't been added yet)

I would presume that in an example like the following, the members *are* set by the time `type_new()` is called. What am I missing?
class Color(enum.Enum):
     RED = 1
     GREEN = 2
     BLUE = 4
Maybe the problem is that the members are still set to their "naive" initial values (1, 2, 4) rather than to the corresponding enum values (e.g. `<Color.Red: 1>`)?

Before `type_new()` is called all the (future) members are removed from `namespace`, so they are not present when `__init_subclass__` is called. Even if they were left in as `{'RED': 1, `GREEN`: 2, `BLUE`: 4}` it would not be possible for an `__init_subclass__` to customize the members further, or record them in custom data structures, or run validation code on them, or etc.

...

Without a more elaborate use case I can't give that more than a shrug. This is how the `__init_subclass__()` protocol is designed.

The `__init_subclass__` and `__set_name__` protocols are intended to be run before a new type is finished, but creating a new type has three major steps:

- `__prepare__` to get the namespace - `__new__` to get the memory and data structures - `__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.

...

Note that for ABC, if you add or change the abstraction status of some class attributes, you can just call `update_abstractmethods()` and it will update `__abstractmethods__` based on the new contents of the class. This also sounds like no biggie to me.

The issue tracker sample code that fails (from #35815):

``` import abc

class Base(abc.ABC): # def __init_subclass__(cls, **kwargs): instance = cls() print(f"Created instance of {cls} easily: {instance}") # @abc.abstractmethod def do_something(self): pass

class Derived(Base): pass ```

And the output:

`Created instance of <class '__main__.Derived'> easily: <__main__.Derived object at 0x10a6dd6a0>`

If `Base` had been completed before the call to `__init_subclass__`, then `Derived` would have raised an error -- and it does raise an error with the patch I have submitted on Github.

-- ~Ethan~

Guido van Rossum

29 Dec 29 Dec

9:59 a.m.

On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman ethan@stoneleaf.us wrote:

...

On 12/28/20 9:31 PM, Guido van Rossum wrote:

...
Let me see if I can unpack this.

I observe that `type.__new__() ` is really the C function `type_new()`

in typeobject.c, and hence I will refer to it by

...
that name.

I understand that `type_new()` is the only way to create type objects,

and it includes a call to `__init_subclass__()`.

...
In the source code of `type_new()`, calling `__init_subclass__()` is the

last thing it does before returning the newly

...
created class object, so at this point the class object is complete,

*except* that any updates made by the caller of

...
`type_new()` after `type_new()` returns have not been made, of course.

(For example, `new_class.some_attr = 9` from

...
Ethan's post, or `__abstractmethods__`, which is set by

update_abstractmethods() in abc.py.)

This is really the heart of the issue. A major reason to write a custom metaclass is to be able to modify the returned class before giving it back to the user -- so even though `type_new` is complete the new class could easily not be complete, and calling `__init_subclass__` and `__set_name__` from `type_new` is premature.

...
Now here's something that Ethan said that I don't follow:

...
For Enum, this means that `__init_subclass__` doesn't have access to

the new Enum's members (they haven't been added yet)

...
I would presume that in an example like the following, the members *are*

set by the time `type_new()` is called. What am

...
I missing?
class Color(enum.Enum):
     RED = 1
     GREEN = 2
     BLUE = 4
Maybe the problem is that the members are still set to their "naive"
initial values (1, 2, 4) rather than to the

...
corresponding enum values (e.g. `<Color.Red: 1>`)?

Before `type_new()` is called all the (future) members are removed from `namespace`, so they are not present when `__init_subclass__` is called. Even if they were left in as `{'RED': 1, `GREEN`: 2, `BLUE`: 4}` it would not be possible for an `__init_subclass__` to customize the members further, or record them in custom data structures, or run validation code on them, or etc.

...
Without a more elaborate use case I can't give that more than a shrug.

This is how the `__init_subclass__()` protocol is

...
designed.

The `__init_subclass__` and `__set_name__` protocols are intended to be run before a new type is finished, but creating a new type has three major steps:

`__prepare__` to get the namespace

`__new__` to get the memory and data structures

`__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.

No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__` starts, and you propose to move the subclass after that window. There may be code that depends on the class being initialized at that point, and you will break that code.

Honestly I disapprove of the shenanigans you're committing in enum.py, and if those are in in the 3.10 (master) branch I recommend that you take them out. It just looks too fragile and obscure.

...

...
Note that for ABC, if you add or change the abstraction status of some

class attributes, you can just call

...
`update_abstractmethods()` and it will update `__abstractmethods__`

based on the new contents of the class. This also

...
sounds like no biggie to me.

The issue tracker sample code that fails (from #35815):
import abc

class Base(abc.ABC):
     #
     def __init_subclass__(cls, **kwargs):
         instance = cls()
         print(f"Created instance of {cls} easily: {instance}")
     #
     @abc.abstractmethod
     def do_something(self):
         pass

class Derived(Base):
     pass
And the output:

`Created instance of <class '__main__.Derived'> easily: <__main__.Derived object at 0x10a6dd6a0>`

If `Base` had been completed before the call to `__init_subclass__`, then `Derived` would have raised an error -- and it does raise an error with the patch I have submitted on Github.

Again, big shrug. A static checker will catch that.

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/

Ethan Furman

11:37 a.m.

On 12/29/20 8:59 AM, Guido van Rossum wrote:

...

On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:

...

...
The `__init_subclass__` and `__set_name__` protocols are intended to be run before a new type is finished, but creating a new type has three major steps:

`__prepare__` to get the namespace

`__new__` to get the memory and data structures

`__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.

No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__` starts, and you propose to move the subclass after that window. There may be code that depends on the class being initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to `__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully participate in the `__set_name__` and `__init_subclass__` protocols.

Just in the stdlib we have two custom metaclasses that would start working correctly with this change (for the `__init_subclass__` case).

...

Honestly I disapprove of the shenanigans you're committing in enum.py, and if those are in in the 3.10 (master) branch I recommend that you take them out. It just looks too fragile and obscure.

Trust me, I don't like them either. But I like even less that a custom `__init_subclass__` for an Enum would fail if it tried to do anything with the members. That's one of the reasons why I would like to see this fix put in (the shenanigans would be unnecessary then).

From the point of view of a metaclass author, the current behavior feels buggy. In theory, `__init_subclass__` and `__set_name__` are supposed to be called after a class is created, and yet they are being called somewhere in the middle of my metaclass' `__new__`.

Looked at another way, `__init_subclass__` should be receiving a `cls` that is ready for use, i.e. done and fully constructed and complete. But if that class is being created by a custom metaclass, then the thing that is being given to `__init_subclass__` could easily be only partial.

-- ~Ethan~

Guido van Rossum

11 p.m.

I need to think about this more.

Technically the class *is* created at the point `__init_subclass__` is called. What the metaclass does after that point is embellishment.

Most metaclasses will have sufficient control for their needs because they can manipulate the contents of the namespace that's passed to type_new() -- I understand that doesn't work for Enum because the instances have a reference to the class in them.

I suspect the classes that you claim are "buggy" are fine in practice, even if you can construct a counterexample.

All in all I just worry about the backward compatibility here: one could easily imagine a metaclass whose `__new__` relies on the effect of the `__init_subclass__` called by type_new(), and moving the call would break that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such a case, the status quo should win.

--Guido

On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman ethan@stoneleaf.us wrote:

...

On 12/29/20 8:59 AM, Guido van Rossum wrote:

...
On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:

...
...
The `__init_subclass__` and `__set_name__` protocols are intended to be

run before a new type is finished, but creating

...
...
a new type has three major steps:

`__prepare__` to get the namespace

`__new__` to get the memory and data structures

`__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.

No, we can't. There is a window where the subclass is initialized after

`typing_new()` returned before `__init__`

...
starts, and you propose to move the subclass after that window. There

may be code that depends on the class being

...
initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to `__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully participate in the `__set_name__` and `__init_subclass__` protocols.

Just in the stdlib we have two custom metaclasses that would start working correctly with this change (for the `__init_subclass__` case).

...
Honestly I disapprove of the shenanigans you're committing in enum.py,

and if those are in in the 3.10 (master) branch I

...
recommend that you take them out. It just looks too fragile and obscure.

Trust me, I don't like them either. But I like even less that a custom `__init_subclass__` for an Enum would fail if it tried to do anything with the members. That's one of the reasons why I would like to see this fix put in (the shenanigans would be unnecessary then).

From the point of view of a metaclass author, the current behavior feels buggy. In theory, `__init_subclass__` and `__set_name__` are supposed to be called after a class is created, and yet they are being called somewhere in the middle of my metaclass' `__new__`.

Looked at another way, `__init_subclass__` should be receiving a `cls` that is ready for use, i.e. done and fully constructed and complete. But if that class is being created by a custom metaclass, then the thing that is being given to `__init_subclass__` could easily be only partial.

-- ~Ethan~ _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL26... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/

Glenn Linderman

11:32 p.m.

On 12/29/2020 10:00 PM, Guido van Rossum wrote:

...

I need to think about this more.

Both techniques could coexist.

...

Technically the class *is* created at the point `__init_subclass__` is called. What the metaclass does after that point is embellishment.

Most metaclasses will have sufficient control for their needs because they can manipulate the contents of the namespace that's passed to type_new() -- I understand that doesn't work for Enum because the instances have a reference to the class in them.

I suspect the classes that you claim are "buggy" are fine in practice, even if you can construct a counterexample.

All in all I just worry about the backward compatibility here: one could easily imagine a metaclass whose `__new__` relies on the effect of the `__init_subclass__` called by type_new(), and moving the call would break that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such a case, the status quo should win.

--Guido

On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman <ethan@stoneleaf.us mailto:ethan@stoneleaf.us> wrote:
On 12/29/20 8:59 AM, Guido van Rossum wrote:
> On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:

>> The `__init_subclass__` and `__set_name__` protocols are
intended to be run before a new type is finished, but creating
>> a new type has three major steps:
>>
>> - `__prepare__` to get the namespace
>> - `__new__` to get the memory and data structures
>> - `__init__` for any final polishing
>>
>> We can easily move the calls from `type_new()` to `type_init`.
>
> No, we can't. There is a window where the subclass is
initialized after `typing_new()` returned before `__init__`
> starts, and you propose to move the subclass after that window.
There may be code that depends on the class being
> initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move
some code from their `__new__` to `__init__` instead, and
a few that need to add an `__init__` to consume any keyword
arguments that don't need to get passed to
`__init_subclass__`.  That seems like a small price to pay to be
able to write custom metaclasses that are able to fully
participate in the `__set_name__` and `__init_subclass__` protocols.

Just in the stdlib we have two custom metaclasses that would start
working correctly with this change (for the
`__init_subclass__` case).

> Honestly I disapprove of the shenanigans you're committing in
enum.py, and if those are in in the 3.10 (master) branch I
> recommend that you take them out. It just looks too fragile and
obscure.

Trust me, I don't like them either.  But I like even less that a
custom `__init_subclass__` for an Enum would fail if it
tried to do anything with the members.  That's one of the reasons
why I would like to see this fix put in (the
shenanigans would be unnecessary then).

 From the point of view of a metaclass author, the current
behavior feels buggy.  In theory, `__init_subclass__` and
`__set_name__` are supposed to be called after a class is created,
and yet they are being called somewhere in the middle
of my metaclass' `__new__`.

Looked at another way, `__init_subclass__` should be receiving a
`cls` that is ready for use, i.e. done and fully
constructed and complete. But if that class is being created by a
custom metaclass, then the thing that is being given
to `__init_subclass__` could easily be only partial.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org
<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
<https://mail.python.org/mailman3/lists/python-dev.python.org/>
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
<https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/>
Code of Conduct: http://python.org/psf/codeofconduct/
<http://python.org/psf/codeofconduct/>
-- --Guido van Rossum (python.org/~guido http://python.org/~guido) /Pronouns: he/him //(why is my pronoun here?)/ http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AD2ORFW6... Code of Conduct: http://python.org/psf/codeofconduct/

Jonathan Goble

11:44 p.m.

On Wed, Dec 30, 2020, 1:05 AM Guido van Rossum guido@python.org wrote:

...

I need to think about this more.

Technically the class *is* created at the point `__init_subclass__` is called. What the metaclass does after that point is embellishment.

Most metaclasses will have sufficient control for their needs because they can manipulate the contents of the namespace that's passed to type_new() -- I understand that doesn't work for Enum because the instances have a reference to the class in them.

I suspect the classes that you claim are "buggy" are fine in practice, even if you can construct a counterexample.

All in all I just worry about the backward compatibility here: one could easily imagine a metaclass whose `__new__` relies on the effect of the `__init_subclass__` called by type_new(), and moving the call would break that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such a case, the status quo should win.

I admit I don't know anything about the metaclass internals, but if backward compatibility is an issue, couldn't we create a new dunder instead of moving the existing one? Say, call it __init_subclass_2__ (and bikeshed the name later), and document that as being called at the point to which it's currently proposed to move the calling of __init_subclass__.

With that, there is zero breakage of existing code, and code that can benefit from the new behavior simply uses a different dunder. __init_subclass__ could potentially be deprecated in the future if it's determined that it has no benefit over the new dunder, but doesn't need to deprecated immediately.

...

Nick Coghlan

5 Jan 5 Jan

5:22 a.m.

On Wed, 30 Dec 2020 at 04:38, Ethan Furman ethan@stoneleaf.us wrote:

...

...
No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__` starts, and you propose to move the subclass after that window. There may be code that depends on the class being initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to `__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully participate in the `__set_name__` and `__init_subclass__` protocols.

From https://www.python.org/dev/peps/pep-0487/#implementation-details:

"""As a third option, all the work could have been done in type.__init__. Most metaclasses do their work in __new__, as this is recommended by the documentation. Many metaclasses modify their arguments before they pass them over to super().__new__. For compatibility with those kind of classes, the hooks should be called from __new__."""

That isn't the clearest way of saying it, but the intent was to make sure that metaclass __new__ implementations continued to see fully initialised class objects as soon as they returned from type.__new__(), with type.__init__() continuing to be essentially a no-op.

If a metaclass wants to modify the attributes on the type in a way that is visible to __init_subclass__ and __set_name__, it needs to modify the namespace dict passed to type().__new__, not modify the type after it has already been created:

``` class Meta(type): # def __new__(mcls, name, bases, namespace, **kwds): # create new class, which will call __init_subclass__ and __set_name__ new_namespace = namespace.copy() # Customize class contents new_namespace["some_attr"] = 9 return type.__new__(mcls, name, bases, new_namespace, **kwds) ```

If you need to access the class being defined from __init_subclass__, it gets passed in, and if you need to access it from __set_name__, then it gets passed in there as well.

If you need to access the class being defined from elsewhere, then the `__class__` cell used by zero-arg super is already populated before any of those hooks get called.

As a CPython implementation detail, that cell is actually passed through to type().__new__ as `__classcell__` in the namespace. If that was elevated to a language feature rather than a CPython implementation detail, then metaclasses could write:

cls_cell = namespace["__classcell__"] @property def defining_class(obj): return cls_cell.get() namespace[defining_class.fget.__name__] = defining_class

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Ethan Furman

9:45 a.m.

On 1/5/21 4:22 AM, Nick Coghlan wrote:

...

On Wed, 30 Dec 2020 at 04:38, Ethan Furman wrote:

...

...
... there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to `__init_subclass__`. That seems like a small price to pay to be able to write custom metaclasses that are able to fully participate in the `__set_name__` and `__init_subclass__` protocols.

If a metaclass wants to modify the attributes on the type in a way that is visible to __init_subclass__ and __set_name__, it needs to modify the namespace dict passed to type().__new__, not modify the type after it has already been created.

And if that's not possible? (More on that later.)

...

If you need to access the class being defined from elsewhere, then the `__class__` cell used by zero-arg super is already populated before any of those hooks get called.

If the custom metaclass hasn't finished its work, I don't see how that is helpful.

...

As a CPython implementation detail, that cell is actually passed through to type().__new__ as `__classcell__` in the namespace. If that was elevated to a language feature rather than a CPython implementation detail, then metaclasses could write:
 cls_cell = namespace["__classcell__"]
 @property
 def defining_class(obj):
     return cls_cell.get()
 namespace[defining_class.fget.__name__] = defining_class

I have two concrete examples of custom metaclasses currently broken by calling `__init_subclass__` from `type.__new__`:

- `ABCMeta`

- `Enum`

Perhaps `ABCMeta` can be easily fixed -- it is calling the function that records all abstract methods, etc., after the `type.__new__` call; can it be called before?

`Enum` cannot be fixed: it must call `type.__new__` to get the new class object, and it must have the new class object in order to create the members as instances of that `Enum` class -- which means that anyone trying to use `__init_subclass__` to do anything with an `Enum`'s members is doomed to failure because those members don't exist yet.

Possible work-arounds for `Enum`:

- call `__init_subclass__` twice - once from `type.__new__`, and then again by `EnumMeta` after the members have been added

* problem: authors of `__init_subclass__` have to be aware that they will get called twice

- do horribly brittle mro rewrites in `EnumMeta` so the `type.__new__` call is a no-op, then `EnumMeta` calls `__init_subclass__` when the members have been created

* problem: horribly brittle

------------------------------------------------------------------------------

While I would prefer to see the calls to `__init_subclass__` and `__set_name__` (and any other such enhancements) moved to `type.__init__`, another possibility would be to add a class attribute/keyword, `_pep487_to_init`, which if present causes `type.__new__` to skip those calls and `type.__init__` to make them. Then the only headaches would be when metaclasses with different desires were combined.

I am also open to other solutions that would allow `EnumMeta` and its users to be able to use `__init_subclass__` with a fully created class.

-- ~Ethan~

Guido van Rossum

11:25 a.m.

On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman ethan@stoneleaf.us wrote:

...

[...] Perhaps `ABCMeta` can be easily fixed -- it is calling the function that records all abstract methods, etc., after the `type.__new__` call; can it be called before?

Why do you keep insisting that this is "broken"? Yes, you have identified a way that an abstract class can be instantiated *by its own constructor* despite being abstract. I think that's not something to get worked up about. As the doctor in the old joke said, "then don't do that." Or perhaps this is one of those situations where we're all still considered to be consenting adults -- surely abstract classes and methods are meant as a way to help users do the right thing, not as an absolute guarantee.

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/

Ethan Furman

12:41 p.m.

On 1/5/21 10:25 AM, Guido van Rossum wrote:

...

On Tue, Jan 5, 2021 at 8:50 AM Ethan Furman wrote:

...

...
[...] Perhaps `ABCMeta` can be easily fixed -- it is calling the function that records all abstract methods, etc., after the `type.__new__` call; can it be called before?

Why do you keep insisting that this is "broken"?

Because `__init_subclass__` is getting a new class object that is not ready to be gotten.

...

Yes, you have identified a way that an abstract class can be instantiated *by its own constructor* despite being abstract.

It doesn't feel that way to me. Referring to the code below [1], the constructor for `MyPlugin` is `abc.ABCMeta`, but `PluginBase.__init_subclass__` is getting `MyPlugin` before `abc.ABCMeta` has finished doing its job.

...

I think that's not something to get worked up about. As the doctor in the old joke said, "then don't do that." Or perhaps this is one of those situations where we're all still considered to be consenting adults -- surely abstract classes and methods are meant as a way to help users do the right thing, not as an absolute guarantee.

According to the documentation:

...

This module provides abstract base classes that can be used to test whether a class provides a particular interface; for example, whether it is hashable or whether it is a mapping.

Do we need to add a warning to that?

...

except inside `__init_subclass__` as the class has not been fully initialized.

For that matter, if we are not going to address this issue with code, we should definitely update the `__init_subclass__` and `__set_name__` documentation:

...

Note: custom metaclasses must have all changes needed already in the class namespace before calling `type.__new__`, as `type.__new__` will invoke `__init_subclass__` and `__set_name__` before returning. Any modifications made to the class after that point will not be available to those two methods.

Note: because of the above, any `__init_subclass__` used with `Enum` classes will not see any members as members are added after the call to `type.__new__`.

Note: because of the above, any `__init_subclass__` used with `ABC` classes will have to determine for itself if the class is abstract or concrete as that is determined after the call to `type.__new__`.

As I said before, perhaps `_abc_init` can be called before `type.__new__`, which would make that last note unnecessary.

That would not solve the problem for `Enum`, though, nor for any other metaclass that cannot complete its customization before calling `type.__new__`.

-- ~Ethan~

[1] class PluginBase(abc.ABC):

# base class for plugins # must override `do_something`

def __init_subclass__(cls, **kwargs):

instance = cls() print(f"Created instance of {cls} easily: {instance}")

@abc.abstractmethod def do_something(self): pass

class MyPlugin(PluginBase): pass

[2] https://docs.python.org/3/library/collections.abc.html

Nick Coghlan

7 Jan 7 Jan

5:56 a.m.

Both EnumMeta and ABCMeta should probably be relying on __set_name__ for their per-member set up work these days, rather than deferring that work until after __new__ returns.

Cheers, Nick.

Ethan Furman

11 Jan 11 Jan

6:42 p.m.

On 1/7/21 4:56 AM, Nick Coghlan wrote:

...

Both EnumMeta and ABCMeta should probably be relying on `__set_name__` for their per-member set up work these days, rather than deferring that work until after __new__ returns.

And here I was thinking that `__set_name__` was for, well, setting the name. ;-)

But it does work.

-- ~Ethan~

Nick Coghlan

16 Jan 16 Jan

9:03 a.m.

On Tue, 12 Jan 2021, 11:55 am Ethan Furman, ethan@stoneleaf.us wrote:

...

On 1/7/21 4:56 AM, Nick Coghlan wrote:

...
Both EnumMeta and ABCMeta should probably be relying on `__set_name__` for their per-member set up work these days, rather than deferring that work until after __new__ returns.

And here I was thinking that `__set_name__` was for, well, setting the name. ;-)

Yeah, __set_owner__ was also discussed as a possible name, since the method notifies the descriptor of both the defining class and the attribute name within that class.

We decided we preferred __set_name__ out of the two short options.

...