Re: [Python-Dev] PEP487: Simpler customization of class creation

Hello, This is my first post on python-dev and I hope that I am not breaking any rule. I wanted to react on the discussion regarding PEP487. This year, we have been working on a refactoring of the `traitlets` library, an implementation of the descriptor pattern that is used in Project Jupyter / IPython. The motivations for the refactoring was similar to those of this PEP: having a more generic metaclass allowing more flexibility in terms of types of descriptors, in order to avoid conflicts between meta classes. We ended up with: - A metaclass called MetaHasDescriptor - A base class of meta MetaHasDescriptor named HasDescriptors Usage: class MyClass(HasDescriptors): attr = DesType() DesType inherits from a base Descriptor type. The key is that their initialization is done in three stages - the main DesType.__init__ - the part of the initialization of DesType that depends on the definition of MyClass DesType.class_init(self, cls, name) which is called from MetaHasDescriptors.__new__ - a method of DesType that depends on the definition of instances of MyClass DesType.instance_init(self, obj) which is called from HasDescriptors.__new__. instance_init, may make modifications on the HasDescriptors instance. My understanding is that the proposed __set_name__ in PEP487 exactly corresponds to our class_init, although interestingly we often do much more in class_init than setting the name of the descriptor, such as setting a this_class attribute or calling class_init on contained descriptors. Therefore I do not think that the names __set_name__ or __set_owner__ are appropriate for this use case. In a way, the long-form explicit names for our class_init and instance_init methods would be something like __init_fom_owner_class__, and __touch_instance__. Thanks, Sylvain PS: thanks to Neil Girdhar for the heads up on the traitlets repo.

Hi Sylvain, Thanks for getting in touch! The traitlets library sounds interesting, and provides good additional evidence that this is a capability that folks are interested in having available. On 20 July 2016 at 15:26, Sylvain Corlay <sylvain.corlay@gmail.com> wrote:
It's certainly a reasonable question/concern, but we've learned from experience that we're better off using protocol method names that are very specific to a particular intended use case, even if they can be adapted for other purposes. The trick is that we want educators teaching Python to be able to very easily answer the question of "What is this special method for?" (even if they later go on to say "And it's also used for these other things...") One previous example of that is the __index__ protocol, where the actual semantics are "instances of this type can be losslessly converted to integers", but the protocol is named for the particular use case "instances of this type can be used as sequence indices". For PEP 487, the two operations guiding the naming of the methods are "notify a base class when a new subclass is defined" and "notify a descriptor of its attribute name when assigned to a class". The precise verbs then mirror those already used in other parts of the related protocols (with __init__ leading to __init_subclass__, and __set__ leading to __set_name__). The main capability that __set_name__ provides that was previously difficult is letting a descriptor know its own name in the class namespace. The fact a descriptor implementor can do anything else they want as a side-effect of that new method being called isn't substantially different from the ability to add side-effects to the existing __get__, __set__ and __delete__ protocol methods. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi Nick, Thank you for your reply. I understand your argument about using protocol method names that are very specific to a particular intended use case. Interestingly, the one example that is provided in the PEP is that of a "trait" which is pretty much the same as traitlets. (traitlets started as a pure python implementation of Enthought's traits library). My point is that in any real-world implementation of traits, __set_name__ will do a lot more than setting the name, which makes the name misleading. Cheers, Sylvain On Wed, Jul 20, 2016 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 21 July 2016 at 03:40, Sylvain Corlay <sylvain.corlay@gmail.com> wrote:
My point is that in any real-world implementation of traits, __set_name__ will do a lot more than setting the name, which makes the name misleading.
I suspect the point of disagreement on that front may be in how we view the names of the existing __get__, __set__ and __delete__ methods in the descriptor protocols - all 3 of those are in the form of event notifications to the descriptor to say "someone is getting the attribute", "someone is setting the attribute" and "someone is deleting the attribute". What the descriptor does in response to those notifications is up to the descriptor, with it being *conventional* that they be at least plausibly associated with the "obj.attr", "obj.attr = value" and "del obj.attr" operations (with folks voting by usage as to whether or not they consider a particular API's side effects in response to those notifications reasonable). The new notification is merely "someone is setting the name of the attribute", with that taking place when the contents of a class namespace are converted into class attributes. However, phrasing it that way suggest that it's possible we *did* miss something in the PEP: we haven't specified whether or not __set_name__ should be called when someone does someone does "cls.attr = descr". Given the name, I think we *should* call it in that case, and then the semantics during class creation are approximately what would happen if we actually built up the class attributes as: for attr, value in cls_ns.items(): setattr(cls, attr, value) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Whoa. That's not how I read it. --Guido (mobile)

In any case I find this PEP great. If we can implement a library like traitlets only using these new hooks without a custom metaclass, it will be a big improvement. My only concern was that calling the hook __set_name__ was misleading while it could not set the name at all and do pretty much anything else. Regards, Sylvain On Thu, Jul 21, 2016 at 4:53 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Hi list, Hi Nick, Sorry for my delayed response, it is summer here...
That's a very good point and actually easy to solve: we would just need to override type.__setattr__ to do so. Actually, it is already overridden, so we just need to add code to type.__setattr__ to also call __set_name__. One could be of the other standpoint that in your above example it's the duty of the caller of setattr to also call __set_name__. It would be pretty easy to add a line in the loop that also calls __set_owner__. Greetings Martin

On 07/24/2016 08:20 AM, Guido van Rossum wrote:
I am very much against this. The two are not at all like each other. Also, what's the use case?
To be clear: you are against the automatic calling of __set_name__ and/or __set_owner__ when using setattr outside of class creation? Said another way: class creation mechanics should only happen during class creation? -- ~Ethan~

On 25 July 2016 at 03:00, Guido van Rossum <gvanrossum@gmail.com> wrote:
Yes.
OK, we can cover that in the documentation - if folks want to emulate what happens during class construction after the fact, they'll need to do: cls.name = attr attr.__set_name__(cls, "name") Semantically, I agree that approach makes sense - by default, descriptors created outside a class body won't have a defined owning class or attribute name, and if you want to give them one, you'll have to do it explicitly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi - sorry for steppign in late - I've just reread the PEP and tried out the reference implementation, and I have two sugestions/issues with it as is: 1) Why does `__init_subclass__` is not run in the class it is defined itself?? That makes no sense to me as in very unpythonic. I applied the patch at issue, 27366 went to the terminal, and created a "hello world" __init_subclass__ with a simple print statement, and was greatly surprised that it did not printout. Only upon reviewing the PEP text I inferred that it was supposed to work (as it did) in further subclasses of my initial class. After all, issubclass(A, A) is usually True. I pledge for this behavior to be changed on the PEP. If one does not want it to run on the baseclass, a simple default argument and an `if param is None: return` on the method body can do the job, with less exceptions and surprises. Otherwise, I'd suggest at least some PEP rewording to make explicit the fact it is not run on the class it is defined at all. (I can help with that, but I'd rather see it implemented as above). I can see the fact that it woudl have little effect, as any eventual parameter on the Base class could be hardcoded into the class body itself - but just imagine a class hierarchy with cooperative "__init_subclass__" methods - it would be rather surprising that each class has to count on its parents __init_subclass__ being run, without the one defined in its own body being called - that is rather surprising. ------- 2) I have another higher level concern with the PEP as well: It will pass all class keyword parameters, but for "metaclass" to "__init_subclass__" - thus making it all but impossible to any other keyword to the class creation machinery to ever be defined again. We can't think of any such other keyword now, but what might come in a couple of years? What about just denoting in the PEP that "double under" keywords should be reserved and not relied to not be used by "type" itself in the future? (or any other way of marking reserved class kewyords) - actually it would even make sense to make "__metaclass__" an alias for "metaclass" in the class creation machinery. Anyway the PEP itself should mention that currently the keyword-arg "metaclass" is swallowed by the class creation machinery and will never reach `__init_subclass__` - this behavior is less surprising for me, but it should be documented) Or, an even less exceptional behavior for the future: make it so that "metaclass" specifies a custom metaclass (due to compatibility issues) AND is passed to __init_subclass__, and "__metaclass__" specifies a metaclass and is not passed (along with other double-unders as they are defined)). (as an extra bonus, people migrating from Python 2 to Python 3.6 will not even be surprised by the keyword argument being __metaclass__) Best regards, js -><- On 25 July 2016 at 00:49, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 28 July 2016 at 03:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
This is covered in the PEP: https://www.python.org/dev/peps/pep-0487/#calling-the-hook-on-the-class-itse...
This isn't a new problem, as it already exists today for custom metaclasses. It just means introducing new class construction keywords at the language level is something that needs to be handled judiciously, and with a view to the fact that it might have knock-on effects for other APIs which need to find a new parameter name. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 July 2016 at 22:30, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, as documented on the PEP (and I just confirmed at a Python 3.5 prompt), you actually can't use custom keywords for class defintions. This PEP fixes that, but at the same time makes any other class reserved keyword impossible in the future - that is, unless a single line warning against reserved name patterns is added. I think it is low a cost not to be paid now, blocking too many - yet to be imagined - future possibilities. (as for the example in Py 3.5): In [17]: def M(type): ...: def __new__(metacls, name, bases, dict, **kw): ...: print(kw) ...: return super().__new__(name, bases, dict) ...: def __init__(cls, name, bases, dict, **kw): ...: print("init...", kw) ...: return super().__init__(name, bases, dict) ...: In [18]: class A(metaclass=M, test=23, color="blue"): ...: pass ...: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-18-86d111b5b69c> in <module>() ----> 1 class A(metaclass=M, test=23, color="blue"): 2 pass TypeError: M() got an unexpected keyword argument 'color'

On 28 July 2016 at 13:55, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
We don't warn against people defining new dunder-protocols as methods, why would we warn against a similar breach of convention in this case? I'm also wondering how you would want such a warning to work if we ever claimed a parameter name for a base class in the standard library, but didn't claim it as a name supported by type/object. Note that I'm not denying that it *may* be annoying *if* we define a new universal class parameter at some point in the future *and* it collides with a custom parameter in a pre-existing API *and* the authors of that API miss the related PEP. However, given that we've come up with exactly one named class parameter to date (metaclass), and explicitly decided against adding another (namespace, replaced with PEP 520's simpler option of just making the standard namespace provide attribute ordering data), the odds of actually encountering the posited problematic scenario seem pretty remote. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 July 2016 at 04:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
That is alright. (Even though, I think somewhere around there are remarks against one putting forth his own dunder methods) . But that elaves another issue open: the "metaclass" parameter get in to a very odd position, in which it has nothing distinctive about it, still is the only parameter that will be swallowed and won't reach "__init_subclass__". Although I know it is not straightforward to implement (as the "metaclass" parameter is not passed to the metaclass __new__ or __init__), wouldn't it make sense to make it be passed to __init_subclass__ just like all other keywords? (the default __init_subclass__ then could swallow it, if it reaches there). I am putting the question now, because it is a matter of "now" or "never" - I can see it can does make sense if it is not passed down. Anyway, do you have any remarks on the first issue I raised? About __init_subclass__ being called in the class it is defined in, not just on it's descendant subclasses?

On 28 July 2016 at 23:12, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
It would complicate the implementation, and has the potential to be confusing (since the explicit metaclass hint and the actual metaclass aren't guaranteed to be the same), so I don't think it makes sense to pass it down. I'll make sure we note it in the documentation for the new protocol method, though.
That's already covered in the PEP: https://www.python.org/dev/peps/pep-0487/#calling-the-hook-on-the-class-itse... We want to make it easy for mixin classes to use __init_subclass__ to define "required attributes" on subclasses, and that's straightforward with the current definition: class MyMixin: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) if not hasattr(cls, "mixin_required_attribute"): raise TypeError(f"Subclasses of {__class__} must define a 'mixin_required_attribute' attribute") If you actually do want the init_subclass__ method to also run on the "base" class, then you'll need to tweak the class hierarchy a bit to look like: class _PrivateInitBase: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) ... def MyOriginalClass(_PrivateInitBase): ... (You don't want to call __init_subclass__ from a class decorator, as that would call any parent __init_subclass__ implementations a second time) By contrast, when we had the default the other way around, opting *out* of self-application required boilerplate inside of __init_subclass__ to special case the situation where "cls is __class__": class MyMixin: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) if cls is __class__: return # Don't init the base class if not getattr(cls, "mixin_required_attribute", None) is None: raise TypeError(f"Subclasses of {__class__} must define a non-None 'mixin_required_attribute' attribute") This raises exciting new opportunities for subtle bugs, like bailing out *before* calling the parent __init_subclass__ method, and then figure out that a later error from an apparently unrelated method is because your __init_subclass__ implementation is buggy. There's still an opportunity for bugs with the current design decision (folks expecting __init_subclass__ to be called on the class defining it when that isn't the case), but they should be relatively shallow ones, and once people learn the rule that __init_subclass__ is only called on *strict* subclasses, it's a pretty easy behaviour to remember. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 July 2016 at 10:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks for the extensive reasoning. Maybe then adding a `run_init_subclass` class decorator on the stdlib to go along with the pep? It should be a two liner that would avoid boiler plate done wrong - but more important than thatm is that it being documented alog with the __init_sublass__ method will make it more obvious it is not run where it is defined. (I had missed it in the PEP text and just understood that part when re-reading the PEP after being surprised) js -><-

On 29 July 2016 at 00:06, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
The class decorator approach looks like this: def run_init_subclass(cls, **kwds): cls.__init_subclass__(**kwds) However, it's not the right way to do it, as it means super(cls, cls).__init_subclass__(**kwds) will get called a second time (since the class machinery will have already done it implicitly before invoking the class decorators). If the parent class methods are idempotent then calling them again won't matter, but if you're supplying extra keyword arguments, you'd need to specify them in both the decorator call *and* the class header. (I didn't actually realise this problem until writing the earlier email, so I'll probably tweak that part of the PEP to be more explicit about this aspect) So the simplest approach if you want "this class *and* all its descendants" behaviour is to adhere to the "strict subclasses only" requirement, and put the __init_subclass__ implementation in a base class, even if that means creating an additional mixin just to hold it. If that recommended approach isn't acceptable for some reason, then the decorator-centric alternative would be to instead write it this way: def decorate_class(cls, **kwds): ... @decorate_class class MyClass: def __init_subclass__(cls, **kwds): super().__init_subclass__(**kwds) decorate_class(cls) So the decorator gets defined outside the class, applied explicitly to the base class, and then the __init_subclass__ hook applies it implicitly to all subclasses. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hello, there has been quite some discussion on why PEP 487's __init_subclass__ initializes subclasses, and not the class itself. I think the most important details have been already thoroughly discussed here. One thing I was missing in the discussion is practical examples. I have been using PEP 487-like metaclasses since several years now, and I have never come across an application where it would have even been nice to initialize the class itself. Also, while researching other people's code when I wrote PEP 487, I couldn't find any such code elsewhere, yet I found a lot of code where people took the wildest measure to prevent a metaclass in doing its job on the first class it is used for. One example is enum.EnumMeta, which contains code not to make enum.Enum an enum (I do not try to propose that the enum module should use PEP 487, it's way to complicated for that). So once we have a practical example, we could start discussing how to mitigate the problem. Btw, everyone is still invited to review the patch for PEP 487 at http://bugs.python.org/issue27366. Many thanks to Nick who already reviewed, and also to Guido who left helpful comments! Greetings Martin

In the traitlets library I mentioned earlier, we do have a need for this. The corresponding function is called `setup_class`. What it does is setting some class attributes that are required for certain types of descriptors to be able to initialize themselves. class MetaHasTraits(MetaHasDescriptors): """A metaclass for HasTraits.""" def setup_class(cls, classdict): cls._trait_default_generators = {} super(MetaHasTraits, cls).setup_class(classdict) Sylvain On Fri, Jul 29, 2016 at 5:01 PM, Martin Teichmann <lkb.teichmann@gmail.com> wrote:

Hi Sylvain, thanks for the example, it's a great example to illustrate PEP 487 and its design decisions.
While in a metaclass solution this does need that the metaclass needs to execute code on the first class it is used for, in a PEP 487 solution this is not the case. A PEP 487 class HasTraits (no Meta here) would not have Traits-descriptors itself, the classes inheriting from it would have traits to be initialized. The PEP 487 HasTraits takes the role of the metaclass. A completely different problem shows up here. In your example, HasTraits needs to initialize things on the class before the Descriptors are run. This is not possible with PEP 487, where the descriptors are initialized before __init_subclass__ is even called. There are two ways to mitigate that problem: - the first initialized descriptor could do the necessary initialization, or - the descriptors are initialized from within __init_subclass__ At first sight, the first option seems hackish and the second option looks much saner. Nonetheless PEP 487 proposes the second solution. The rationale behind that is that people tend to forget to call super(), and suddenly descriptors don't work anymore. I realized later there is another benefit to this: if the first initialized descriptor is doing the class initialization, often __init_subclass__ isn't needed at all anymore, which means that those kind of descriptors can be used on any class, without the need to tell users that they have to inherit from a certain base class for the descriptors to work. Only if some finalizing code needs to run after the last descriptor is initialized one needs to write an __init_subclass__. This is unavoidable as the last descriptor doesn't know that it is the last. Greetings Martin

Hi Martin, I think that we are on the same page. The differences in the approaches here makes me think that I should attempt a PEP487-based migration of traitlets and see if there is a major roadblock. After all, that would probably be a good test for the PEP if the aim is to enable usecases like traitlets without a metaclass. Sylvain

On 07/29/2016 08:01 AM, Martin Teichmann wrote:
Actually, enum.Enum is an enum. The guards are there because part of creating a new Enum class is searching for the previous Enum classes, and of course the very first time through there is no previous Enum class. My apologies if I have misunderstood what you were trying to say. -- ~Ethan~

Hi Sylvain, Thanks for getting in touch! The traitlets library sounds interesting, and provides good additional evidence that this is a capability that folks are interested in having available. On 20 July 2016 at 15:26, Sylvain Corlay <sylvain.corlay@gmail.com> wrote:
It's certainly a reasonable question/concern, but we've learned from experience that we're better off using protocol method names that are very specific to a particular intended use case, even if they can be adapted for other purposes. The trick is that we want educators teaching Python to be able to very easily answer the question of "What is this special method for?" (even if they later go on to say "And it's also used for these other things...") One previous example of that is the __index__ protocol, where the actual semantics are "instances of this type can be losslessly converted to integers", but the protocol is named for the particular use case "instances of this type can be used as sequence indices". For PEP 487, the two operations guiding the naming of the methods are "notify a base class when a new subclass is defined" and "notify a descriptor of its attribute name when assigned to a class". The precise verbs then mirror those already used in other parts of the related protocols (with __init__ leading to __init_subclass__, and __set__ leading to __set_name__). The main capability that __set_name__ provides that was previously difficult is letting a descriptor know its own name in the class namespace. The fact a descriptor implementor can do anything else they want as a side-effect of that new method being called isn't substantially different from the ability to add side-effects to the existing __get__, __set__ and __delete__ protocol methods. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi Nick, Thank you for your reply. I understand your argument about using protocol method names that are very specific to a particular intended use case. Interestingly, the one example that is provided in the PEP is that of a "trait" which is pretty much the same as traitlets. (traitlets started as a pure python implementation of Enthought's traits library). My point is that in any real-world implementation of traits, __set_name__ will do a lot more than setting the name, which makes the name misleading. Cheers, Sylvain On Wed, Jul 20, 2016 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 21 July 2016 at 03:40, Sylvain Corlay <sylvain.corlay@gmail.com> wrote:
My point is that in any real-world implementation of traits, __set_name__ will do a lot more than setting the name, which makes the name misleading.
I suspect the point of disagreement on that front may be in how we view the names of the existing __get__, __set__ and __delete__ methods in the descriptor protocols - all 3 of those are in the form of event notifications to the descriptor to say "someone is getting the attribute", "someone is setting the attribute" and "someone is deleting the attribute". What the descriptor does in response to those notifications is up to the descriptor, with it being *conventional* that they be at least plausibly associated with the "obj.attr", "obj.attr = value" and "del obj.attr" operations (with folks voting by usage as to whether or not they consider a particular API's side effects in response to those notifications reasonable). The new notification is merely "someone is setting the name of the attribute", with that taking place when the contents of a class namespace are converted into class attributes. However, phrasing it that way suggest that it's possible we *did* miss something in the PEP: we haven't specified whether or not __set_name__ should be called when someone does someone does "cls.attr = descr". Given the name, I think we *should* call it in that case, and then the semantics during class creation are approximately what would happen if we actually built up the class attributes as: for attr, value in cls_ns.items(): setattr(cls, attr, value) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Whoa. That's not how I read it. --Guido (mobile)

In any case I find this PEP great. If we can implement a library like traitlets only using these new hooks without a custom metaclass, it will be a big improvement. My only concern was that calling the hook __set_name__ was misleading while it could not set the name at all and do pretty much anything else. Regards, Sylvain On Thu, Jul 21, 2016 at 4:53 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Hi list, Hi Nick, Sorry for my delayed response, it is summer here...
That's a very good point and actually easy to solve: we would just need to override type.__setattr__ to do so. Actually, it is already overridden, so we just need to add code to type.__setattr__ to also call __set_name__. One could be of the other standpoint that in your above example it's the duty of the caller of setattr to also call __set_name__. It would be pretty easy to add a line in the loop that also calls __set_owner__. Greetings Martin

On 07/24/2016 08:20 AM, Guido van Rossum wrote:
I am very much against this. The two are not at all like each other. Also, what's the use case?
To be clear: you are against the automatic calling of __set_name__ and/or __set_owner__ when using setattr outside of class creation? Said another way: class creation mechanics should only happen during class creation? -- ~Ethan~

On 25 July 2016 at 03:00, Guido van Rossum <gvanrossum@gmail.com> wrote:
Yes.
OK, we can cover that in the documentation - if folks want to emulate what happens during class construction after the fact, they'll need to do: cls.name = attr attr.__set_name__(cls, "name") Semantically, I agree that approach makes sense - by default, descriptors created outside a class body won't have a defined owning class or attribute name, and if you want to give them one, you'll have to do it explicitly. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hi - sorry for steppign in late - I've just reread the PEP and tried out the reference implementation, and I have two sugestions/issues with it as is: 1) Why does `__init_subclass__` is not run in the class it is defined itself?? That makes no sense to me as in very unpythonic. I applied the patch at issue, 27366 went to the terminal, and created a "hello world" __init_subclass__ with a simple print statement, and was greatly surprised that it did not printout. Only upon reviewing the PEP text I inferred that it was supposed to work (as it did) in further subclasses of my initial class. After all, issubclass(A, A) is usually True. I pledge for this behavior to be changed on the PEP. If one does not want it to run on the baseclass, a simple default argument and an `if param is None: return` on the method body can do the job, with less exceptions and surprises. Otherwise, I'd suggest at least some PEP rewording to make explicit the fact it is not run on the class it is defined at all. (I can help with that, but I'd rather see it implemented as above). I can see the fact that it woudl have little effect, as any eventual parameter on the Base class could be hardcoded into the class body itself - but just imagine a class hierarchy with cooperative "__init_subclass__" methods - it would be rather surprising that each class has to count on its parents __init_subclass__ being run, without the one defined in its own body being called - that is rather surprising. ------- 2) I have another higher level concern with the PEP as well: It will pass all class keyword parameters, but for "metaclass" to "__init_subclass__" - thus making it all but impossible to any other keyword to the class creation machinery to ever be defined again. We can't think of any such other keyword now, but what might come in a couple of years? What about just denoting in the PEP that "double under" keywords should be reserved and not relied to not be used by "type" itself in the future? (or any other way of marking reserved class kewyords) - actually it would even make sense to make "__metaclass__" an alias for "metaclass" in the class creation machinery. Anyway the PEP itself should mention that currently the keyword-arg "metaclass" is swallowed by the class creation machinery and will never reach `__init_subclass__` - this behavior is less surprising for me, but it should be documented) Or, an even less exceptional behavior for the future: make it so that "metaclass" specifies a custom metaclass (due to compatibility issues) AND is passed to __init_subclass__, and "__metaclass__" specifies a metaclass and is not passed (along with other double-unders as they are defined)). (as an extra bonus, people migrating from Python 2 to Python 3.6 will not even be surprised by the keyword argument being __metaclass__) Best regards, js -><- On 25 July 2016 at 00:49, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 28 July 2016 at 03:56, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
This is covered in the PEP: https://www.python.org/dev/peps/pep-0487/#calling-the-hook-on-the-class-itse...
This isn't a new problem, as it already exists today for custom metaclasses. It just means introducing new class construction keywords at the language level is something that needs to be handled judiciously, and with a view to the fact that it might have knock-on effects for other APIs which need to find a new parameter name. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 July 2016 at 22:30, Nick Coghlan <ncoghlan@gmail.com> wrote:
Actually, as documented on the PEP (and I just confirmed at a Python 3.5 prompt), you actually can't use custom keywords for class defintions. This PEP fixes that, but at the same time makes any other class reserved keyword impossible in the future - that is, unless a single line warning against reserved name patterns is added. I think it is low a cost not to be paid now, blocking too many - yet to be imagined - future possibilities. (as for the example in Py 3.5): In [17]: def M(type): ...: def __new__(metacls, name, bases, dict, **kw): ...: print(kw) ...: return super().__new__(name, bases, dict) ...: def __init__(cls, name, bases, dict, **kw): ...: print("init...", kw) ...: return super().__init__(name, bases, dict) ...: In [18]: class A(metaclass=M, test=23, color="blue"): ...: pass ...: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-18-86d111b5b69c> in <module>() ----> 1 class A(metaclass=M, test=23, color="blue"): 2 pass TypeError: M() got an unexpected keyword argument 'color'

On 28 July 2016 at 13:55, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
We don't warn against people defining new dunder-protocols as methods, why would we warn against a similar breach of convention in this case? I'm also wondering how you would want such a warning to work if we ever claimed a parameter name for a base class in the standard library, but didn't claim it as a name supported by type/object. Note that I'm not denying that it *may* be annoying *if* we define a new universal class parameter at some point in the future *and* it collides with a custom parameter in a pre-existing API *and* the authors of that API miss the related PEP. However, given that we've come up with exactly one named class parameter to date (metaclass), and explicitly decided against adding another (namespace, replaced with PEP 520's simpler option of just making the standard namespace provide attribute ordering data), the odds of actually encountering the posited problematic scenario seem pretty remote. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 July 2016 at 04:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
That is alright. (Even though, I think somewhere around there are remarks against one putting forth his own dunder methods) . But that elaves another issue open: the "metaclass" parameter get in to a very odd position, in which it has nothing distinctive about it, still is the only parameter that will be swallowed and won't reach "__init_subclass__". Although I know it is not straightforward to implement (as the "metaclass" parameter is not passed to the metaclass __new__ or __init__), wouldn't it make sense to make it be passed to __init_subclass__ just like all other keywords? (the default __init_subclass__ then could swallow it, if it reaches there). I am putting the question now, because it is a matter of "now" or "never" - I can see it can does make sense if it is not passed down. Anyway, do you have any remarks on the first issue I raised? About __init_subclass__ being called in the class it is defined in, not just on it's descendant subclasses?

On 28 July 2016 at 23:12, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
It would complicate the implementation, and has the potential to be confusing (since the explicit metaclass hint and the actual metaclass aren't guaranteed to be the same), so I don't think it makes sense to pass it down. I'll make sure we note it in the documentation for the new protocol method, though.
That's already covered in the PEP: https://www.python.org/dev/peps/pep-0487/#calling-the-hook-on-the-class-itse... We want to make it easy for mixin classes to use __init_subclass__ to define "required attributes" on subclasses, and that's straightforward with the current definition: class MyMixin: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) if not hasattr(cls, "mixin_required_attribute"): raise TypeError(f"Subclasses of {__class__} must define a 'mixin_required_attribute' attribute") If you actually do want the init_subclass__ method to also run on the "base" class, then you'll need to tweak the class hierarchy a bit to look like: class _PrivateInitBase: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) ... def MyOriginalClass(_PrivateInitBase): ... (You don't want to call __init_subclass__ from a class decorator, as that would call any parent __init_subclass__ implementations a second time) By contrast, when we had the default the other way around, opting *out* of self-application required boilerplate inside of __init_subclass__ to special case the situation where "cls is __class__": class MyMixin: def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) if cls is __class__: return # Don't init the base class if not getattr(cls, "mixin_required_attribute", None) is None: raise TypeError(f"Subclasses of {__class__} must define a non-None 'mixin_required_attribute' attribute") This raises exciting new opportunities for subtle bugs, like bailing out *before* calling the parent __init_subclass__ method, and then figure out that a later error from an apparently unrelated method is because your __init_subclass__ implementation is buggy. There's still an opportunity for bugs with the current design decision (folks expecting __init_subclass__ to be called on the class defining it when that isn't the case), but they should be relatively shallow ones, and once people learn the rule that __init_subclass__ is only called on *strict* subclasses, it's a pretty easy behaviour to remember. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 July 2016 at 10:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks for the extensive reasoning. Maybe then adding a `run_init_subclass` class decorator on the stdlib to go along with the pep? It should be a two liner that would avoid boiler plate done wrong - but more important than thatm is that it being documented alog with the __init_sublass__ method will make it more obvious it is not run where it is defined. (I had missed it in the PEP text and just understood that part when re-reading the PEP after being surprised) js -><-

On 29 July 2016 at 00:06, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
The class decorator approach looks like this: def run_init_subclass(cls, **kwds): cls.__init_subclass__(**kwds) However, it's not the right way to do it, as it means super(cls, cls).__init_subclass__(**kwds) will get called a second time (since the class machinery will have already done it implicitly before invoking the class decorators). If the parent class methods are idempotent then calling them again won't matter, but if you're supplying extra keyword arguments, you'd need to specify them in both the decorator call *and* the class header. (I didn't actually realise this problem until writing the earlier email, so I'll probably tweak that part of the PEP to be more explicit about this aspect) So the simplest approach if you want "this class *and* all its descendants" behaviour is to adhere to the "strict subclasses only" requirement, and put the __init_subclass__ implementation in a base class, even if that means creating an additional mixin just to hold it. If that recommended approach isn't acceptable for some reason, then the decorator-centric alternative would be to instead write it this way: def decorate_class(cls, **kwds): ... @decorate_class class MyClass: def __init_subclass__(cls, **kwds): super().__init_subclass__(**kwds) decorate_class(cls) So the decorator gets defined outside the class, applied explicitly to the base class, and then the __init_subclass__ hook applies it implicitly to all subclasses. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hello, there has been quite some discussion on why PEP 487's __init_subclass__ initializes subclasses, and not the class itself. I think the most important details have been already thoroughly discussed here. One thing I was missing in the discussion is practical examples. I have been using PEP 487-like metaclasses since several years now, and I have never come across an application where it would have even been nice to initialize the class itself. Also, while researching other people's code when I wrote PEP 487, I couldn't find any such code elsewhere, yet I found a lot of code where people took the wildest measure to prevent a metaclass in doing its job on the first class it is used for. One example is enum.EnumMeta, which contains code not to make enum.Enum an enum (I do not try to propose that the enum module should use PEP 487, it's way to complicated for that). So once we have a practical example, we could start discussing how to mitigate the problem. Btw, everyone is still invited to review the patch for PEP 487 at http://bugs.python.org/issue27366. Many thanks to Nick who already reviewed, and also to Guido who left helpful comments! Greetings Martin

In the traitlets library I mentioned earlier, we do have a need for this. The corresponding function is called `setup_class`. What it does is setting some class attributes that are required for certain types of descriptors to be able to initialize themselves. class MetaHasTraits(MetaHasDescriptors): """A metaclass for HasTraits.""" def setup_class(cls, classdict): cls._trait_default_generators = {} super(MetaHasTraits, cls).setup_class(classdict) Sylvain On Fri, Jul 29, 2016 at 5:01 PM, Martin Teichmann <lkb.teichmann@gmail.com> wrote:

Hi Sylvain, thanks for the example, it's a great example to illustrate PEP 487 and its design decisions.
While in a metaclass solution this does need that the metaclass needs to execute code on the first class it is used for, in a PEP 487 solution this is not the case. A PEP 487 class HasTraits (no Meta here) would not have Traits-descriptors itself, the classes inheriting from it would have traits to be initialized. The PEP 487 HasTraits takes the role of the metaclass. A completely different problem shows up here. In your example, HasTraits needs to initialize things on the class before the Descriptors are run. This is not possible with PEP 487, where the descriptors are initialized before __init_subclass__ is even called. There are two ways to mitigate that problem: - the first initialized descriptor could do the necessary initialization, or - the descriptors are initialized from within __init_subclass__ At first sight, the first option seems hackish and the second option looks much saner. Nonetheless PEP 487 proposes the second solution. The rationale behind that is that people tend to forget to call super(), and suddenly descriptors don't work anymore. I realized later there is another benefit to this: if the first initialized descriptor is doing the class initialization, often __init_subclass__ isn't needed at all anymore, which means that those kind of descriptors can be used on any class, without the need to tell users that they have to inherit from a certain base class for the descriptors to work. Only if some finalizing code needs to run after the last descriptor is initialized one needs to write an __init_subclass__. This is unavoidable as the last descriptor doesn't know that it is the last. Greetings Martin

Hi Martin, I think that we are on the same page. The differences in the approaches here makes me think that I should attempt a PEP487-based migration of traitlets and see if there is a major roadblock. After all, that would probably be a good test for the PEP if the aim is to enable usecases like traitlets without a metaclass. Sylvain

On 07/29/2016 08:01 AM, Martin Teichmann wrote:
Actually, enum.Enum is an enum. The guards are there because part of creating a new Enum class is searching for the previous Enum classes, and of course the very first time through there is no previous Enum class. My apologies if I have misunderstood what you were trying to say. -- ~Ethan~
participants (7)
-
Ethan Furman
-
Guido van Rossum
-
Guido van Rossum
-
Joao S. O. Bueno
-
Martin Teichmann
-
Nick Coghlan
-
Sylvain Corlay