Submitting PEP 422 (Simple class initialization hook) for pronouncement
I wrote PEP 422 (simple class initialization hook) last year after PJE pointed out that Python 3 style metaclasses made some Python 2 code impossible to migrate (since the class body could no longer request modifications to be made to the class after initialization was complete). It then languished, as I never found the time to actually implement it. Fortunately, Daniel Urban has now agreed to be a co-author on the PEP, and has fixed a couple of lingering technical errors in the PEP, as well as providing a reference implementation at http://bugs.python.org/issue17044 For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass. There's currently no BDFL delegate assigned, so this is a request for a pronouncement from Guido. Regards, Nick. ======================================= PEP: 422 Title: Simple class initialisation hook Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Daniel Urban <urban.dani+py@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Jun-2012 Python-Version: 3.4 Post-History: 5-Jun-2012, 10-Feb-2012 Abstract ======== In Python 2, the body of a class definition could modify the way a class was created (or simply arrange to run other code after the class was created) by setting the ``__metaclass__`` attribute in the class body. While doing this implicitly from called code required the use of an implementation detail (specifically, ``sys._getframes()``), it could also be done explicitly in a fully supported fashion (for example, by passing ``locals()`` to a function that calculated a suitable ``__metaclass__`` value) There is currently no corresponding mechanism in Python 3 that allows the code executed in the class body to directly influence how the class object is created. Instead, the class creation process is fully defined by the class header, before the class body even begins executing. This PEP proposes a mechanism that will once again allow the body of a class definition to more directly influence the way a class is created (albeit in a more constrained fashion), as well as replacing some current uses of metaclasses with a simpler, easier to understand alternative. Background ========== For an already created class ``cls``, the term "metaclass" has a clear meaning: it is the value of ``type(cls)``. *During* class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case: * If the metaclass hint refers to an instance of ``type``, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint. * Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls ``type``, or, in Python 3.3 or later, ``types.new_class``) the actual metaclass is then determined based on the parent classes. It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses. In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` keyword syntax in the class header. This allows the ``__prepare__`` method on the metaclass to be used to create the ``locals()`` namespace used during execution of the class body (for example, specifying the use of ``collections.OrderedDict`` instead of a regular ``dict``). In Python 2, there was no ``__prepare__`` method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__`` attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is `published code`_ that makes use of this feature. Another new feature in Python 3 is the zero-argument form of the ``super()`` builtin, introduced by PEP 3135. This feature uses an implicit ``__class__`` reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit ``__class__`` reference (as it is not populated until after the metaclass has returned control to the class creation machiner). Proposal ======== This PEP proposes that a mechanism be added to Python 3 that meets the following criteria: 1. Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style ``__metaclass__`` hook 2. Integrates nicely with class inheritance structures (including mixins and multiple inheritance) 3. Integrates nicely with the implicit ``__class__`` reference and zero-argument ``super()`` syntax introduced by PEP 3135 4. Can be added to an existing base class without a significant risk of introducing backwards compatibility problems One mechanism that can achieve this goal is to add a new class initialisation hook, modelled directly on the existing instance initialisation hook, but with the signature constrained to match that of an ordinary class decorator. Specifically, it is proposed that class definitions be able to provide a class initialisation hook as follows:: class Example: @classmethod def __init_class__(cls): # This is invoked after the class is created, but before any # explicit decorators are called # The usual super() mechanisms are used to correctly support # multiple inheritance. The class decorator style signature helps # ensure that invoking the parent class is as simple as possible. If present on the created object, this new hook will be called by the class creation machinery *after* the ``__class__`` reference has been initialised. For ``types.new_class()``, it will be called as the last step before returning the created class object. If a metaclass wishes to block class initialisation for some reason, it must arrange for ``cls.__init_class__`` to trigger ``AttributeError``. Note, that when ``__init_class__`` is called, the name of the class is not bound to the new class object yet. As a consequence, the two argument form of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)`` wouldn't work in the example above). However, the zero argument form of ``super()`` works as expected, since the ``__class__`` reference is already initialised. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering. Key Benefits ============ Replaces many use cases for dynamic setting of ``__metaclass__`` ----------------------------------------------------------------- For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3. Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. An earlier version of this PEP actually included such a mistake: it stated "subclass of type" for a constraint that is actually "instance of type". Understanding the proposed class initialisation hook only requires understanding decorators and ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_class__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. In fact, due to the constrained signature of ``__init_class__``, the risk in this case is actually even lower than in the case of ``__init__``. Integrates cleanly with \PEP 3135 --------------------------------- Unlike code that runs as part of the metaclass, code that runs as part of the new hook will be able to freely invoke class methods that rely on the implicit ``__class__`` reference introduced by PEP 3135, including methods that use the zero argument form of ``super()``. Alternatives ============ The Python 3 Status Quo ----------------------- The Python 3 status quo already offers a great deal of flexibility. For changes which only affect a single class definition and which can be specified at the time the code is written, then class decorators can be used to modify a class explicitly. Class decorators largely ignore class inheritance and can make full use of methods that rely on the ``__class__`` reference being populated. Using a custom metaclass provides the same level of power as it did in Python 2. However, it's notable that, unlike class decorators, a metaclass cannot call any methods that rely on the ``__class__`` reference, as that reference is not populated until after the metaclass constructor returns control to the class creation code. One major use case for metaclasses actually closely resembles the use of class decorators. It occurs whenever a metaclass has an implementation that uses the following pattern:: class Metaclass(type): def __new__(meta, *args, **kwds): cls = super(Metaclass, meta).__new__(meta, *args, **kwds) # Do something with cls return cls The key difference between this pattern and a class decorator is that it is automatically inherited by subclasses. However, it also comes with a major disadvantage: Python does not allow you to inherit from classes with unrelated metaclasses. Thus, the status quo requires that developers choose between the following two alternatives: * Use a class decorator, meaning that behaviour is not inherited and must be requested explicitly on every subclass * Use a metaclass, meaning that behaviour is inherited, but metaclass conflicts may make integration with other libraries and frameworks more difficult than it otherwise would be If this PEP is ultimately rejected, then this is the existing design that will remain in place by default. Restoring the Python 2 metaclass hook ------------------------------------- One simple alternative would be to restore support for a Python 2 style ``metaclass`` hook in the class body. This would be checked after the class body was executed, potentially overwriting the metaclass hint provided in the class header. The main attraction of such an approach is that it would simplify porting Python 2 applications that make use of this hook (especially those that do so dynamically). However, this approach does nothing to simplify the process of adding *inherited* class definition time behaviour, nor does it interoperate cleanly with the PEP 3135 ``__class__`` and ``super()`` semantics (as with any metaclass based solution, the ``__metaclass__`` hook would have to run before the ``__class__`` reference has been populated. Dynamic class decorators ------------------------ The original version of this PEP was called "Dynamic class decorators" and focused solely on a significantly more complicated proposal than that presented in the current version. As with the current version, it proposed that a new step be added to the class creation process, after the metaclass invocation to construct the class instance and before the application of lexical decorators. However, instead of a simple process of calling a single class method that relies on normal inheritance mechanisms, it proposed a far more complicated procedure that walked the class MRO looking for decorators stored in iterable ``__decorators__`` attributes. Using the current version of the PEP, the scheme originally proposed could be implemented as:: class DynamicDecorators(Base): @classmethod def __init_class__(cls): # Process any classes later in the MRO try: mro_chain = super().__init_class__ except AttributeError: pass else: mro_chain() # Process any __decorators__ attributes in the MRO for entry in reversed(cls.mro()): decorators = entry.__dict__.get("__decorators__", ()) for deco in reversed(decorators): cls = deco(cls) Any subclasses of ``DynamicDecorators`` would then automatically have the contents of any ``__decorators__`` attributes processed and invoked. The mechanism in the current PEP is considered superior, as many issues to do with ordering and the same decorator being invoked multiple times just go away, as that kind of thing is taken care of through the use of an ordinary class method invocation. Automatic metaclass derivation ------------------------------ When no appropriate metaclass is found, it's theoretically possible to automatically derive a metaclass for a new type based on the metaclass hint and the metaclasses of the bases. While adding such a mechanism would reduce the risk of spurious metaclass conflicts, it would do nothing to improve integration with PEP 3135, would not help with porting Python 2 code that set ``__metaclass__`` dynamically and would not provide a more straightforward inherited mechanism for invoking additional operations after the class invocation is complete. In addition, there would still be a risk of metaclass conflicts in cases where the base metaclasses were not written with multiple inheritance in mind. In such situations, there's a chance of introducing latent defects if one or more metaclasses are not invoked correctly. Calling the new hook from ``type.__init__`` ------------------------------------------- Calling the new hook automatically from ``type.__init__``, would achieve most of the goals of this PEP. However, using that approach would mean that ``__init_class__`` implementations would be unable to call any methods that relied on the ``__class__`` reference (or used the zero-argument form of ``super()``), and could not make use of those features themselves. Reference Implementation ======================== The reference implementation has been posted to the `issue tracker`_. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _issue tracker: http://bugs.python.org/issue17044 Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 10 Feb 2013 22:32:50 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Replaces many use cases for dynamic setting of ``__metaclass__`` -----------------------------------------------------------------
For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3.
So, what are the use cases? I probably write metaclasses once a year, I wonder how much incentive there is to bring an additional complication to the already complicated class construction process. Regards Antoine.
On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 10 Feb 2013 22:32:50 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Replaces many use cases for dynamic setting of ``__metaclass__`` -----------------------------------------------------------------
For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3.
At least PEAK uses it for interface declarations, I believe Zope do something similar. However, just as I championed PEP 414 to make porting Unicode-aware projects easier, I'm championing this one because I don't consider it acceptable for Python 3 to be a less capable language than Python 2 in this regard. Having come up with what I consider to be an elegant design to restore that capability, I'm not especially interested in passing judgment on the worthiness of what PJE is doing with Python 2 metaclasses that cannot currently be replicated in Python 3.
So, what are the use cases? I probably write metaclasses once a year, I wonder how much incentive there is to bring an additional complication to the already complicated class construction process.
One of the main goals of the PEP is actually to give people even *less* reason to write custom metaclasses, as you won't need to write one any more if all you want to do is adjust the class attributes after execution of the class body completes, and automatically do the same for any subclasses. (The latter part is key - if you don't want inheritance, then class decorators fill the role just fine. That's why the design in the PEP is the way it us - __init_class__ effectively becomes the innermost class decorator, which is called even for subclasses) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 10 Feb 2013 23:17:00 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Feb 10, 2013 at 10:47 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 10 Feb 2013 22:32:50 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Replaces many use cases for dynamic setting of ``__metaclass__`` -----------------------------------------------------------------
For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_class__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3.
At least PEAK uses it for interface declarations, I believe Zope do something similar.
zope.interface has been ported to Python 3, so the annoyance can't be very blocking.
However, just as I championed PEP 414 to make porting Unicode-aware projects easier, I'm championing this one because I don't consider it acceptable for Python 3 to be a less capable language than Python 2 in this regard.
This sounds like a theoretical concern rather than a practical one.
Having come up with what I consider to be an elegant design to restore that capability, I'm not especially interested in passing judgment on the worthiness of what PJE is doing with Python 2 metaclasses that cannot currently be replicated in Python 3.
It may be locally elegant, but it still brings additional complication to the global model. Reading your proposal reminds me of the various ways of influencing the pickle protocol: http://docs.python.org/dev/library/pickle.html#pickling-class-instances Nobody can claim this is simple and easy to wrap your head around. It is a maintenance burden, and it discourages understanding of the underlying model by anyone but language experts. Regards Antoine.
On Sun, Feb 10, 2013 at 11:34 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Nobody can claim this is simple and easy to wrap your head around. It is a maintenance burden, and it discourages understanding of the underlying model by anyone but language experts.
You think anyone but language experts fully understands the metaclass mechanism now? *I* got the description of some details wrong in the PEP, and I'm probably in the top half dozen people on the planet when it comes to understanding how Python 3's class model works. Plenty of people understand decorators though, and that's all __init_class__ is: a special class decorator that is applied before any other decorators, and is automatically inherited by subclasses. It massively lowers the barrier to entry for inherited post-modification of classes. Will it mean people will only bother to understand metaclasses if they actually *need* metaclass.__prepare__ or to control the identity of the object returned by metaclass.__call__? Absolutely. However, I don't see that as any different from the fact that vastly more people understand how to use instance __init__ methods correctly than understand the ins and outs of using __new__ to control the *creation* of the object, rather than merely initialising it after it is already created. In some respects, cleanly separating out the two steps of controlling initialisation and controlling creation can actually make the two easier to grasp (since you can tackle them individually as distinct concepts, rather than having to wrap your head around both of them at the same time). As far as the maintenance burden goes, the patch to enable PEP 422 for types.new_class() is trivial: - return meta(name, bases, ns, **kwds) + cls = meta(name, bases, ns, **kwds) + try: + initcl = cls.__init_class__ + except AttributeError: + pass + else: + initcl() + return cls The change in the builtin.__build_class__ implementation is similarly trivial (and has the same semantics), it's just more verbose due to its being written in C. The documentation changes are quite straightforward (moving __init_class__ and decorator invocation out to their own subsection), and the tests Daniel has provided are extensive. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
As far as the maintenance burden goes, the patch to enable PEP 422 for types.new_class() is trivial:
- return meta(name, bases, ns, **kwds) + cls = meta(name, bases, ns, **kwds) + try: + initcl = cls.__init_class__ + except AttributeError: + pass + else: + initcl() + return cls
I didn't know types.new_class(). I suppose that's because I'm not part of the top half dozen people on the planet :-) I've stopped trying to understand when I saw about the __build_class__ additions and whatnot. I now consider that part of the language totally black magic. I'm not sure it's a good thing when even some maintainers of the language don't understand some of its (important) corners.
The change in the builtin.__build_class__ implementation is similarly trivial (and has the same semantics), it's just more verbose due to its being written in C.
Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it. Regards Antoine.
Antoine Pitrou, 10.02.2013 15:28:
On Mon, 11 Feb 2013 00:09:55 +1000 Nick Coghlan wrote:
As far as the maintenance burden goes, the patch to enable PEP 422 for types.new_class() is trivial:
- return meta(name, bases, ns, **kwds) + cls = meta(name, bases, ns, **kwds) + try: + initcl = cls.__init_class__ + except AttributeError: + pass + else: + initcl() + return cls
I didn't know types.new_class(). I suppose that's because I'm not part of the top half dozen people on the planet :-) I've stopped trying to understand when I saw about the __build_class__ additions and whatnot. I now consider that part of the language totally black magic.
I'm not sure it's a good thing when even some maintainers of the language don't understand some of its (important) corners.
I, for one, only learned most of this stuff when we implemented it for Cython and I'm sure there are lots of little details that I'm not aware of. Use cases are too rare to fully learn it and the matter is certainly complex.
The change in the builtin.__build_class__ implementation is similarly trivial (and has the same semantics), it's just more verbose due to its being written in C.
Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it.
FWIW, I agree with Nick that this actually kills some use cases of metaclasses and could thus lower the barrier for "doing stuff to classes" for those who don't want to dig into metaclasses for one reason or another. However, it's hard to say if this "new way of doing it" doesn't come with its own can of worms. For example, would cooperative calls to "__init_class__" work if a superclass already defines it? Do implementors have to remember that? And is it clear how this should be done, e.g. what should normally go first, my own code or the superclass call? Supporting cooperative __init_class__() calls properly might actually be a good thing. Currently, there's only one metaclass, plus a sequence of decorators, which makes the setup rather static and sometimes tedious for subclasses that need to do certain things by themselves, but in addition to what happens already. So, the way to explain it to users would be 1) don't use it, 2) if you really need to do something to a class, use a decorator, 3) if you need to decide dynamically what to do, define __init_class__() and 4) don't forget to call super's __init_class__() in that case, and 5) only if you need to do something substantially more involved and know what you're doing, use a metaclass. Sounds a bit smoother than the current metaclass hole that we throw people into directly when a decorator won't catch it for them. It's usually not a good idea to add complexity for something that's essentially a rare use case. Only the assumption that it might actually make it easier for users to handle certain other use cases where they'd currently end up with their own home-grown metaclass is a point that makes it worth considering for me. But then again, maybe these use cases are really rare enough to just let people figure out how to do it with metaclasses, or even a combination of metaclasses and decorators. Stefan
On Sun, Feb 10, 2013 at 5:48 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
However, it's hard to say if this "new way of doing it" doesn't come with its own can of worms. For example, would cooperative calls to "__init_class__" work if a superclass already defines it? Do implementors have to remember that? And is it clear how this should be done, e.g. what should normally go first, my own code or the superclass call? Supporting cooperative __init_class__() calls properly might actually be a good thing. Currently, there's only one metaclass, plus a sequence of decorators, which makes the setup rather static and sometimes tedious for subclasses that need to do certain things by themselves, but in addition to what happens already.
Cooperative multiple inheritance of __init_class__ would work exactly the same way as it works e.g., for __init__ of any other method (actually it is easier, since as Nick mentioned, the signature is always the same): __init_class__ can simply use the zero argument form of super. There is a simple example in the tests at http://bugs.python.org/issue17044. Daniel
On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
So, the way to explain it to users would be 1) don't use it, 2) if you really need to do something to a class, use a decorator, 3) if you need to decide dynamically what to do, define __init_class__() and 4) don't forget to call super's __init_class__() in that case, and 5) only if you need to do something substantially more involved and know what you're doing, use a metaclass.
I'd revise that to: 1) if there's no harm in forgetting to decorate a subclass, use a class decorator 2) if you want to ensure that a modification is applied to every subclass of a single common base class, define __init_class__ (and always call its super) 3) If you need to make the class object *act* differently (not just initialize it or trigger some other side-effect at creation time), or if you want the class suite to return some other kind of object, you'll need a metaclass. Essentially, this change fixes a hole in class decorators that doesn't exist with function decorators: if you need the decoration applied to subclasses, you can end up with silent failures right now. Conversely, if you try prevent such failures using a metaclass, you not only have a big hill to climb, but the resulting code will be vulnerable to metaclass conflicts. The proposed solution neatly fixes both of these problems, providing One Obvious Way to do subclass initialization. (An alternative, I suppose, would be to let you do something like @@someClassDecorator to have inheritable class decorators, but as Nick has pointed out, implementing inheritable decorators is a lot more complicated.)
On Mon, Feb 11, 2013 at 7:41 AM, PJ Eby <pje@telecommunity.com> wrote:
On Sun, Feb 10, 2013 at 11:48 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
So, the way to explain it to users would be 1) don't use it, 2) if you really need to do something to a class, use a decorator, 3) if you need to decide dynamically what to do, define __init_class__() and 4) don't forget to call super's __init_class__() in that case, and 5) only if you need to do something substantially more involved and know what you're doing, use a metaclass.
I'd revise that to:
1) if there's no harm in forgetting to decorate a subclass, use a class decorator 2) if you want to ensure that a modification is applied to every subclass of a single common base class, define __init_class__ (and always call its super) 3) If you need to make the class object *act* differently (not just initialize it or trigger some other side-effect at creation time), or if you want the class suite to return some other kind of object, you'll need a metaclass.
I like that. Perhaps the PEP should propose some additional guidance in PEP 8 regarding class based metaprogramming?
Essentially, this change fixes a hole in class decorators that doesn't exist with function decorators: if you need the decoration applied to subclasses, you can end up with silent failures right now. Conversely, if you try prevent such failures using a metaclass, you not only have a big hill to climb, but the resulting code will be vulnerable to metaclass conflicts.
The proposed solution neatly fixes both of these problems, providing One Obvious Way to do subclass initialization.
I also realised last night that one significant benefit of cleanly separating class creation from class initialisation (as __new__ and __init__ separate instance creation and initialisation) is the ability to create a shared metaclass that just changes the namespace type with __prepare__, and then use __init_class__ to control what you do with it. Here's the more extended example I'm now considering adding to the PEP in order to show the improved composability the PEP offers (writing the below example with only metaclasses would be... challenging). It's still a toy example, but I don't believe there is any non-toy use case for metaclass composition that is going to be short enough to fit in a PEP: # Define a metaclass as in Python 3.3 and earlier import collections class OrderedMeta(type): def __prepare__(self, *args, **kwds): return collections.OrderedDict() # Won't be needed if we add a noop __init_class__ to type def __init_class__(cls): pass class OrderedClass(metaclass=OrderedMeta): pass # Easily supplement the metaclass behaviour in a class definition class SimpleRecord(OrderedClass): """Simple ordered record type (inheritance not supported)""" @classmethod def __init_class__(cls): super().__init_class__() cls.__fields = fields = [] for attr, obj in cls.__dict__.items(): if attr.startswith("_") or callable(obj): continue fields.append(attr) def __init__(self, *values): super().__init__(*values) for attr, obj in zip(self.__fields, values): setattr(self, attr, obj) def to_dict(self): fields = ((k, getattr(self, k)) for k in self.__fields) return collections.OrderedDict(fields) # Supplement the metaclass differently in another class definition class InheritableRecord(OrderedClass): """More complex record type that supports inheritance""" @classmethod def __init_class__(cls): super().__init_class__() cls.__fields = fields = [] for mro_cls in cls.mro(): for attr, obj in cls.__dict__.items(): if attr.startswith("_") or callable(obj): continue fields.append(attr) def __init__(self, *values): super().__init__(*values) for attr, obj in zip(self.__fields, values): setattr(self, attr, obj) def to_dict(self): fields = ((k, getattr(self, k)) for k in self.__fields) return collections.OrderedDict(fields) # Compared to custom metaclasses, composition is much simpler class ConfusedRecord(InheritableRecord, SimpleRecord): """Odd record type, only included to demonstrate composition""" # to_dict is inherited from InheritableRecord def to_simple_dict(self): return SimpleRecord.to_dict(self) Perhaps it would sweeten the deal if the PEP also provided types.OrderedMeta and types.OrderedClass, such that inheriting from types.OrderedClass and defining __init_class__ became the one-obvious-way to do order dependent class bodies? (I checked, we can make types depend on collections without a circular dependency. We would need to fix the increasingly inaccurate docstring, though) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Feb 11, 2013, at 08:33 PM, Nick Coghlan wrote:
I like that. Perhaps the PEP should propose some additional guidance in PEP 8 regarding class based metaprogramming?
I wouldn't put it in PEP 8, since it'll glaze the eyes of all but 6 people on the planet. Probably better as a HOWTO in the documentation. Cheers, -Barry
On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote:
Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it.
Why Antoine, that surely isn't the case with the import system! <sarcastic wink>. -Barry
Le Mon, 11 Feb 2013 10:15:36 -0500, Barry Warsaw <barry@python.org> a écrit :
On Feb 10, 2013, at 03:28 PM, Antoine Pitrou wrote:
Sure, every little addition is "trivial". At the end you have a scary monster made of many little trivial additions along the years, and everyone has to take care not to break it.
Why Antoine, that surely isn't the case with the import system! <sarcastic wink>.
Nor with pythonrun.c. I wouldn't even mention typeobject.c or the build system ;-) Regards Antoine.
Hi Nick, I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need. I only have a few nits/questions/suggestions. - With PJE, I think __init_class__ should automatically be a class method. The same way that __new__ is automatically a class method. - Would it make any sense to require that __init_class__ *returns* the new class object (to complete the similarity with class decorators)? Frankly, I lost track of things when you were discussing the exact point in time where __class__ is added to the class. (This may place me outside the set of people who fully understand Python's metaclasses. So be it. :-) Perhaps some explanation of the existing mechanism would be in place? It would be nice if you explained reason for why __class__ cannot be set earlier in the game (otherwise the response to this argument would be "so fix it", but somehow I doubt that is possible). I think it may also be unclear at all times whether __class__, when referenced, is meant to refer to the metaclass or to the class under construction. - I wouldn't add the OrderedMeta/OrderedClass to the PEP. Keep it focused on enabling. - Adding more examples would be great. -- --Guido van Rossum (python.org/~guido)
On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido@python.org> wrote:
Hi Nick,
I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need.
I only have a few nits/questions/suggestions.
- With PJE, I think __init_class__ should automatically be a class method.
Actually, I didn't say that as such, because I'm not sure how the heck we'd implement that. ;-) For example, at what point is it converted to a classmethod? Is it going to be a slot method with special C-level handling? Handled by the compiler? What happens if somebody makes it a
The same way that __new__ is automatically a class method.
Actually, isn't it automatically a staticmethod? Oh crap. Now that I'm thinking about it, doesn't this *have* to be a static method, explicitly passing in the class? I mean, otherwise, won't calling super().__init_class__() invoke it on the base class, rather than the current class? ISTM that EIBTI argues for the __new__/staticmethod approach, especially if you're returning the class (per below)
- Would it make any sense to require that __init_class__ *returns* the new class object (to complete the similarity with class decorators)?
It would certainly be quite useful to do so, but in that case, perhaps the method should be named __decorate_class__? And in that event the standard usage would look like: def __decorate_class__(cls): cls = super().__decorate_class__(cls) # do stuff return cls On the other hand, one could just drop the super() requirement and make the usage even simpler by having the class machinery walk the MRO and pass each method the result of invoking the previous one. Then the methods are short and sweet, and super() and __class__ don't come into it. (Though I guess the class machinery could keep setting __class__ to whatever the last-returned class was.) In his first draft, Nick implemented inheritable decorators instead, using a __decorators__ attribute in the class body, or something like that. While that approach had an issue or two of its own, it's possible that just going with a single __decorate_class__ method would work out better.
On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje@telecommunity.com> wrote:
On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido@python.org> wrote:
Hi Nick,
I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need.
I only have a few nits/questions/suggestions.
- With PJE, I think __init_class__ should automatically be a class method.
Actually, I didn't say that as such, because I'm not sure how the heck we'd implement that. ;-)
For example, at what point is it converted to a classmethod? Is it going to be a slot method with special C-level handling? Handled by the compiler? What happens if somebody makes it a
The same way that __new__ is automatically a class method.
Actually, isn't it automatically a staticmethod? Oh crap. Now that I'm thinking about it, doesn't this *have* to be a static method, explicitly passing in the class? I mean, otherwise, won't calling super().__init_class__() invoke it on the base class, rather than the current class?
ISTM that EIBTI argues for the __new__/staticmethod approach, especially if you're returning the class (per below)
Let's see what Nick and the implementer say.
- Would it make any sense to require that __init_class__ *returns* the new class object (to complete the similarity with class decorators)?
It would certainly be quite useful to do so, but in that case, perhaps the method should be named __decorate_class__? And in that event the standard usage would look like:
def __decorate_class__(cls): cls = super().__decorate_class__(cls) # do stuff return cls
On the other hand, one could just drop the super() requirement and make the usage even simpler by having the class machinery walk the MRO and pass each method the result of invoking the previous one. Then the methods are short and sweet, and super() and __class__ don't come into it. (Though I guess the class machinery could keep setting __class__ to whatever the last-returned class was.)
In his first draft, Nick implemented inheritable decorators instead, using a __decorators__ attribute in the class body, or something like that. While that approach had an issue or two of its own, it's possible that just going with a single __decorate_class__ method would work out better.
Half-baked idea: Maybe the base class __decorate_class__ would call the class decorators? Or does that not make sense? -- --Guido van Rossum (python.org/~guido)
On 12 Feb 2013 07:44, "Guido van Rossum" <guido@python.org> wrote:
On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje@telecommunity.com> wrote:
On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido@python.org>
wrote:
Hi Nick,
I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need.
I only have a few nits/questions/suggestions.
- With PJE, I think __init_class__ should automatically be a class method.
Actually, I didn't say that as such, because I'm not sure how the heck we'd implement that. ;-)
For example, at what point is it converted to a classmethod? Is it going to be a slot method with special C-level handling? Handled by the compiler? What happens if somebody makes it a
The same way that __new__ is automatically a class method.
Actually, isn't it automatically a staticmethod? Oh crap. Now that I'm thinking about it, doesn't this *have* to be a static method, explicitly passing in the class? I mean, otherwise, won't calling super().__init_class__() invoke it on the base class, rather than the current class?
ISTM that EIBTI argues for the __new__/staticmethod approach, especially if you're returning the class (per below)
Let's see what Nick and the implementer say.
I think these are some interesting ideas and it's going to take me a while to digest them and update the PEP :) A few random thoughts: 1. I like the idea of a metaprogramming "howto" that provides advice on choosing a suitable level of metaprogramming (with the default choice being "use existing decorators", then escalating through creating custom decorators all the way to creating custom metaclasses). I don't think the PEP needs to be conditional on writing that, but I will at least add PJE's list to the PEP itself. 2. I see the new method as more analogous to__init__ than to__new__, so the __decorate_class__ idea makes me nervous, as it's closer to a __new__ method. Composition gets a *lot* harder when your parent class can switch out the object on you. 3. I'm trying to avoid any custom magic specific to this method, but making it implicitly a static or class method is fairly easy if we so choose - the standard retrieval code during class creation can just bypass the descriptor machinery, and wrap it in staticmethod or classmethod if it isn't already. Given that __new__ is already implicitly static, it may be easier to follow that precedent here rather than trying to explain why an explicit @classmethod is needed in one case but not the other. 4.__class__ is already bound as soon as we have a class object to bind it to, so we can't move it any earlier. However, it's already early enough to let references to it from the new method (including the implied one in zero-arg super) work properly. The black magic that is zero-arg super also deals with PJE's concern about propagating the actual class up the MRO (as it is equivalent to "super(__class__, first_argument)"). 5. Implicitly walking the MRO bothers me, as it becomes a special case for people to learn. We don't do that for __init__ or __new__, so I don't think it makes sense to do it here. We can include a recommended structure in the docs, where the first step is calling the parent through. As PJE suggested, a no-op method on type will make that simple and fairly safe (people using a metaclass hierarchy not anchored on type can figure out their own equivalent) Cheers, Nick.
- Would it make any sense to require that __init_class__ *returns* the new class object (to complete the similarity with class decorators)?
It would certainly be quite useful to do so, but in that case, perhaps the method should be named __decorate_class__? And in that event the standard usage would look like:
def __decorate_class__(cls): cls = super().__decorate_class__(cls) # do stuff return cls
On the other hand, one could just drop the super() requirement and make the usage even simpler by having the class machinery walk the MRO and pass each method the result of invoking the previous one. Then the methods are short and sweet, and super() and __class__ don't come into it. (Though I guess the class machinery could keep setting __class__ to whatever the last-returned class was.)
In his first draft, Nick implemented inheritable decorators instead, using a __decorators__ attribute in the class body, or something like that. While that approach had an issue or two of its own, it's possible that just going with a single __decorate_class__ method would work out better.
Half-baked idea: Maybe the base class __decorate_class__ would call the
class decorators? Or does that not make sense?
-- --Guido van Rossum (python.org/~guido)
On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 12 Feb 2013 07:44, "Guido van Rossum" <guido@python.org> wrote:
On Mon, Feb 11, 2013 at 12:57 PM, PJ Eby <pje@telecommunity.com> wrote:
On Mon, Feb 11, 2013 at 12:44 PM, Guido van Rossum <guido@python.org>
wrote:
Hi Nick,
I think this will make a fine addition to the language. I agree that it is superior to the alternatives and fulfills a real (if rare) need.
I only have a few nits/questions/suggestions.
- With PJE, I think __init_class__ should automatically be a class method.
Actually, I didn't say that as such, because I'm not sure how the heck we'd implement that. ;-)
For example, at what point is it converted to a classmethod? Is it going to be a slot method with special C-level handling? Handled by the compiler? What happens if somebody makes it a
The same way that __new__ is automatically a class method.
Actually, isn't it automatically a staticmethod? Oh crap. Now that I'm thinking about it, doesn't this *have* to be a static method, explicitly passing in the class? I mean, otherwise, won't calling super().__init_class__() invoke it on the base class, rather than the current class?
ISTM that EIBTI argues for the __new__/staticmethod approach, especially if you're returning the class (per below)
Let's see what Nick and the implementer say.
I think these are some interesting ideas and it's going to take me a while to digest them and update the PEP :)
Heh. :-)
A few random thoughts:
1. I like the idea of a metaprogramming "howto" that provides advice on choosing a suitable level of metaprogramming (with the default choice being "use existing decorators", then escalating through creating custom decorators all the way to creating custom metaclasses). I don't think the PEP needs to be conditional on writing that, but I will at least add PJE's list to the PEP itself.
Sure.
2. I see the new method as more analogous to__init__ than to__new__, so the __decorate_class__ idea makes me nervous, as it's closer to a __new__ method. Composition gets a *lot* harder when your parent class can switch out the object on you.
Fair enough.
3. I'm trying to avoid any custom magic specific to this method, but making it implicitly a static or class method is fairly easy if we so choose - the standard retrieval code during class creation can just bypass the descriptor machinery, and wrap it in staticmethod or classmethod if it isn't already. Given that __new__ is already implicitly static, it may be easier to follow that precedent here rather than trying to explain why an explicit @classmethod is needed in one case but not the other.
Also fair enough.
4.__class__ is already bound as soon as we have a class object to bind it to, so we can't move it any earlier. However, it's already early enough to let references to it from the new method (including the implied one in zero-arg super) work properly. The black magic that is zero-arg super also deals with PJE's concern about propagating the actual class up the MRO (as it is equivalent to "super(__class__, first_argument)").
So where is it not bound when a metaclass runs? I guess in the metaclass's __new__()? Because in the metaclass's __init__() it should exist IIUC.
5. Implicitly walking the MRO bothers me, as it becomes a special case for people to learn. We don't do that for __init__ or __new__, so I don't think it makes sense to do it here. We can include a recommended structure in the docs, where the first step is calling the parent through. As PJE suggested, a no-op method on type will make that simple and fairly safe (people using a metaclass hierarchy not anchored on type can figure out their own equivalent)
Agreed. -- --Guido van Rossum (python.org/~guido)
On Tue, Feb 12, 2013 at 8:35 AM, Guido van Rossum <guido@python.org> wrote:
On Mon, Feb 11, 2013 at 2:29 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
4.__class__ is already bound as soon as we have a class object to bind it to, so we can't move it any earlier. However, it's already early enough to let references to it from the new method (including the implied one in zero-arg super) work properly. The black magic that is zero-arg super also deals with PJE's concern about propagating the actual class up the MRO (as it is equivalent to "super(__class__, first_argument)").
So where is it not bound when a metaclass runs? I guess in the metaclass's __new__()? Because in the metaclass's __init__() it should exist IIUC.
I stand corrected, we don't *quite* bind it as soon as the class object exists - we bind it as soon as the call to the metaclass returns. So even though the class object exists when __init__ runs, it isn't bound to __class__ yet. Given that the class construction machinery treats the metaclass constructor as an opaque callable, trying to get the cell binding to happen between the call to __new__ and the one to __init__ would be rather challenging (Translation: I'm not sure where you would even start...). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
11.02.2013 23:29, Nick Coghlan wrote:
3. I'm trying to avoid any custom magic specific to this method, but making it implicitly a static or class method is fairly easy if we so choose - the standard retrieval code during class creation can just bypass the descriptor machinery, and wrap it in staticmethod or classmethod if it isn't already. Given that __new__ is already implicitly static, it may be easier to follow that precedent here rather than trying to explain why an explicit @classmethod is needed in one case but not the other.
Though __new__ is implicitly a *static* rather than a *class* method (so we can use it e.g. by calling object.__new__(MyClass), though -- besides -- in Py3k unbound methods have gone so the difference between static and non-static-and-non-class-methods is smaller than in Py2.x), in case of __init_class__ + super() it'd have to be called: super().__init_class__(__class__) ...and it seems to me a bit awkward. And making it implicitly a *class* rather than a *static* method whould make *impossible* to do calls such as: ExplicitAncestor.__init_class__(ExplicitDescendant) ...though I'm not sure we'd ever need such a call. If not -- implicit *class* method may be a good idea, but if we would? *** On the margin: is that middle underscore in '__init_class__' necessary? We had __metaclass__, not __meta_class__... OK, it's one world, but still we also have __getattr__, __getattribute__, __getitem__, __instancecheck__, __qualname__, __truediv__ etc. (not __get_attr__, __instance_check__ etc.). [I remember only one exception: __reduce_ex__, rather rarely used, and easy to defend against weird __reduceex__]. Wouldn't __initclass__ be readable enough? IMHO it could spare users trouble with remembering special case. Cheers. *j
On 12/02/13 10:56, Jan Kaliszewski wrote:
Wouldn't __initclass__ be readable enough? IMHO it could spare users trouble with remembering special case.
+1 I approve of the colour of this bikeshed. __init_class__ has too many underscores. -- Steven
On Feb 10, 2013, at 02:34 PM, Antoine Pitrou wrote:
zope.interface has been ported to Python 3, so the annoyance can't be very blocking.
The syntax is different, but I actually prefer the Python 3-compatible syntax better. It uses a class decorator instead of a magic class attribute, so it's much clearer IMHO. Works in Python 2, too, fsvo. -Barry
On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass.
Couldn't one just write a metaclass that calls __init_class__ for you? Schiavo Simon
On Sun, Feb 10, 2013 at 11:33 PM, Simon Cross <hodgestar+pythondev@gmail.com> wrote:
On Sun, Feb 10, 2013 at 2:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For those that don't recall the original discussion, the proposal is to add a new __init_class__ hook, invoked after the class object is created, but before the class decorators are applied. This provides a simple approach to inherited post-creation modification of classes, without the need for a custom metaclass.
Couldn't one just write a metaclass that calls __init_class__ for you?
And, indeed, that's what Zope did more than 10 years ago. The problem is that you're now locked in to that metaclass - metaclass conflicts become a permanent risk. You also can't *add* a metaclass to public classes, the risk of introducing metaclass conflicts means that such a change is always backwards incompatible. Adding __init_class__ is fine though, because that's no higher risk than adding a __new__ or __init__ method (it's actually lower risk, since the signature is always consistent). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
class Example: @classmethod def __init_class__(cls):
Is the @classmethod required? What happens if it's not present? Second, will type have a default __init_class__? (IMO, it should, otherwise it will be impossible to write co-operative __init_class__ functions.) Only other comment is that the PEP could use a more concrete use case, e.g.: class Record: __fields = {} @classmethod def __init_class__(cls): cls.__fields = dict(cls.__fields) # inherited fields cls.__fields.update({attr:val for attr, val in cls.__dict__.iteritems() if isinstance(val, Field)}) super().__init_class__() # be co-operative # ...other methods that use the __fields attribute class SomeRecord(Record): foo = Field(int) bar = Field(str) Putting something like this early on might help to demonstrate the usefulness of the feature on its own merits, independent of the porting issue, etc. ;-)
On Sun, 10 Feb 2013 16:27:49 -0500 PJ Eby <pje@telecommunity.com> wrote:
On Sun, Feb 10, 2013 at 7:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
class Example: @classmethod def __init_class__(cls):
Is the @classmethod required? What happens if it's not present?
Second, will type have a default __init_class__? (IMO, it should, otherwise it will be impossible to write co-operative __init_class__ functions.)
Only other comment is that the PEP could use a more concrete use case, e.g.:
class Record: __fields = {}
@classmethod def __init_class__(cls): cls.__fields = dict(cls.__fields) # inherited fields cls.__fields.update({attr:val for attr, val in cls.__dict__.iteritems() if isinstance(val, Field)}) super().__init_class__() # be co-operative
# ...other methods that use the __fields attribute
class SomeRecord(Record): foo = Field(int) bar = Field(str)
Putting something like this early on might help to demonstrate the usefulness of the feature on its own merits, independent of the porting issue, etc. ;-)
Can you explain what the example does / is supposed to do?
participants (10)
-
Antoine Pitrou
-
Barry Warsaw
-
Daniel Urban
-
Guido van Rossum
-
Jan Kaliszewski
-
Nick Coghlan
-
PJ Eby
-
Simon Cross
-
Stefan Behnel
-
Steven D'Aprano