
Hello everybody, recently, I started a discussion on how metaclasses could be improved. This lead to a major discussion about PEP 422, where I discussed some possible changes to PEP 422. Nick proposed I write a competing PEP, so that the community may decide. Here it comes. I propose that a class may define a classmethod called __init_subclass__ which initialized subclasses of this class, as a lightweight version of metaclasses. Compared to PEP 422, this proposal is much simpler to implement, but I do believe that it also makes more sense. PEP 422 has a method called __autodecorate__ which acts on the class currently being generated. This is normally not what one wants: when writing a registering class, for example, one does not want to register the registering class itself. Most evident it become if one wants to introduce such a decorator as a mixin: certainly one does not want to tamper with the mixin class, but with the class it is mixed in! A lot of the text in my new PEP has been stolen from PEP 422. This is why I hesitated writing myself as the only author of my new PEP. On the other hand, I guess the original authors don't want to be bothered with my stuff, so I credited them in the body of my new PEP. Greetings Martin And here it comes, the draft for a new PEP: PEP: 422a Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann <lkb.teichmann@gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.5 Post-History: 27-Feb-2015 Replaces: 422 Abstract ======== Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``namespace`` parameter in the class header, and a new ``__init_subclass__`` hook in the class body. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Connection to other PEP ======================= This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban. It shares both most of the PEP text and proposed code, but has major differences in how to achieve its goals. Background ========== For an already created class ``cls``, the term "metaclass" has a clear meaning: it is the value of ``type(cls)``. *During* class creation, it has another meaning: it is also used to refer to the metaclass hint that may be provided as part of the class definition. While in many cases these two meanings end up referring to one and the same object, there are two situations where that is not the case: * If the metaclass hint refers to an instance of ``type``, then it is considered as a candidate metaclass along with the metaclasses of all of the parents of the class being defined. If a more appropriate metaclass is found amongst the candidates, then it will be used instead of the one given in the metaclass hint. * Otherwise, an explicit metaclass hint is assumed to be a factory function and is called directly to create the class object. In this case, the final metaclass will be determined by the factory function definition. In the typical case (where the factory functions just calls ``type``, or, in Python 3.3 or later, ``types.new_class``) the actual metaclass is then determined based on the parent classes. It is notable that only the actual metaclass is inherited - a factory function used as a metaclass hook sees only the class currently being defined, and is not invoked for any subclasses. In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` keyword syntax in the class header. This allows the ``__prepare__`` method on the metaclass to be used to create the ``locals()`` namespace used during execution of the class body (for example, specifying the use of ``collections.OrderedDict`` instead of a regular ``dict``). In Python 2, there was no ``__prepare__`` method (that API was added for Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__`` attribute, and the class creation process would extract that value from the class namespace to use as the metaclass hint. There is `published code`_ that makes use of this feature. Another new feature in Python 3 is the zero-argument form of the ``super()`` builtin, introduced by PEP 3135. This feature uses an implicit ``__class__`` reference to the class being defined to replace the "by name" references required in Python 2. Just as code invoked during execution of a Python 2 metaclass could not call methods that referenced the class by name (as the name had not yet been bound in the containing scope), similarly, Python 3 metaclasses cannot call methods that rely on the implicit ``__class__`` reference (as it is not populated until after the metaclass has returned control to the class creation machinery). Finally, when a class uses a custom metaclass, it can pose additional challenges to the use of multiple inheritance, as a new class cannot inherit from parent classes with unrelated metaclasses. This means that it is impossible to add a metaclass to an already published class: such an addition is a backwards incompatible change due to the risk of metaclass conflicts. Proposal ======== This PEP proposes that a new mechanism to customise class creation be added to Python 3.5 that meets the following criteria: 1. Integrates nicely with class inheritance structures (including mixins and multiple inheritance), 2. Integrates nicely with the implicit ``__class__`` reference and zero-argument ``super()`` syntax introduced by PEP 3135, 3. Can be added to an existing base class without a significant risk of introducing backwards compatibility problems, and 4. Restores the ability for class namespaces to have some influence on the class creation process (above and beyond populating the namespace itself), but potentially without the full flexibility of the Python 2 style ``__metaclass__`` hook. Those goals can be achieved by adding two functionalities: 1. A ``__init_subclass__`` hook that initializes all subclasses of a given class, and 2. A new keyword parameter ``namespace`` to the class creation statement, that gives an initialization of the namespace. As an example, the first proposal looks as follows:: class SpamBase: # this is implicitly a @classmethod def __init_subclass__(cls, ns, **kwargs): # This is invoked after a subclass is created, but before # explicit decorators are called. # The usual super() mechanisms are used to correctly support # multiple inheritance. # ns is the classes namespace # **kwargs are the keyword arguments to the subclasses' # class creation statement super().__init_subclass__(cls, ns, **kwargs) class Spam(SpamBase): pass # the new hook is called on Spam To simplify the cooperative multiple inheritance case, ``object`` will gain a default implementation of the hook that does nothing:: class object: def __init_subclass__(cls, ns): pass Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion. The second part of the proposal is to have a ``namespace`` keyword argument to the class declaration statement. If present, its value will be called without arguments to initialize a subclasses namespace, very much like a metaclass ``__prepare__`` method would do. In addition, the introduction of the metaclass ``__prepare__`` method in PEP 3115 allows a further enhancement that was not possible in Python 2: this PEP also proposes that ``type.__prepare__`` be updated to accept a factory function as a ``namespace`` keyword-only argument. If present, the value provided as the ``namespace`` argument will be called without arguments to create the result of ``type.__prepare__`` instead of using a freshly created dictionary instance. For example, the following will use an ordered dictionary as the class namespace:: class OrderedBase(namespace=collections.OrderedDict): pass class Ordered(OrderedBase): # cls.__dict__ is still a read-only proxy to the class namespace, # but the underlying storage is an OrderedDict instance .. note:: This PEP, along with the existing ability to use __prepare__ to share a single namespace amongst multiple class objects, highlights a possible issue with the attribute lookup caching: when the underlying mapping is updated by other means, the attribute lookup cache is not invalidated correctly (this is a key part of the reason class ``__dict__`` attributes produce a read-only view of the underlying storage). Since the optimisation provided by that cache is highly desirable, the use of a preexisting namespace as the class namespace may need to be declared as officially unsupported (since the observed behaviour is rather strange when the caches get out of sync). Key Benefits ============ Easier use of custom namespaces for a class ------------------------------------------- Currently, to use a different type (such as ``collections.OrderedDict``) for a class namespace, or to use a pre-populated namespace, it is necessary to write and use a custom metaclass. With this PEP, using a custom namespace becomes as simple as specifying an appropriate factory function in the class header. Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. Understanding the proposed implicit class initialization hook only requires ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_subclass__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. Integrates cleanly with \PEP 3135 --------------------------------- Given that the method is called on already existing classes, the new hook will be able to freely invoke class methods that rely on the implicit ``__class__`` reference introduced by PEP 3135, including methods that use the zero argument form of ``super()``. Replaces many use cases for dynamic setting of ``__metaclass__`` ---------------------------------------------------------------- For use cases that don't involve completely replacing the defined class, Python 2 code that dynamically set ``__metaclass__`` can now dynamically set ``__init_subclass__`` instead. For more advanced use cases, introduction of an explicit metaclass (possibly made available as a required base class) will still be necessary in order to support Python 3. A path of introduction into Python ================================== Most of the benefits of this PEP can already be implemented using a simple metaclass. For the ``__init_subclass__`` hook this works all the way down to python 2.7, while the namespace needs python 3.0 to work. Such a class has been `uploaded to PyPI`_. The only drawback of such a metaclass are the mentioned problems with metaclasses and multiple inheritance. Two classes using such a metaclass can only be combined, if they use exactly the same such metaclass. This fact calls for the inclusion of such a class into the standard library, let's call it ``SubclassMeta``, with a base class using it called ``SublassInit``. Once all users use this standard library metaclass, classes from different packages can easily be combined. But still such classes cannot be easily combined with other classes using other metaclasses. Authors of metaclasses should bear that in mind and inherit from the standard metaclass if it seems useful for users of the metaclass to add more functionality. Ultimately, if the need for combining with other metaclasses is strong enough, the proposed functionality may be introduced into python's ``type``. Those arguments strongly hint to the following procedure to include the proposed functionality into python: 1. The metaclass implementing this proposal is put onto PyPI, so that it can be used and scrutinized. 2. Once the code is properly mature, it can be added to the python standard library. There should be a new module called ``metaclass`` which collects tools for metaclass authors, as well as a documentation of the best practices of how to write metaclasses. 3. If the need of combining this metaclass with other metaclasses is strong enough, it may be included into python itself. New Ways of Using Classes ========================= This proposal has many usecases like the following. In the examples, we still inherit from the ``SubclassInit`` base class. This would become unnecessary once this PEP is included in Python directly. Subclass registration --------------------- Especially when writing a plugin system, one likes to register new subclasses of a plugin baseclass. This can be done as follows:: class PluginBase(SubclassInit): subclasses = [] def __init_subclass__(cls, ns, **kwargs): super().__init_subclass__(ns, **kwargs) cls.subclasses.append(cls) One should note that this also works nicely as a mixin class. Trait descriptors ----------------- There are many designs of python descriptors in the wild which, for example, check boundaries of values. Often those "traits" need some support of a metaclass to work. This is how this would look like with this PEP:: class Trait: def __get__(self, instance, owner): return instance.__dict__[self.key] def __set__(self, instance, value): instance.__dict__[self.key] = value class Int(Trait): def __set__(self, instance, value): # some boundary check code here super().__set__(instance, value) class HasTraits(SubclassInit): def __init_subclass__(cls, ns, **kwargs): super().__init_subclass__(ns, **kwargs) for k, v in ns.items(): if isinstance(v, Trait): v.key = k The new ``namespace`` keyword in the class header enables a number of interesting options for controlling the way a class is initialised, including some aspects of the object models of both Javascript and Ruby. Order preserving classes ------------------------ :: class OrderedClassBase(namespace=collections.OrderedDict): pass class OrderedClass(OrderedClassBase): a = 1 b = 2 c = 3 Prepopulated namespaces ----------------------- :: seed_data = dict(a=1, b=2, c=3) class PrepopulatedClass(namespace=seed_data.copy): pass Cloning a prototype class ------------------------- :: class NewClass(namespace=Prototype.__dict__.copy): pass Rejected Design Options ======================= Calling the hook on the class itself ------------------------------------ Adding an ``__autodecorate__`` hook that would be called on the class itself was the proposed idea of PEP 422. Most examples work the same way or even better if the hook is called on the subclass. In general, it is much easier to explicitly call the hook on the class in which it is defined (to opt-in to such a behavior) than to opt-out, meaning that one does not want the hook to be called on the class it is defined in. This becomes most evident if the class in question is designed as a mixin: it is very unlikely that the code of the mixin is to be executed for the mixin class itself, as it is not supposed to be a complete class on its own. The original proposal also made major changes in the class initialization process, rendering it impossible to back-port the proposal to older python versions. Other variants of calling the hook ---------------------------------- Other names for the hook were presented, namely ``__decorate__`` or ``__autodecorate__``. This proposal opts for ``__init_subclass__`` as it is very close to the ``__init__`` method, just for the subclass, while it is not very close to decorators, as it does not return the class. Requiring an explicit decorator on ``__init_subclass__`` -------------------------------------------------------- One could require the explicit use of ``@classmethod`` on the ``__init_subclass__`` decorator. It was made implicit since there's no sensible interpretation for leaving it out, and that case would need to be detected anyway in order to give a useful error message. This decision was reinforced after noticing that the user experience of defining ``__prepare__`` and forgetting the ``@classmethod`` method decorator is singularly incomprehensible (particularly since PEP 3115 documents it as an ordinary method, and the current documentation doesn't explicitly say anything one way or the other). Passing in the namespace directly rather than a factory function ---------------------------------------------------------------- At one point, PEP 422 proposed that the class namespace be passed directly as a keyword argument, rather than passing a factory function. However, this encourages an unsupported behaviour (that is, passing the same namespace to multiple classes, or retaining direct write access to a mapping used as a class namespace), so the API was switched to the factory function version. Possible Extensions =================== Some extensions to this PEP are imaginable, which are postponed to a later pep: * A ``__new_subclass__`` method could be defined which acts like a ``__new__`` for classes. This would be very close to ``__autodecorate__`` in PEP 422. * ``__subclasshook__`` could be made a classmethod in a class instead of a method in the metaclass. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _uploaded to PyPI: https://pypi.python.org/pypi/metaclass Copyright ========= This document has been placed in the public domain.

On 28 February 2015 at 02:56, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Aye, I like this design, it just wasn't until I saw it written out fully that it finally clicked why restricting it to subclasses let both zero-argument super() and explicit namespace access to the defining class both work. Getting the latter to also work is a definite strength of your approach over mine.
New PEP numbers can be a good idea for significant redesigns anyway, otherwise people may end up having to qualify which version they're talking about while deciding between two competing approaches. In this case, unless someone comes up with an objection I've missed, I think your __init_subclass__ design is just plain better than my __autodecorate__ idea, and as you say, if we find a pressing need for it, we could always add "__new_subclass__" later to get comparable functionality. Assuming we don't hear any such objections, I'll withdraw PEP 422 and we can focus on getting PEP 487 ready for submission to python-dev. That leaves my one concern being the "namespace" keyword parameter. I'd started reconsidering that for PEP 422, and I think the argument holds at least as strongly for your improved design: https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the... Since speed usually isn't a critical concern for class definitions, it's likely better to just make them ordered by default, which can be its own PEP, rather than being part of this one. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
FWIW, Guido already told me a while back that OrderedDict by default for class definition namespace was okay. It's just been waiting on me to get back to the C OrderedDict implementation (which I've started to recently). -eric

On 1 March 2015 at 11:30, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Yeah, that was the conversation that prompted me to question the utility of the namespace keyword idea. The prototype class and class extension examples are cute, but fall into the "don't do this in production" category for me, making them prime candidates for custom metaclasses. The one really useful one is using an ordered dictionary for namespace execution, and your solution addresses that better by just making it the normal behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

That said, I would love if it was just possible to use super() in classmethods called from the __init__ of a metaclass. Until now, the __class__ of a method is only initialized after the metaclasses __init__ has run. Fortunately, this is easily changeable: it is possible to move the initialization of __class__ into type.__new__. As the class is (typically) created there, that is about as early as possible. I put the necessary changes to github: https://github.com/tecki/cpython/commits/cell
I feel the same way as you, so I am happy to drop the namespace idea from PEP 487, but I agree that then ordered namespaces would simply be great, but reading Eric's comment, that seems just around the corner. Greetings Martin

On 1 Mar 2015 19:28, "Martin Teichmann" <lkb.teichmann@gmail.com> wrote:
Unfortunately, it's too early - we can't control whether or not subclasses actually return the result of type.__new__ from their own __new__ implementation, so that object could have been replaced by something else by the time __init__ gets called. However, it may be feasible to move it into type.__init__. That way class methods would start working properly as soon as the metaclass __init__ implementation had called up into type.__init__. Regards, Nick.

Well, this is actually just a question of definition and documentation. Sure, a metaclasses __new__ may return whatever it pleases, but this is not in contradiction of setting __class__ of the methods in type.__new__. There are two possible scenarios: 1. metaclass.__new__ does not call type.__new__. Once we document that it is type.__new__'s job to set __class__, nobody can complain if __class__ is not set if type.__new__ is not called, as this would be documented behavior. In my changes I actually still left in the old code, so __build_class__ would still try to set __class__ even if type.__new__ did not get called. 2. somehow type.__new__ does get called (most likely by metaclass.__new__). It will do its job and set the __class__ of the methods of the class being created. Whatever metaclass.__new__ returns then doesn't matter much, because what sense would it make to set __class__ of that object it creates? To give an example: Imagine a soliton-generating metaclass: class Soliton(type): def __new__(cls, name, bases, ns): self = super().__new__(name, bases, ns) return self() And generate such a soliton: class A(metaclass=Soliton): def f(self): print(__class__) As of now, writing "A.f()" interestingly prints "<__main__.A object>", so __class__ is indeed set to what Soliton.__new__ returns, the object, not the class. This is currently correct behavior, but I think it actually is not what one expects, nor what one desires. (Does anyone out there make use of such a construct? Please speak up!) super() certainly won't work. So I think it would actually be a change for the better to let type.__new__ set the __class__ of the generated class. I know that it technically breaks backward compatibility. But I ran all the tests and none failed, so apparently until now nobody got weird ideas like me... I even tried to remove the old code from __build_class__ and still all tests run. (just a note, if anyone else here is trying to do so, I tampered with the compiler, so you better delete your .pyc files before running the tests) Greetings Martin

On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote:
You have to pass cls as an explicit argument to __new__, otherwise you get a TypeError: TypeError: type.__new__(X): X is not a type object (str) (that's in Python 3.3). The line should be: self = super().__new__(cls, name, bases, ns)
Is "soliton" the usual terminology for this? Do you perhaps mean singleton? I've googled for "soliton" but nothing relevant is coming up.
I do now! Seriously, I think you have just solved a major problem for me. I'll need to do some further experimentation, but that is almost exactly what I have been looking for. I've been looking for a way to have a class statement return a custom instance and I think this might just do it. -- Steve

On 2 March 2015 at 21:46, Steven D'Aprano <steve@pearwood.info> wrote:
Martin's question wasn't about using __new__ on the metaclass to do factory function magic (that's "normal usage" as far as metaclasses go). The "construct" in question here is: 1. Returning something from a metaclass __new__ implementation other than the result of type.__new__; AND 2. Relying on __class__ to refer to the returned object, rather than the result of type.__new__ That seems likely to be a vanishingly small subset of Python usage, and even if it does happen, the subset that couldn't switch to using explicit super() via the bound name is likely to be even smaller. That seems like a risk well worth taking when the pay-off is having class methods that use implicit super() just work when called from the metaclass. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2 March 2015 at 20:57, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Thinking about this a bit more, I think a good way to look at it is to consider the impact of: 1. Using explicit super() rather than implicit super() 2. Using explicit decorators Currently, in your example, "__class__" and "A" will both refer to an *instance* of the class. If, instead, you used a "singleton" explicit decorator, then "__class__ " would refer to the actual class object, while "A" would refer to the instance. With your proposed change, then "__class__" will always refer to the actual class object, even if the metaclass __new__ plays implicit decoration games. It also means that class methods that rely on __class__ (whether explicitly or implicitly through super()) will "just work" in metaclass __init__ and __new__ methods, although only after the call up to the base class implementation in the latter case.
Aye, and the discrepancy I was concerned about can already happen when using explicit decorators. That suggest to me that even if someone *was* relying on __class__ pointing to the same object as the bound name when a metaclass does something odd, they can likely switch to referring to it by name instead. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 March 2015 at 03:28, Ethan Furman <ethan@stoneleaf.us> wrote:
No, it's printing out the type referenced from the definition time cell*, rather than the type of the object actually passed in to the method. Cheers, Nick. P.S. *For anyone that isn't already aware, the general gist of how that works: * The compiler actually special cases "super()" and "__class__" in method bodies * If it sees either of them, it implicitly adds a "__class__" cell reference to the function * In the "super()" case, the call is implicitly rewritten as "super(<first param>, __class__)" * __class__ is later filled in with a reference to the class being defined, once that actually exists In my view, that's never going to win any awards for "elegance in language design", but it makes super() so much easier to use I'm happy to tell my sense of aesthetics to be quiet :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 7 March 2015 at 00:11, Mark Young <marky1991@gmail.com> wrote:
Is that documented somewhere? I have never heard of that functionality.
The implicit __class__ cell? PEP 3135 introduced the ability to call super() without any arguments in Python 3 to ask the interpreter to just "do the right thing": https://www.python.org/dev/peps/pep-3135/ The actual language reference docs are here: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object Cheers, Nick. P.S. Those language reference class creation docs weren't fully updated for the original Python 3.0 release. They were updated in 3.3 when the new dynamic type creation APIs were added to the types module to simplify PEP 3115 compliant dynamic type creation: https://docs.python.org/3/library/types.html#dynamic-type-creation -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9 Mar 2015 03:18, "Mark Young" <marky1991@gmail.com> wrote:
Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess
not. Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) Cheers, Nick.

On 9 Mar 2015 10:12, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 9 Mar 2015 03:18, "Mark Young" <marky1991@gmail.com> wrote:
Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I
guess not.
Every other reference on that page is to the class attribute rather than
the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) With the benefit of hindsight, if I had the chance to do this over, I'd suggest we call the implicit method nonlocal "__classdef__" instead, specifically to avoid the name collision with the runtime type information on object instances. My general impression is that the current name triggers too many incorrect assumptions for most readers, and the name collision makes it harder than it needs to be to build a more correct mental model. It would actually be possible to still make that transition, with a view to deprecating the current "__class__" spelling at some point in the distant future. I'm too close to the problem to judge whether or not it would actually help reduce confusion, though. Regards, Nick.
Cheers, Nick.

On Mar 8, 2015, at 5:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I can tell you that every single time I've mentioned the __class__ nonlocal in a StackOverflow answer, someone mistakenly "corrected" me to tell me either "you need to write self.__class__" or "that won't work because it's the runtime type of self, which could be a subclass". I think a better solution, if you were doing it from scratch, might be to get rid of the __class__ member and force people to use type(self) to get it, so TOOTDI. (Of course that implies that code that needs to change __class__ would have to call some function to do so, but I'm not sure that's a bad thing. Dynamically changing your type is a rare thing to do in Python, and unusual or impossible in most other OO languages, except maybe during initialization in the Smalltalk family, so why shouldn't it be signaled by something scary?) On the other hand, is __class__ the only thing that's spelled with double underscored that isn't an attribute? Maybe that's the part that's confusing people...

On 9 March 2015 at 22:40, Andrew Barnert <abarnert@yahoo.com> wrote:
They're not substitutes for each other, as __class__ lets you lie about your type to the consumer, while type() doesn't, and this is by design:
For many use cases, you actually want obj.__class__ in order to handle proxy objects correctly.
It's not (just) about being able to dynamically change your type, it's about the fact that __class__ goes through the descriptor and class attribute lookup machinery, while type() doesn't.
It's not the only one - implicit module attributes like __name__ work that way, as do the __debug__ and __import__ builtins. As far as I can tell, it's specifically the fact that "__class__" looks like a plausible typo for "self.__class__" that causes problems, as this also creates the problem that if you search for "__class__" not only on the data model page, but anywhere, you're far more likely to get a hit on the attribute (which has been around for as long as Python has had classes) than you will on the implicit method nonlocal introduced in Python 3. Hence why I suspect changing/aliasing the preferred name for referencing the nonlocal to be __classdef__ may help improve things, as it better hints at the lexical scoping, where you're referring to the class where the method is being defined at compile time, rather than the one where it was retrieved at run time. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mar 6, 2015, at 5:21, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think it's clearer to explain it as: Every method has a nonlocal variable named __class__ that refers to the class it's defined in. (As an optimization, CPython doesn't create a closure cell for __class__ if the method body doesn't need it.) The only problem with this definition is that someone implementing Python and writing type.__new__ has to know at exactly what point the __class__ cell becomes available. But I don't think an implementation that made it available too early would be considered broken, would it?

On 28 February 2015 at 02:56, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
I've now withdrawn PEP 422 in favour of this proposal: https://hg.python.org/peps/rev/4db0c1a3e363 (main site hadn't updated at time of writing, but it should propagate before too long)
In writing the withdrawal notice for PEP 422, I realised there's a case that needs to be more clearly specified here: the handling of cases where the base class already defines the hook. Since object has a default implementation, we should be able formally define this as the type creation machinery calling: super(cls, cls).__init_subclass__(ns, **kwargs) Rather than: cls.__init_subclass__(ns, **kwargs) We may also want to consider defining object.__init_subclass__ as accepting "cls, ns, *, metaclass=type" so we don't need to implicitly pop metaclass from the kwargs dict when it gets specified. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 February 2015 at 02:56, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Aye, I like this design, it just wasn't until I saw it written out fully that it finally clicked why restricting it to subclasses let both zero-argument super() and explicit namespace access to the defining class both work. Getting the latter to also work is a definite strength of your approach over mine.
New PEP numbers can be a good idea for significant redesigns anyway, otherwise people may end up having to qualify which version they're talking about while deciding between two competing approaches. In this case, unless someone comes up with an objection I've missed, I think your __init_subclass__ design is just plain better than my __autodecorate__ idea, and as you say, if we find a pressing need for it, we could always add "__new_subclass__" later to get comparable functionality. Assuming we don't hear any such objections, I'll withdraw PEP 422 and we can focus on getting PEP 487 ready for submission to python-dev. That leaves my one concern being the "namespace" keyword parameter. I'd started reconsidering that for PEP 422, and I think the argument holds at least as strongly for your improved design: https://www.python.org/dev/peps/pep-0422/#is-the-namespace-concept-worth-the... Since speed usually isn't a critical concern for class definitions, it's likely better to just make them ordered by default, which can be its own PEP, rather than being part of this one. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Feb 28, 2015 at 10:10 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
FWIW, Guido already told me a while back that OrderedDict by default for class definition namespace was okay. It's just been waiting on me to get back to the C OrderedDict implementation (which I've started to recently). -eric

On 1 March 2015 at 11:30, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Yeah, that was the conversation that prompted me to question the utility of the namespace keyword idea. The prototype class and class extension examples are cute, but fall into the "don't do this in production" category for me, making them prime candidates for custom metaclasses. The one really useful one is using an ordered dictionary for namespace execution, and your solution addresses that better by just making it the normal behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

That said, I would love if it was just possible to use super() in classmethods called from the __init__ of a metaclass. Until now, the __class__ of a method is only initialized after the metaclasses __init__ has run. Fortunately, this is easily changeable: it is possible to move the initialization of __class__ into type.__new__. As the class is (typically) created there, that is about as early as possible. I put the necessary changes to github: https://github.com/tecki/cpython/commits/cell
I feel the same way as you, so I am happy to drop the namespace idea from PEP 487, but I agree that then ordered namespaces would simply be great, but reading Eric's comment, that seems just around the corner. Greetings Martin

On 1 Mar 2015 19:28, "Martin Teichmann" <lkb.teichmann@gmail.com> wrote:
Unfortunately, it's too early - we can't control whether or not subclasses actually return the result of type.__new__ from their own __new__ implementation, so that object could have been replaced by something else by the time __init__ gets called. However, it may be feasible to move it into type.__init__. That way class methods would start working properly as soon as the metaclass __init__ implementation had called up into type.__init__. Regards, Nick.

Well, this is actually just a question of definition and documentation. Sure, a metaclasses __new__ may return whatever it pleases, but this is not in contradiction of setting __class__ of the methods in type.__new__. There are two possible scenarios: 1. metaclass.__new__ does not call type.__new__. Once we document that it is type.__new__'s job to set __class__, nobody can complain if __class__ is not set if type.__new__ is not called, as this would be documented behavior. In my changes I actually still left in the old code, so __build_class__ would still try to set __class__ even if type.__new__ did not get called. 2. somehow type.__new__ does get called (most likely by metaclass.__new__). It will do its job and set the __class__ of the methods of the class being created. Whatever metaclass.__new__ returns then doesn't matter much, because what sense would it make to set __class__ of that object it creates? To give an example: Imagine a soliton-generating metaclass: class Soliton(type): def __new__(cls, name, bases, ns): self = super().__new__(name, bases, ns) return self() And generate such a soliton: class A(metaclass=Soliton): def f(self): print(__class__) As of now, writing "A.f()" interestingly prints "<__main__.A object>", so __class__ is indeed set to what Soliton.__new__ returns, the object, not the class. This is currently correct behavior, but I think it actually is not what one expects, nor what one desires. (Does anyone out there make use of such a construct? Please speak up!) super() certainly won't work. So I think it would actually be a change for the better to let type.__new__ set the __class__ of the generated class. I know that it technically breaks backward compatibility. But I ran all the tests and none failed, so apparently until now nobody got weird ideas like me... I even tried to remove the old code from __build_class__ and still all tests run. (just a note, if anyone else here is trying to do so, I tampered with the compiler, so you better delete your .pyc files before running the tests) Greetings Martin

On Mon, Mar 02, 2015 at 11:57:09AM +0100, Martin Teichmann wrote:
You have to pass cls as an explicit argument to __new__, otherwise you get a TypeError: TypeError: type.__new__(X): X is not a type object (str) (that's in Python 3.3). The line should be: self = super().__new__(cls, name, bases, ns)
Is "soliton" the usual terminology for this? Do you perhaps mean singleton? I've googled for "soliton" but nothing relevant is coming up.
I do now! Seriously, I think you have just solved a major problem for me. I'll need to do some further experimentation, but that is almost exactly what I have been looking for. I've been looking for a way to have a class statement return a custom instance and I think this might just do it. -- Steve

On 2 March 2015 at 21:46, Steven D'Aprano <steve@pearwood.info> wrote:
Martin's question wasn't about using __new__ on the metaclass to do factory function magic (that's "normal usage" as far as metaclasses go). The "construct" in question here is: 1. Returning something from a metaclass __new__ implementation other than the result of type.__new__; AND 2. Relying on __class__ to refer to the returned object, rather than the result of type.__new__ That seems likely to be a vanishingly small subset of Python usage, and even if it does happen, the subset that couldn't switch to using explicit super() via the bound name is likely to be even smaller. That seems like a risk well worth taking when the pay-off is having class methods that use implicit super() just work when called from the metaclass. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2 March 2015 at 20:57, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
Thinking about this a bit more, I think a good way to look at it is to consider the impact of: 1. Using explicit super() rather than implicit super() 2. Using explicit decorators Currently, in your example, "__class__" and "A" will both refer to an *instance* of the class. If, instead, you used a "singleton" explicit decorator, then "__class__ " would refer to the actual class object, while "A" would refer to the instance. With your proposed change, then "__class__" will always refer to the actual class object, even if the metaclass __new__ plays implicit decoration games. It also means that class methods that rely on __class__ (whether explicitly or implicitly through super()) will "just work" in metaclass __init__ and __new__ methods, although only after the call up to the base class implementation in the latter case.
Aye, and the discrepancy I was concerned about can already happen when using explicit decorators. That suggest to me that even if someone *was* relying on __class__ pointing to the same object as the bound name when a metaclass does something odd, they can likely switch to referring to it by name instead. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 March 2015 at 03:28, Ethan Furman <ethan@stoneleaf.us> wrote:
No, it's printing out the type referenced from the definition time cell*, rather than the type of the object actually passed in to the method. Cheers, Nick. P.S. *For anyone that isn't already aware, the general gist of how that works: * The compiler actually special cases "super()" and "__class__" in method bodies * If it sees either of them, it implicitly adds a "__class__" cell reference to the function * In the "super()" case, the call is implicitly rewritten as "super(<first param>, __class__)" * __class__ is later filled in with a reference to the class being defined, once that actually exists In my view, that's never going to win any awards for "elegance in language design", but it makes super() so much easier to use I'm happy to tell my sense of aesthetics to be quiet :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 7 March 2015 at 00:11, Mark Young <marky1991@gmail.com> wrote:
Is that documented somewhere? I have never heard of that functionality.
The implicit __class__ cell? PEP 3135 introduced the ability to call super() without any arguments in Python 3 to ask the interpreter to just "do the right thing": https://www.python.org/dev/peps/pep-3135/ The actual language reference docs are here: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object Cheers, Nick. P.S. Those language reference class creation docs weren't fully updated for the original Python 3.0 release. They were updated in 3.3 when the new dynamic type creation APIs were added to the types module to simplify PEP 3115 compliant dynamic type creation: https://docs.python.org/3/library/types.html#dynamic-type-creation -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9 Mar 2015 03:18, "Mark Young" <marky1991@gmail.com> wrote:
Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I guess
not. Every other reference on that page is to the class attribute rather than the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) Cheers, Nick.

On 9 Mar 2015 10:12, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 9 Mar 2015 03:18, "Mark Young" <marky1991@gmail.com> wrote:
Thanks Nick! I swear I checked on the 3.x "Data Model" page, but I
guess not.
Every other reference on that page is to the class attribute rather than
the implicit method nonlocal, so it took a moment for me to relocate it myself. I should have searched for "super" instead :) With the benefit of hindsight, if I had the chance to do this over, I'd suggest we call the implicit method nonlocal "__classdef__" instead, specifically to avoid the name collision with the runtime type information on object instances. My general impression is that the current name triggers too many incorrect assumptions for most readers, and the name collision makes it harder than it needs to be to build a more correct mental model. It would actually be possible to still make that transition, with a view to deprecating the current "__class__" spelling at some point in the distant future. I'm too close to the problem to judge whether or not it would actually help reduce confusion, though. Regards, Nick.
Cheers, Nick.

On Mar 8, 2015, at 5:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I can tell you that every single time I've mentioned the __class__ nonlocal in a StackOverflow answer, someone mistakenly "corrected" me to tell me either "you need to write self.__class__" or "that won't work because it's the runtime type of self, which could be a subclass". I think a better solution, if you were doing it from scratch, might be to get rid of the __class__ member and force people to use type(self) to get it, so TOOTDI. (Of course that implies that code that needs to change __class__ would have to call some function to do so, but I'm not sure that's a bad thing. Dynamically changing your type is a rare thing to do in Python, and unusual or impossible in most other OO languages, except maybe during initialization in the Smalltalk family, so why shouldn't it be signaled by something scary?) On the other hand, is __class__ the only thing that's spelled with double underscored that isn't an attribute? Maybe that's the part that's confusing people...

On 9 March 2015 at 22:40, Andrew Barnert <abarnert@yahoo.com> wrote:
They're not substitutes for each other, as __class__ lets you lie about your type to the consumer, while type() doesn't, and this is by design:
For many use cases, you actually want obj.__class__ in order to handle proxy objects correctly.
It's not (just) about being able to dynamically change your type, it's about the fact that __class__ goes through the descriptor and class attribute lookup machinery, while type() doesn't.
It's not the only one - implicit module attributes like __name__ work that way, as do the __debug__ and __import__ builtins. As far as I can tell, it's specifically the fact that "__class__" looks like a plausible typo for "self.__class__" that causes problems, as this also creates the problem that if you search for "__class__" not only on the data model page, but anywhere, you're far more likely to get a hit on the attribute (which has been around for as long as Python has had classes) than you will on the implicit method nonlocal introduced in Python 3. Hence why I suspect changing/aliasing the preferred name for referencing the nonlocal to be __classdef__ may help improve things, as it better hints at the lexical scoping, where you're referring to the class where the method is being defined at compile time, rather than the one where it was retrieved at run time. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mar 6, 2015, at 5:21, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think it's clearer to explain it as: Every method has a nonlocal variable named __class__ that refers to the class it's defined in. (As an optimization, CPython doesn't create a closure cell for __class__ if the method body doesn't need it.) The only problem with this definition is that someone implementing Python and writing type.__new__ has to know at exactly what point the __class__ cell becomes available. But I don't think an implementation that made it available too early would be considered broken, would it?

On 28 February 2015 at 02:56, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
I've now withdrawn PEP 422 in favour of this proposal: https://hg.python.org/peps/rev/4db0c1a3e363 (main site hadn't updated at time of writing, but it should propagate before too long)
In writing the withdrawal notice for PEP 422, I realised there's a case that needs to be more clearly specified here: the handling of cases where the base class already defines the hook. Since object has a default implementation, we should be able formally define this as the type creation machinery calling: super(cls, cls).__init_subclass__(ns, **kwargs) Rather than: cls.__init_subclass__(ns, **kwargs) We may also want to consider defining object.__init_subclass__ as accepting "cls, ns, *, metaclass=type" so we don't need to implicitly pop metaclass from the kwargs dict when it gets specified. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (7)
-
Andrew Barnert
-
Eric Snow
-
Ethan Furman
-
Mark Young
-
Martin Teichmann
-
Nick Coghlan
-
Steven D'Aprano