On Feb 16, 2015, at 8:19, Martin Teichmann wrote:

Hi list,

...

Again, Python already has a metaclass that everyone is supposed to use, namely `type`.

well, there is a problem out here: people are indeed using other metaclasses. Even the standard library, like ABC or enums.

But almost all such metaclasses are instances (and direct subclasses, but that's not as important here) of type; I think he should have said "a type that everyone can usually use as a metaclass, and can at least pretty much always use as a metametaclass when that isn't appropriate". Anyway, for the problem you're trying to solve: there _is_ a real problem that there are a handful of popular metaclasses (in the stdlib, in PyQt, maybe elsewhere) that people other than the implementors often need to combine with other metaclasses, and that's hard, and unless PEP 422 makes all those metaclasses unnecessary, you still need a solution, right? Maybe the answer is not a generic way of combining any two metaclasses, but some way for the author of a metaclass to make it more easily combinable. Whether that's a decorator, a metametaclass, or something even more complicated, I think you'd have an easier time convincing people of its utility: if you're writing a metaclass like Qt's, use this and other people can subclass your metaclass just by doing X instead of having to do Y. The fact that it also happens to work in a wider variety of cases is nice, but not the selling point. (But it can hardly be used as an argument against your design--any decorator can be used to monkey patch a function or class after the fact; any ABC can be registered after the fact; etc. That's a normal part of Python.) However, if it can be used to more easily backport PEP 422 code, that _is_ a selling point, of course.

And there is a problem here in python: once you do multiple inheritance, and two classes have different metaclasses, there is no way of programmatically determining an appropriate metaclass.

Sure, one can be of Thomas' opinion that doing so is a bad idea anyways, but there are other people who think that writing and combining metaclasses actually is thing one can do.

Especially for simple cases I do think that metaclasses are a good idea and there should be a way to tell python how to combine two metaclasses.

I am trying to solve this problem. What I posted was just one simple idea how to solve it. Another one is the idea I posted earlier, the idea was to add metaclasses using __add__. People didn't like the __add__, so I implemented that with a new method called merge, it's here: https://github.com/tecki/cpython/commits/metaclass-merge

Instead of yelling at me that my solutions are too complicated, it would be very helpful and constructive to answer the following questions:

- is it desirable to have a programmatic way to combine metaclasses? - if yes, how should that be done?

My idea is that yes, it is desirable to have a programmatic way of combining metaclasses, and I have now proposed two ways how to do that.

Greetings

Martin _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Hi Petr, Hi Andrew, Hi list, (responding to my own post, as I cannot respond to Petr's and Andrew's at the same time)

- is it desirable to have a programmatic way to combine metaclasses?

there seems to be general agreement that there should be no automatic combination of metaclasses, while many think that explicit combination should be possible. The only way to do that currently is by subclassing existing metaclasses. This is typically done by the user of the metaclass, not its author. It would be a nice thing if an author could do that. Petr argues s/he already can, by subclassing. Unfortunately, this is not so easy as this means that every user of such a class then has to use all inherited metaclasses. As an example: imagine you have a metaclass that mixes well with PyQt's metaclass. Then you could simply inherit from it. But this would mean that every user of you class also has to install PyQt, no matter if the code has anything to do with it. Now Thomas argues that this should simply not be done at all. I understand his argument well that multiple inheritance across project boundaries is probably a bad idea in most cases. But I have an example where it does make a lot of sense: ABCs. It would just be nice if one could implement and ABC which is a QObject. That's what ABCs are for. Sure, you can do this by registering the class, but this means you loose a lot like already predefined methods of an ABC. But maybe we can just special-case ABCs.

- if yes, how should that be done?

So I proposed two ways how this could be implemented: One is to change the current metaclass combination code to check whether a class is a subclass not by PyType_IsSubtype, but by PyObject_IsSubclass, meaning that the __subclasshook__ is called. This has the advantage of being only a minimal change to the python core, I'm not sure how many people out there new at all how the subclass detection was done. It is also very systematic: it is very analog to ABCs, just the other way around. An AbstractMetaClass would simply now which classes exist and how they can be combined. Either by inheriting from AbstractMetaClass, or by registering a foreign class. This way has the drawback of being rather complex on the python side. But actually, I prefer complexity on the Python- over complexity on the C side. This proposal is at https://github.com/tecki/cpython/commits/metaclass-issubclass My other proposal was to have a way to explicitly combine (meta)classes, by having a method in the type object (I originally called that __add__, people didn't like that so I renamed it to merge). This idea has somewhat more code on the C side, but is simpler on the python side. It might also be easier to grasp by newbies. Otherwise it is about as powerful as the other idea. This proposal is at https://github.com/tecki/cpython/commits/metaclass-merge Once we have one of my ideas implemented, I think we can implement PEP 422 in the standard library. Petr claims that this is just doing simple things with complex code. I disagree, I think that the current metaclass code in the python interpreter is already incredibly context, and just throwing more code at it is actually a very complex solution. Sure, on the python side all looks pretty, but only because we swept all complexity under the C carpet. But I won't fight for that point. What I am fighting for is to modify PEP 422 in a way that it can be backported to older versions of python and be put on PyPI. This means that __init_class__ should become __init_subclass__, meaning that only subclasses are initialized. (__init_class__ doesn't work properly, as super() won't work. This problem already exists in the zope version). There is also a completely different way we could solve all those problems: abandon metaclasses for good, and replace them eventually with a PEP 422 like approach. This would mean that on the long run the following should be added: - a __new_subclass__ as a replacement for the metaclass __new__. This shows that it is necessary to operate on subclasses, otherwise one will have a hard time to create the class itself... - __subclasscheck__ and __instancecheck__ must be called on the class, not the metaclass - sometimes metaclasses are used to add methods to the class only, that cannot be reached by an instance. A new @classonlymethod decorator should do that. - I'm sure I forgot something. A pure python version of this proposal (it would go into C, for sure) can be found here: https://github.com/tecki/metaclasses Note that this is much more complicated as it would need to be if only the original PEP422 ideas were implemented. This was a long post. Enjoy. Greetings Martin

But if you did have a merging mechanism, would it solve this problem? To work properly, the merge mechanism in the new metaclass would have to check that it's dealing with the PyQt metaclass, wouldn't it? How whould that check work if the PyQt metaclass is not available?

Well, both would inherit from the same baseclass, let's call it SimpleMixable, or AbstractMetaclass. This will now mix the two classes. (In Nick's words: "You're asking us to assume covariant metaclasses in a subset of cases, while retaining the assumption of contravariance otherwise.") If PyQt doesn't want to inherit from AbstractMetaclass, it can later be registered by AbstractMetaclass.register(PyQt4.QtMetaclass)

I don't quite get the example. Why would you want an ABC that is a QObject?

well, imagine you have an ABC called WindowABC that defines abstract methods that apply to a window, then you could write a class class Window(WindowABC, QWindow): # add some code here which creates a concrete class implementing the WindowABC. The documentation states about this: "An ABC can be subclassed directly, and then acts as a mix-in class."

...

- a __new_subclass__ as a replacement for the metaclass __new__. This shows that it is necessary to operate on subclasses, otherwise one will have a hard time to create the class itself...

Well, super() cooperative inheritance won't help if you really need different types of objects, rather than differently initialized objects. This pretty much needs the full power of metaclasses.

It doesn't. In the test code that I posted to github there is the example: class A(Object): @staticmethod def __new_subclass__(name, bases, dict): return 3 class B(A): pass self.assertEqual(B, 3) it already works.

Such a decorator isn't terribly hard to write today.

Done.

You're asking us to assume covariant metaclasses in a subset of cases, while retaining the assumption of contravariance otherwise. That isn't going to happen - we expect type designers to declare metaclass covariance by creating an explicit subclass and using it where appropriate..

How do you solve my example with ABCs? That means if a QObject (from PyQt) tries to implement an ABC by inheritance, then one first has to create a new metaclass inheriting from ABCMeta and whatever the PyQt metaclass is. Sure, one could register the class to the ABC, but then you loose the option of inheriting stuff from the ABC. One could also write an adaptor - so using composition. But to me that defeats the point of the concept of ABCs: that you can make anything fit appropriately. In general, that smells to me like a lot of boilerplate code.

PEP 422 bypasses the default assumption of metaclass contravariance by operating directly on type instances, without touching any other part of the metaclass machinery.

So there is a problem with the metaclass machinery, and we solve that by making one certain special case work? When is the next special case going to show up? Will it also be included into type? Greetings Martin

On Feb 17, 2015, at 14:15, Nick Coghlan wrote:

On 17 Feb 2015 23:13, "Martin Teichmann" wrote:

...

...

You're asking us to assume covariant metaclasses in a subset of cases, while retaining the assumption of contravariance otherwise. That isn't going to happen - we expect type designers to declare metaclass covariance by creating an explicit subclass and using it where appropriate..

How do you solve my example with ABCs? That means if a QObject (from PyQt) tries to implement an ABC by inheritance, then one first has to create a new metaclass inheriting from ABCMeta and whatever the PyQt metaclass is.

Sure, one could register the class to the ABC, but then you loose the option of inheriting stuff from the ABC. One could also write an adaptor - so using composition. But to me that defeats the point of the concept of ABCs: that you can make anything fit appropriately. In general, that smells to me like a lot of boilerplate code.

If you're doing something incredibly hard and incredibly rare, the code you have to write to tell the interpreter how to handle it is not really boilerplate - the "boilerplate" appellation is generally reserved for verbose solutions to *common* problems. This is not a common problem - using a custom metaclass at all is rare (as it should be), and combining them even more so.

If you specifically want to write QtABCs, then the appropriate solution is to either write your own combining metaclass, or else to go to whichever Qt wrapper project you're using and propose adding it there.

That's a readable and relatively simple approach that will make sense to anyone that understands Python metaclasses, and will also work in all existing versions of Python that supports ABCs.

It is *not* a good enough reason to try to teach the interpreter how to automatically combine metaclasses. Even if you could get it to work reliably (which seems unlikely), it would be impossible to get it to a point where a *human* (even one who already understood metaclasses in general) could look at the code and easily tell you what was going on behind the scenes.

...

...

PEP 422 bypasses the default assumption of metaclass contravariance by operating directly on type instances, without touching any other part of the metaclass machinery.

So there is a problem with the metaclass machinery, and we solve that by making one certain special case work? When is the next special case going to show up?

Probably never. We've had the metaclass machinery in place for over a decade, and the "run code at subclass creation time" pattern is the only particularly common metaclass idiom I am aware of where we would benefit from a simpler alternative that people can use without first needing to understand how metaclasses work.

Well, this is the second such pattern discovered in 14 years. (Remember that Py3k added __prepare__ to make it possible to simplify and clean up the machinery. And now there's this.) So, maybe in another 7 years or so, someone will come up with another such pattern. But I think that's fine. If Python 3.11 can benefit from another small change to type that nobody's thought of yet, I think there's a good chance you'd still want to make that change even if it was possible to work around thanks to a more powerful metaclass system or not. (After all, you want PEP 422 even though it's already possible to work around that need with the existing metaclass system.) So I think you're right that the argument "We don't want to keep modifying type" doesn't really work here.

(I'm aware that "common" is a relative term when we're discussing metaclasses, but establishing or otherwise ensuring class invariants in subclasses really is one of their main current use cases)

...

Will it also be included into type?

Unfortunately, you're making the same mistake the numeric folks did before finally asking specifically for a matrix multiplication operator: trying to teach the interpreter to solve a hard general problem (in their case, defining custom infix operators), rather than doing pattern extraction on known use cases to create an easier to use piece of syntactic sugar. When the scientific & data analysis community realised that the general infix operator case didn't *need* solving, and that all they really wanted was a matrix multiplication operator, that's what they asked for and that's what they got for Python 3.5+.

Changes to the language syntax and semantics should make it clear to *readers* that a particular idiom is in use, rather than helping to hide it from them.

PEP 422 achieves that - as soon as you see "__init_class__" you know that inheriting from that base class will just run code at subclass initialisation time, rather than potentially making arbitrary changes to the subclass behaviour the way a custom metaclass can.

By contrast, implicitly combining metaclasses at runtime does nothing to make the code easier to read - it only makes it (arguably) easier to write by letting the developer omit the currently explicit combining metaclass. That means that not only does the interpreter have to be taught how to do implicit metaclass combination but future maintainers have to learn to do it in their heads, making the code *harder* to read, rather than easier.

Regards, Nick.

...

Greetings

Martin _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 17 Feb 2015 23:29, "Martin Teichmann" wrote:

Hi again,

OK, so if my ideas don't get through, would it at least be possible to make PEP 422 have an __init_subclass__ only initializing the subclasses of the class? That would have the great advantage of being back-portable as a metaclass all the way down to python 2.7. I guess this would help PEP 422 being actually used a lot.

Zope already implements __classinit__ via a metaclass, so I'm not clear on how this addition would help with a backport of the __init_class__ spelling to Python 2. PEP 422 largely proposes standardisation of an existing custom metaclass capability (using a different method name to avoid conflicting) for improved ease of use, discoverability and compatibility, rather than making any fundamental changes to the type system design. I agree a clean idiom for detecting whether you're in the base class or not might be useful, but even without specific support in the PEP you'll at least be able to do that either by inspecting the MRO, or else by assuming that the class name not being bound yet means you're still in the process of creating it. Regards, Nick.

On 18 Feb 2015 18:51, "Martin Teichmann" wrote:

...

Zope already implements __classinit__ via a metaclass, so I'm not clear

on

...

how this addition would help with a backport of the __init_class__ spelling to Python 2.

Well, there is just a small issue in the zope implementation. They don't support super(). So you cannot write:

from ExtensionClass import Base class Spam(Base): def __class_init__(self): super().__class_init__(self) # this line raises RuntimeError # do something here

The precise error is: RuntimeError: super(): empty __class__ cell

My pure python implementation with __init_subclass__ support super() without problems.

The problem is that at the point they call __class_init__ (at the end of ExtensionClass.__new__) the __class__ in the methods is not initialized yet. Interestingly, it also doesn't help to move that call into __init__, which is still a mystery to me, I was under the assumtion that at the point of __init__ the class should be fully initialized. Interestingly, type manages to set the __class__ of the methods AFTER __init__ has finished. How it manages to do so I don't know, but probably that's special-cased in C.

...

I agree a clean idiom for detecting whether you're in the base class or not might be useful, but even without specific support in the PEP you'll at least be able to do that either by inspecting the MRO, or else by assuming that the class name not being bound yet means you're still in the

Yes, populating __class__ happens inside the interpreter only after the class is fully constructed, so even though the *class* is fully initialised at the end of __init__, a reference hasn't been inserted into the cell yet. (The __class__ cell doesn't actually live on the class object - it's created implicitly by the interpreter and then referenced from the individual methods as a closure variable for each method that looks up "__class__" or "super") Python 2 doesn't provide the implicitly populated __class___ cell at all though, so you have to refer to the class object by its name - zero argument super is a Python 3 only feature. process of

...

creating it.

That sounds very complicated to me. Could you please give me an example how I am supposed to do that?

Second one is fairly straightforward - either do a check for the class name in the module globals() or catching the resulting NameError. Checking the MRO would be a bit more fragile, and likely not worth fiddling with given the ability to check for the name binding of the base class directly. You also have the option of just setting an initialisation flag on the class object itself (since the initial execution in the base class will always be the first execution). Cheers, Nick.

In practice, this can be worked around using a method on the metaclass, defined as follows (basically following Nick's idea): def _super_init(cls, name): super_cls = super(sys._getframe(1).f_globals.get(name, cls), cls) try: init_class = super_cls.__init_class__ except AttributeError: pass else: init_class() Now call cls._super_init() whenever you wanted to call super().__init__(). Antony 2015-02-19 4:28 GMT-08:00 Martin Teichmann :

...

Yes, populating __class__ happens inside the interpreter only after the class is fully constructed, so even though the *class* is fully initialised at the end of __init__, a reference hasn't been inserted into the cell yet. (The __class__ cell doesn't actually live on the class object - it's created implicitly by the interpreter and then referenced from the individual methods as a closure variable for each method that looks up "__class__" or "super")

Python 2 doesn't provide the implicitly populated __class___ cell at all though, so you have to refer to the class object by its name - zero argument super is a Python 3 only feature.

In short: PEP 422 in the current form is not back-portable. With my modifications it is portable all the way down to python 2.7.

PEP 422 being back-portable => people will immediately start using it. PEP 422 not back-portable => it won't be used for several years to come.

Greetings

Martin _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/