Re: [Python-ideas] A (meta)class algebra
Hi Thomas, Hi list,
- More magic around combining metaclasses would make it harder to follow
what's going on. Explicit (this has a metaclass which is a combination of two other metaclasses) is better than implicit (this has a metaclass which can combine itself with other metaclasses found from the MRO).
Well I disagree. If you follow this rule, everyone who uses a metaclass needs to know how to combine it with another metaclass. This leads to code like (I'm citing IPython.qt.console.console_widget):
class ConsoleWidget(MetaQObjectHasTraits('NewBase', (LoggingConfigurable, QtGui.QWidget), {})):
which, in my opinion, is hard to read and grasp. Everyone has to be taught on how to use those metaclass mixers.
- I *want* it to be hard for people to write multiple-inheriting code with
multiple metaclasses. That code is most likely going to be a nightmare for anyone else to understand, so I want the person creating it to stop and think if they can do it a different way, like using composition instead of inheritance.
I'm completely with you arguing that inheritance is overrated and composition should be used more often. Yet, I actually think that the above example is actually not such a bad idea, why should a QWidget not be LoggingConfigurable, and have traits? Whatever that might mean in detail, I think most programmers do get what's going on. It's just that
class ConsoleWidget(LoggingConfigurable, QtGui.QWidget):
is much more readable. Is there magic going on in the background? Well, to some point, yes. But it is explicit magic, the authors of the metaclasses have to explicitly write an __add__ method to do the metaclass combination. In this regard, it's actually more explicit than the metaclass mixing function as above and also mentioned by Anthony (how do I respond to two mails on a mailing list at the same time?), which typically stuff together metaclasses without thinking to much about them.
I also don't see how this is harder to refactor. If everyone has to explicitly combine to metaclasses if needed, you will have to change all that code if the details of combining the metaclasses change. In my version, one would just have to touch the metaclass code.
This becomes even worse if you want to add a metaclass to a class. Then you have to change every class that inherits from it. Now you say that code with multiple metaclasses is a bad thing in general, but I don't agree. There are very simple examples where it makes a lot of sense. Say, you want to write a class with traits that implements an abstract base class. Wouldn't it be cool if you could write
class Spam(SpamABC, SomethingWithTraits):
instead of writing
class Spam(SomethingWithTraits): ... SpamABC.register(Spam)
?
Greetings
Martin
On Fri, Feb 13, 2015 at 5:43 AM, Martin Teichmann lkb.teichmann@gmail.com wrote:
Hi Thomas, Hi list,
- More magic around combining metaclasses would make it harder to follow
what's going on. Explicit (this has a metaclass which is a combination of two other metaclasses) is better than implicit (this has a metaclass which can combine itself with other metaclasses found from the MRO).
Well I disagree. If you follow this rule, everyone who uses a metaclass needs to know how to combine it with another metaclass. This leads to code like (I'm citing IPython.qt.console.console_widget):
class ConsoleWidget(MetaQObjectHasTraits('NewBase', (LoggingConfigurable, QtGui.QWidget), {})):
which, in my opinion, is hard to read and grasp. Everyone has to be taught on how to use those metaclass mixers.
- I *want* it to be hard for people to write multiple-inheriting code with
multiple metaclasses. That code is most likely going to be a nightmare for anyone else to understand, so I want the person creating it to stop and think if they can do it a different way, like using composition instead of inheritance.
I'm completely with you arguing that inheritance is overrated and composition should be used more often. Yet, I actually think that the above example is actually not such a bad idea, why should a QWidget not be LoggingConfigurable, and have traits? Whatever that might mean in detail, I think most programmers do get what's going on. It's just that
class ConsoleWidget(LoggingConfigurable, QtGui.QWidget):
is much more readable. Is there magic going on in the background? Well, to some point, yes. But it is explicit magic, the authors of the metaclasses have to explicitly write an __add__ method to do the metaclass combination. In this regard, it's actually more explicit than the metaclass mixing function as above and also mentioned by Anthony (how do I respond to two mails on a mailing list at the same time?), which typically stuff together metaclasses without thinking to much about them.
I also don't see how this is harder to refactor. If everyone has to explicitly combine to metaclasses if needed, you will have to change all that code if the details of combining the metaclasses change. In my version, one would just have to touch the metaclass code.
This becomes even worse if you want to add a metaclass to a class. Then you have to change every class that inherits from it. Now you say that code with multiple metaclasses is a bad thing in general, but I don't agree. There are very simple examples where it makes a lot of sense. Say, you want to write a class with traits that implements an abstract base class. Wouldn't it be cool if you could write
This is the one thing keeping metaclasses from being truly general – using them limits inheritance.
The problem is that metaclasses are too powerful. They can change the C-level type; you can't automatically combine two metaclasses that do this. They can change the class statement's namespace (via __prepare__). Again, you can't automatically combine these. In these cases you really need knowledge of *both* metaclasses to determine if you're mixing them correctly. It can't really be done in a method on one of the classes. (And btw, __add__ is a terrible name for the method. If it's meant to be called implicitly by Python the "+" sugar is unnecessary. Something like __metaclass_combine__ would be a better name.)
So, I argue that automatically combining metaclasses makes sense if at least one of the metaclasses is "simple": it only mutates the class's attributes, after the class is created. In other words, it overrides __init__ but not __new__ or __prepare__. Such metaclasses are very useful – registrars, ORM tables, etc. The problem that limits their adoption is that they can't be freely combined. Similar things can be done by decorators, the problem being that the decorator needs to be repeated on every subclass. (But this may not be a huge problem: in many "Registrar" cases, explicit is better than implicit.)
Anyway, perhaps "simple" metaclasses can be combined automatically? Or there could be an instantiation hook to enable these classes without limiting subclassing?
Hi Petr, Hi list,
This is the one thing keeping metaclasses from being truly general – using them limits inheritance.
I agree. That was why I am proposing a solution to this problem.
The problem is that metaclasses are too powerful. They can change the C-level type; you can't automatically combine two metaclasses that do this.
This is exactly the problem my proposal solves. In my proposal, metaclasses will NOT be combined automatically, but the authors of the metaclasses have to explicitly write a method that does the job. If that's not possible, the method is free to fail, like raising an error (if one follows my idea with __add__, then it should return NotImplemented)
(And btw, __add__ is a terrible name for the method. If it's meant to be called implicitly by Python the "+" sugar is unnecessary. Something like __metaclass_combine__ would be a better name.)
I'm fine with any name. I just thought it to be useful that the entire machinery with __add__ and __radd__ are already there.
Anyway, perhaps "simple" metaclasses can be combined automatically? Or there could be an instantiation hook to enable these classes without limiting subclassing?
well, my proposal is such a hook.
Greetings
Martin
On 13 February 2015 at 20:55, Petr Viktorin encukou@gmail.com wrote:
On Fri, Feb 13, 2015 at 5:43 AM, Martin Teichmann
This becomes even worse if you want to add a metaclass to a class. Then you have to change every class that inherits from it. Now you say that code with multiple metaclasses is a bad thing in general, but I don't agree. There are very simple examples where it makes a lot of sense. Say, you want to write a class with traits that implements an abstract base class. Wouldn't it be cool if you could write
This is the one thing keeping metaclasses from being truly general – using them limits inheritance.
The problem is that metaclasses are too powerful.
FWIW, that's why http://legacy.python.org/dev/peps/pep-0422/ exists - it aims to standardise Zope's notion of "class initialisers" so you can have definition time behaviour that gets inherited by subclasses (unlike explicit class decorators), without running the risk of introducing metaclass conflicts, and without making the metaclass system even more complicated.
Then the cases that can be safely combined with other metaclasses would merely stop using a custom metaclass on 3.5+, and instead rely entirely on having an appropriate base class.
The last round of reviews (back before 3.4 was released) raised a bunch of interesting questions/challenges, and then somebody (*cough*) decided it would be a good idea to go spend more time helping with PyPI ecosystem wrangling instead :)
Regards, Nick.
On Fri, Feb 13, 2015 at 12:59 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 13 February 2015 at 20:55, Petr Viktorin encukou@gmail.com wrote:
On Fri, Feb 13, 2015 at 5:43 AM, Martin Teichmann
This becomes even worse if you want to add a metaclass to a class. Then you have to change every class that inherits from it. Now you say that code with multiple metaclasses is a bad thing in general, but I don't agree. There are very simple examples where it makes a lot of sense. Say, you want to write a class with traits that implements an abstract base class. Wouldn't it be cool if you could write
This is the one thing keeping metaclasses from being truly general – using them limits inheritance.
The problem is that metaclasses are too powerful.
FWIW, that's why http://legacy.python.org/dev/peps/pep-0422/ exists - it aims to standardise Zope's notion of "class initialisers" so you can have definition time behaviour that gets inherited by subclasses (unlike explicit class decorators), without running the risk of introducing metaclass conflicts, and without making the metaclass system even more complicated.
Then the cases that can be safely combined with other metaclasses would merely stop using a custom metaclass on 3.5+, and instead rely entirely on having an appropriate base class.
The last round of reviews (back before 3.4 was released) raised a bunch of interesting questions/challenges, and then somebody (*cough*) decided it would be a good idea to go spend more time helping with PyPI ecosystem wrangling instead :)
Regards, Nick.
Oh! I missed that PEP.
I think this is a much better way to go. It defines a simple thing, rather than providing a way for a powerful thing to say "I'm simple!" :)
Martin, would this proposal work for your needs?
Hi,
me too, I missed that PEP, and yes, it would certainly work for all my needs.
In short words, the PEP reads: forget all that metaclass stuff, here comes the real solution. Maybe we can even abandon metaclasses altogether in Python 4, I think noone would shed a tear over them.
But if the metaclass fans should win the battle, I would like my proposal included sometimes.
Greetings
Martin
While PEP422 is somewhat useful for heavy users of metaclasses (actually not necessarily so heavy; as mentioned in the thread you'll hit the problem as soon as you want to make e.g. a QObject also instance of another metaclass), I don't like it too much because it feels really ad hoc (adding yet another layer to the already complex class initialization machinery), whereas dynamically creating the sub-metaclass (inheriting from all the required classes) seems to be the "right" solution. In a sense, using an explicit and ad hoc mixer ("__add__") would be like saying that every class should know when it is part of a multiple inheritance chain, whereas the implicit ("multi_meta") mixer just relies on Python's MRO to do the right thing. Of course, best would be that the "multi_meta" step be directly implemented at the language level, and I agree that having to add it manually isn't too beautiful.
__init_class__ could then simply be implemented as part of a normal metaclass, from which you could inherit without worrying about other metaclasses.
Antony
2015-02-13 5:27 GMT-08:00 Martin Teichmann lkb.teichmann@gmail.com:
Hi,
me too, I missed that PEP, and yes, it would certainly work for all my needs.
In short words, the PEP reads: forget all that metaclass stuff, here comes the real solution. Maybe we can even abandon metaclasses altogether in Python 4, I think noone would shed a tear over them.
But if the metaclass fans should win the battle, I would like my proposal included sometimes.
Greetings
Martin _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 14 February 2015 at 03:27, Antony Lee antony.lee@berkeley.edu wrote:
While PEP422 is somewhat useful for heavy users of metaclasses (actually not necessarily so heavy; as mentioned in the thread you'll hit the problem as soon as you want to make e.g. a QObject also instance of another metaclass), I don't like it too much because it feels really ad hoc (adding yet another layer to the already complex class initialization machinery), whereas dynamically creating the sub-metaclass (inheriting from all the required classes) seems to be the "right" solution.
The need to better explain the rationale for the design was one of the pieces of feedback from the previous review.
__init_class__ is a pattern extraction refactoring (one first applied successfully by the Zope project more than 10 years ago). A fairly common usage model for metaclasses is to provide class creation time behaviour that gets inherited by subclasses, but otherwise has no effect on the runtime behaviour of the class or subclasses. Currently, the only way to obtain that is through a custom metaclass, which may also impact other class behaviours, and hence they don't combine readily, and you can't add one to an existing public class without breaking backwards compatibility.
That pattern is common enough to potentially be worth pulling out into a mechanism that *can't* have a runtime impact on class object behaviour, expecially since an existing class could adopt such a feature while remaining backwards compatible with subclasses that add a custom metaclass.
While this does add a second mechanism to achieve something that a custom metaclass can already do, developers are still left with a relatively straightforward design decision: if you need to impact the runtime behaviour of the class or its subclasses (the way abc.ABCMeta and enum.EnumMeta do), then you still need a custom metaclass. If you only need inherited class definition time behaviour, and only need to support versions of Python that support __init_class__, then you should use it is instead, as its a much simpler mechanism for people to understand, and hence presents much lower barriers to future maintainability.
That kind of "if the new, more specific, mechanism applies, you should use it, as it will be easier to write, read and maintain" situation is the result you're looking for any time you're attempting to do pattern extraction like this.
The key problem with custom metaclasses is that once they're in the mix, all bets regarding type system behaviour are off - you can use those to make classes do pretty much anything you want (the abc module, enum and SQL ORMs are readily available examples, while systems like Zope and PEAK show just how much customisation of the type system can already be done just through the application of custom metaclasses).
Asking people to learn both how metaclasses in general work *and then* how a new automatic metaclass combination works before they can maintain your code effectively doesn't remove any barriers to project maintainability - instead, it adds a new one.
In a sense, using an explicit and ad hoc mixer ("__add__") would be like saying that every class should know when it is part of a multiple inheritance chain, whereas the implicit ("multi_meta") mixer just relies on Python's MRO to do the right thing. Of course, best would be that the "multi_meta" step be directly implemented at the language level, and I agree that having to add it manually isn't too beautiful.
Custom metaclasses are inherently hard to understand. When type is a subclass of object, while object is itself an instance of type, and when using a custom metaclass lets you completely rewrite the rules for class creation, instance creation, attribute lookup, subclassing checks, instance checks, and more, learning how to wield them effectively does give you great power, but it also means recognising that every time you decide to use one you're significantly reducing the number of people that are going to be able to effectively contribute to and help maintain that specific part of your project. When you do decide its worthwhile to introduce one, it's generally because the additional complexity in the class hierarchy implementation pays for itself elsewhere in code that is more commonly modified (for example, compare the number of ABCs, enumerations and ORM table definitions to the number of ABC implementations, enumeration implementations and ORM implementations).
The key goal of PEP 422 is to take a significant number of use cases that currently demand the use of a custom metaclass and instead make it possible to implement, understand and maintain them with roughly the same level of knowledge of the inner workings of Python's runtime type system as is needed to implement a custom __new__ method.
__init_class__ could then simply be implemented as part of a normal metaclass, from which you could inherit without worrying about other metaclasses.
__init_class__ *does* get implemented as part of a metaclass: type. type anchors the standard library's metaclass hierarchy (and most other metaclass hierarchys) the way object anchors the overall object hierarchy.
Once you grasp how type can be a subclass of object, while object is itself an instance of type, then you've tackled pretty much the most brainbending part of how the runtime type system bootstraps itself :)
Cheers, Nick.
On Feb 13, 2015, at 5:27, Martin Teichmann lkb.teichmann@gmail.com wrote:
Hi,
me too, I missed that PEP, and yes, it would certainly work for all my needs.
In short words, the PEP reads: forget all that metaclass stuff, here comes the real solution. Maybe we can even abandon metaclasses altogether in Python 4, I think noone would shed a tear over them.
But if the metaclass fans should win the battle, I would like my proposal included sometimes.
That sounds reasonable in principle. Even if that PEP got accepted for Python 3.5, there would still be an awful lot of now-unnecessary metaclasses out there for a long time. For example, PyQt isn't going to drop 3.4 support, and, even if it did, it wouldn't be rewritten overnight. All it does is let you write a 3.5 application that works more easily with PyQt by not having to write your own metaclasses and toss them into the mix. If you still need or want to use metaclasses, you still need to mix them somehow.
But I think it also gives weight to Barry's point: the problem is that both you and PyQt are using something that's much more powerful and flexible than needed, and that forced readers of your code to think through how your metaclasses interact before they can understand your classes, and trying to sweep that under the rug may not really help beyond the "shut up and code" perspective that gets you the first 80% of the way before you have to debug something confusing and suddenly you have to figure out something very complicated when nearing a deadline...
On 12 February 2015 at 20:43, Martin Teichmann lkb.teichmann@gmail.com wrote:
class ConsoleWidget(MetaQObjectHasTraits('NewBase', (LoggingConfigurable, QtGui.QWidget), {})):
which, in my opinion, is hard to read and grasp. Everyone has to be taught on how to use those metaclass mixers.
Yes, it's hard to read, but it indicates that there's something complex going on. Your proposal is cleaner, but to my mind it's just sweeping under the carpet the fact that multiple metaclasses are in use. To fully understand the code, you would still need to get a headache understanding how multiple metaclasses interact, but you would first have to notice that one metaclass is able to combine itself with another, and that there is another one to combine with.
I'm completely with you arguing that inheritance is overrated and composition should be used more often. Yet, I actually think that the above example is actually not such a bad idea, why should a QWidget not be LoggingConfigurable, and have traits?
It's not in itself a ridiculous idea, but it's bad practice precisely because multiple inheritance is tricky. If inheritance means 'is a', it's hard to be two completely different kinds of thing at the same time - especially when both of those things rely on advanced features of Python. If I was writing that code from scratch, I would try very hard to make it such that Configurable objects *have* QObjects, rather than being QObjects.
Moreover, I don't think you can overcome this difficulty by adding any new features to inheritance or metaclasses in Python. I doubt you could overcome it even if you could redesign multiple inheritance with a blank slate, but perhaps that is possible.
Given that I think the intersection of multiple inheritance and metaclasses will always be horrible to understand, I think anything that makes it easier to get into that situation is a bad idea. I'd support something that makes it easier to understand such situations when they do exist, but I think your proposal would make it harder, not easier.
Thomas
participants (6)
-
Andrew Barnert -
Antony Lee -
Martin Teichmann -
Nick Coghlan -
Petr Viktorin -
Thomas Kluyver