Is multiple inheritance Pythonic?
Hi List, Summary: Multiple inheritance is not used in new code of the standard library anymore, even where it could be of good use. Should multiple inheritance thus be abandoned, or supported more? I am currently developing a little library for the communication in our project. It is supposed to support different protocols, encryptions and addressing standards. At first I thought I'd just use multiple inheritance. A user of the library would then just write a little stub-class, as in: class MyProtocol(URLAdressMixin, SSLEncryptionMixin, HTTPProtocol): pass and voilà! everything is ready. But because I try to be one of the cool guys, I tried to do all of that using asyncio. It has a pluggable event loop, where you can plug in your own event loop with the specifics of your project. Again, I thought, let's to multiple inheritance! I needed some priority scheduling, so just write a mixin that does that, and mix it into the base class of the event loop. To my astonishment, the designers of asyncio explicitly decided against that option. They prefer not to let users subclass at all. Instead, asyncio has a factory plugin system, as an example, one can call set_task_factory to modify what the BaseEventLoop.create_task method is doing. Things like that used to be clearly the domain of mixin classes. The question arised, why would one do that? Is multiple inheritance bad, or not Pythonic? Guessing what the decisions of the asyncio developers were, several problems come to my mind. Firstly, metaclasses a notoriously problematic in a multiple inheritance context. Some of you might have seen my work on PEP 487 to mitigate that problem. Another issue are the stub classes mentioned above. One ends up writing lots of little stub classes combining mixins together. (An example from https://github.com/jupyter/qtconsole/blob/master/qtconsole/inprocess.py: class QtInProcessKernelManager(QtKernelManagerMixin, InProcessKernelManager):) Many users don't like that, it's too much programming, too little using. Another problem is that mixins cannot be mixed-in at runtime anymore. Once the event loop is running, I cannot sneak in a new task factory anymore. It is questionable whether that's desired. Another argument is that using the factory way, a library might "sneak in" its factories without the user noticing. But in my opinion, that's against the Pythonic concept of being explicit. A library should rather document: "Hey, you need to include SuperTaskMixin into your Eventloop", rather than silently setting a new factory. This also touches mainainability a lot. Imagine you are using two libraries, both at some time deciding they need to set a task factory. That won't work out. All of this is nicely solved in a multiple inheritance scheme: the method resolution order using super() is flexible enough to allow two independent extensions of create_task, as long as it is properly documented how to use it. So both concepts, multiple inheritance as well as pluggable factories, have their pros and cons. According to the Zen, there should ideally only be one way of doing things, but given that I'm not Dutch I just don't see what that way is. Please help me out. In my opinion, either a) multiple inheritance should be declared not Pythonic and frowned upon, or b) made easy to use and well supported In case of a), it would be nice to have proper facilities set up to have a good factory plugin mechanism (for example one that also allows to put in several plugins) Or in the b) case, it should be easier to use multiple inheritance and mixins. I was thinking about a class algebra, so following the above example one could create a mixed class simply as MyProtocol = SSLMixin + HTTPProtocol. This would lead do options (going back to asyncio) like asyncio.set_event_loop((PriorityMixin + QtEventLoop)()) What do you all think about that? Greetings Martin
Discerning use of multiple inheritance is very Pythonic. Look no further than Exception use cases: class PathAccessError(KeyError, IndexError, TypeError): <https://github.com/mahmoud/boltons/blob/master/boltons/iterutils.py#L880> It might look a little wonky at first, but when you understand the use case, you'll see the utility. There are loads of these sorts of use cases sprinkled throughout the community, in some of the most Pythonic libraries around. Mahmoud On Fri, Mar 18, 2016 at 11:11 AM, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
On 18.03.2016 19:11, Martin Teichmann wrote:
Is multiple inheritance bad, or not Pythonic?
Definitely not. But at the same time, it's not always the answer to everything either. Preventing subclassing of code is usually a conscious design decision in Python and has it's purpose as well, e.g. to reduce performance overhead by not having to worry about corner cases which the base code does not address. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 18 2016)
2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89 2016-02-19: Released eGenix PyRun 2.1.2 ... http://egenix.com/go88 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Hi,
Actually, I agree with you. But then I found the discussion about asyncio.BaseEventLoop.create_task (especially: https://github.com/python/asyncio/issues/208 and https://codereview.appspot.com/110820049/ ), which is now in the standard library, probably the most Pythonic code out there. During this discussion, all participants argue as if class inheritance was something EVIL. And they were not just random people, but in my opinion some of the finest Python programmers out there. They desperately try to avoid letting users inherit classes, even in cases where it seemed obvious to me that inheritance is the right choice - but well, who am I to decide that. So I asked myself, and hereby the Python community, is there something so bad about inheritance that I missed? The problem they wanted to solve is that the behavior of BaseEventLoop.create_task should be changeable. Obvious, I thought (and did in my code) inherit from the class, overwrite the method! But what if I want to overwrite that method in something that inherits from BaseEventLoop, say, some QtEventLoop? No problem, I thought, just supply a Mixin, and a simple class EventLoop(AMixin, QtEventLoop) does the job. This is even chainable, multiple mixins can be used as long as they properly call super(). Instead, the developers decided to write a set_task_factory method, which overwrites create_task. It has none of the above mentioned advantages. That's where I started thinking, and I am still puzzled what obstacles might be out there, and decided that inheritance must be something magically non Pythonic, and I am still wondering why that might be. Greetings Martin
On Mar 18, 2016, at 15:34, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
What makes asyncio the most Pythonic code out there, as opposed to other stdlib modules like enum, selectors, etc.?[1]
During this discussion, all participants argue as if class inheritance was something EVIL.
You're misinterpreting them. They believe that making people inherit from API classes and override internal methods is not the best way to allow customizing those APIs. That one particular use of inheritance is almost always a bad one. But that doesn't at all mean that all possible uses of inheritance are bad. For example, nobody suggested that it was evil for IntEnum to subclass both int and Enum--in fact, Enum was specifically designed (and debugged) to make that as easy as possible, and to allow IntEnum to serve as an example for user enum types. And a lot of the same people were involved in that discussion. So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are: 1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override.[2] With a set_spam_cheeser interface, it's obvious what they expected you to customize: just the spam-cheeser. 2. Who constructs the object? Often it's the framework itself, which means if the only way for you to customize framework.MeatCooker.spam_cheeser is to subclass, then you also need some way to tell framework.Kitchen to create an app.MyMeatCooker instead of a framework.MeatCooker. Which means you need a framework.KitchenFactory that you can tell to create an app.MyKitchen... Other times, the app creates the object, synthesizing customizations from various different third-party libs, but inheritance can cause its own problems there.[4] The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way. --- [1]: In fact, I suspect Guido would say that there are many ways in which asyncio is a special case, which is part of the reason he had to write a relatively complex stdlib module in the first place (rather than the usual pattern of letting different alternatives compete and then maybe considering adding a simplified version of the winner to the stdlib years later). [2]: And often, if you override methods the class designer didn't expect, it works in some cases but not others, so you design around it and then realize it doesn't work and isn't supposed to work, and then you end up having to override three other methods and copy and paste all but one line and then subclass an associated class and override two of its methods to fix the bug. [3]: ... which may be an interface with multiple methods, but most simply it's just one function. [4]: Admittedly, those problems aren't as bad in Python as in brace-family language--it just means you have to define and document a way for all those libraries and apps to cooperate properly, and write your code carefully so that bad cooperation can be debugged, which isn't all that hard. But it's still a lot harder than not doing it, and you don't have to do any of that with separate customization hooks. [5]: OK, there were some limitations before 2.2, but I think we can ignore those.
Hi Andrew, Hi List,
So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are:
thanks, that was a interesting read. Hopefully not just for me but also for others. I still have to let that sink in: inheritance, even multiple is fine, just not across a framework boundary, because you never know how the user is overriding the methods. For me, coming from a GUI background (Qt, that is) that's still astonishing, as there inherit-and-override is a concept used very often, especially across the framework boundary.
1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override.
In the GUI world, that's simply solved by documentation. Qt, and also PyQt, document very precisely which methods are supposed to be overridden, and how. And I still like that concept, as it has a lot of flexibility. You may call super(), or not, depending on what you want to do, even overriding a method several times in the inhertance chain is not uncommon, even across framework boundaries. With set_task_factory, I'm still trying to figure out how to call the base implementation, as super() won't do it. (Thats another topic, but if someone could tell me I would be really happy. I simply want to keep track of all created tasks, so I would simply love to call super().create_task(), log the created task and I'm done. No I cannot use Task(), as I don't know whether the EventLoop actually uses that.)
The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way.
What do you mean with the last sentence, what is that better way? Having a set_whatever_factory instead of inheritance, is that what you mean? Thanks for the enlightening post Martin
On 3/18/2016 6:34 PM, Martin Teichmann wrote:
The derived subclasses, SelectorEventLoop and ProactorEventLoop, can be subclassed. Four days ago I posted a trivial subclass that re-defined one method, .run_forever. This allows the use of tkinter and asyncio together. I hope it is not an accident that I was able to do so so easily ;-). -- Terry Jan Reedy
On Mar 18, 2016, at 11:11, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
That means you now have an anonymous class[1]--no name to show up in tracebacks or inspect, no way to jump to the class definition in IDEs, probably even no way to pickle instances unless you invent some new mechanism to make that work. Also, this conflicts with the notion of type algebra from other languages: the sum of two or more types is like a tagged union.[2] Also, it's not much of an algebra if the only operation is (non-commutative) addition. What does MyMixin * YourMixin, or 2 * MyMixin mean?[3] Also, + implies commutativity. That implication only goes so far (it's not true from string concatenation, obviously), but it's still going to flavor people's expectations. And finally, what's the benefit over this: class MyProtocol(SSLMixin, HTTPProtocol): pass Of course if you really want to do this, you can already just call the type function directly (although notice that even there, and even in the C API, you still have to pass some string in for the name...). Or, if you really want, create a metaclass that subclasses type and adds an __add__ method. (You could even write it as a "metaclass mixin"). But I think very little Pythonic code would want such a thing, so it's a bad idea to make it any easier than it is today. From a larger point of view, I don't think there's any problem to be solved here. Your question is like asking whether we should get rid of (single) inheritance or delegation. They're both useful, in mostly different contexts--there's some overlap, and there are also cases where you should be using one but could instead abuse the other, but that doesn't mean either one is bad and needs to be declared unpythonic. It means that both are pythonic within their range and unpythonic outside it (and when your use falls into the overlap, you have to make a judgment call). --- [1]: If you're thinking "but anonymous classes are good in Java", they're needed in Java to work around a problem Python has never had (how do you do closures in a language with no free functions?), and they're acceptable because nobody looks at the classes at runtime. [2]: That is, an object that can act as either a MyMixin or a YourMixin (after you check which) but never both at the same time. [3]: In other languages, these operations create product types--tuples or records.
Discerning use of multiple inheritance is very Pythonic. Look no further than Exception use cases: class PathAccessError(KeyError, IndexError, TypeError): <https://github.com/mahmoud/boltons/blob/master/boltons/iterutils.py#L880> It might look a little wonky at first, but when you understand the use case, you'll see the utility. There are loads of these sorts of use cases sprinkled throughout the community, in some of the most Pythonic libraries around. Mahmoud On Fri, Mar 18, 2016 at 11:11 AM, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
On 18.03.2016 19:11, Martin Teichmann wrote:
Is multiple inheritance bad, or not Pythonic?
Definitely not. But at the same time, it's not always the answer to everything either. Preventing subclassing of code is usually a conscious design decision in Python and has it's purpose as well, e.g. to reduce performance overhead by not having to worry about corner cases which the base code does not address. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 18 2016)
2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89 2016-02-19: Released eGenix PyRun 2.1.2 ... http://egenix.com/go88 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Hi,
Actually, I agree with you. But then I found the discussion about asyncio.BaseEventLoop.create_task (especially: https://github.com/python/asyncio/issues/208 and https://codereview.appspot.com/110820049/ ), which is now in the standard library, probably the most Pythonic code out there. During this discussion, all participants argue as if class inheritance was something EVIL. And they were not just random people, but in my opinion some of the finest Python programmers out there. They desperately try to avoid letting users inherit classes, even in cases where it seemed obvious to me that inheritance is the right choice - but well, who am I to decide that. So I asked myself, and hereby the Python community, is there something so bad about inheritance that I missed? The problem they wanted to solve is that the behavior of BaseEventLoop.create_task should be changeable. Obvious, I thought (and did in my code) inherit from the class, overwrite the method! But what if I want to overwrite that method in something that inherits from BaseEventLoop, say, some QtEventLoop? No problem, I thought, just supply a Mixin, and a simple class EventLoop(AMixin, QtEventLoop) does the job. This is even chainable, multiple mixins can be used as long as they properly call super(). Instead, the developers decided to write a set_task_factory method, which overwrites create_task. It has none of the above mentioned advantages. That's where I started thinking, and I am still puzzled what obstacles might be out there, and decided that inheritance must be something magically non Pythonic, and I am still wondering why that might be. Greetings Martin
On Mar 18, 2016, at 15:34, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
What makes asyncio the most Pythonic code out there, as opposed to other stdlib modules like enum, selectors, etc.?[1]
During this discussion, all participants argue as if class inheritance was something EVIL.
You're misinterpreting them. They believe that making people inherit from API classes and override internal methods is not the best way to allow customizing those APIs. That one particular use of inheritance is almost always a bad one. But that doesn't at all mean that all possible uses of inheritance are bad. For example, nobody suggested that it was evil for IntEnum to subclass both int and Enum--in fact, Enum was specifically designed (and debugged) to make that as easy as possible, and to allow IntEnum to serve as an example for user enum types. And a lot of the same people were involved in that discussion. So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are: 1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override.[2] With a set_spam_cheeser interface, it's obvious what they expected you to customize: just the spam-cheeser. 2. Who constructs the object? Often it's the framework itself, which means if the only way for you to customize framework.MeatCooker.spam_cheeser is to subclass, then you also need some way to tell framework.Kitchen to create an app.MyMeatCooker instead of a framework.MeatCooker. Which means you need a framework.KitchenFactory that you can tell to create an app.MyKitchen... Other times, the app creates the object, synthesizing customizations from various different third-party libs, but inheritance can cause its own problems there.[4] The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way. --- [1]: In fact, I suspect Guido would say that there are many ways in which asyncio is a special case, which is part of the reason he had to write a relatively complex stdlib module in the first place (rather than the usual pattern of letting different alternatives compete and then maybe considering adding a simplified version of the winner to the stdlib years later). [2]: And often, if you override methods the class designer didn't expect, it works in some cases but not others, so you design around it and then realize it doesn't work and isn't supposed to work, and then you end up having to override three other methods and copy and paste all but one line and then subclass an associated class and override two of its methods to fix the bug. [3]: ... which may be an interface with multiple methods, but most simply it's just one function. [4]: Admittedly, those problems aren't as bad in Python as in brace-family language--it just means you have to define and document a way for all those libraries and apps to cooperate properly, and write your code carefully so that bad cooperation can be debugged, which isn't all that hard. But it's still a lot harder than not doing it, and you don't have to do any of that with separate customization hooks. [5]: OK, there were some limitations before 2.2, but I think we can ignore those.
Hi Andrew, Hi List,
So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are:
thanks, that was a interesting read. Hopefully not just for me but also for others. I still have to let that sink in: inheritance, even multiple is fine, just not across a framework boundary, because you never know how the user is overriding the methods. For me, coming from a GUI background (Qt, that is) that's still astonishing, as there inherit-and-override is a concept used very often, especially across the framework boundary.
1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override.
In the GUI world, that's simply solved by documentation. Qt, and also PyQt, document very precisely which methods are supposed to be overridden, and how. And I still like that concept, as it has a lot of flexibility. You may call super(), or not, depending on what you want to do, even overriding a method several times in the inhertance chain is not uncommon, even across framework boundaries. With set_task_factory, I'm still trying to figure out how to call the base implementation, as super() won't do it. (Thats another topic, but if someone could tell me I would be really happy. I simply want to keep track of all created tasks, so I would simply love to call super().create_task(), log the created task and I'm done. No I cannot use Task(), as I don't know whether the EventLoop actually uses that.)
The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way.
What do you mean with the last sentence, what is that better way? Having a set_whatever_factory instead of inheritance, is that what you mean? Thanks for the enlightening post Martin
On 3/18/2016 6:34 PM, Martin Teichmann wrote:
The derived subclasses, SelectorEventLoop and ProactorEventLoop, can be subclassed. Four days ago I posted a trivial subclass that re-defined one method, .run_forever. This allows the use of tkinter and asyncio together. I hope it is not an accident that I was able to do so so easily ;-). -- Terry Jan Reedy
On Mar 18, 2016, at 11:11, Martin Teichmann <lkb.teichmann@gmail.com> wrote:
That means you now have an anonymous class[1]--no name to show up in tracebacks or inspect, no way to jump to the class definition in IDEs, probably even no way to pickle instances unless you invent some new mechanism to make that work. Also, this conflicts with the notion of type algebra from other languages: the sum of two or more types is like a tagged union.[2] Also, it's not much of an algebra if the only operation is (non-commutative) addition. What does MyMixin * YourMixin, or 2 * MyMixin mean?[3] Also, + implies commutativity. That implication only goes so far (it's not true from string concatenation, obviously), but it's still going to flavor people's expectations. And finally, what's the benefit over this: class MyProtocol(SSLMixin, HTTPProtocol): pass Of course if you really want to do this, you can already just call the type function directly (although notice that even there, and even in the C API, you still have to pass some string in for the name...). Or, if you really want, create a metaclass that subclasses type and adds an __add__ method. (You could even write it as a "metaclass mixin"). But I think very little Pythonic code would want such a thing, so it's a bad idea to make it any easier than it is today. From a larger point of view, I don't think there's any problem to be solved here. Your question is like asking whether we should get rid of (single) inheritance or delegation. They're both useful, in mostly different contexts--there's some overlap, and there are also cases where you should be using one but could instead abuse the other, but that doesn't mean either one is bad and needs to be declared unpythonic. It means that both are pythonic within their range and unpythonic outside it (and when your use falls into the overlap, you have to make a judgment call). --- [1]: If you're thinking "but anonymous classes are good in Java", they're needed in Java to work around a problem Python has never had (how do you do closures in a language with no free functions?), and they're acceptable because nobody looks at the classes at runtime. [2]: That is, an object that can act as either a MyMixin or a YourMixin (after you check which) but never both at the same time. [3]: In other languages, these operations create product types--tuples or records.
participants (5)
-
Andrew Barnert -
M.-A. Lemburg -
Mahmoud Hashemi -
Martin Teichmann -
Terry Reedy