Recently I have opened a blog on Artima and I am publishing a few Python-related essays I had in store. In particular a trilogy of papers about "super". From the foreword: """ In 2004 I decided to write a comprehensive paper documenting ``super`` pitfalls and traps, with the goal of publishing it on the Python web site, just as I had published my essay on multiple inheritance and the `Method Resolution Order`_. With time the paper grew longer and longer but I never had the feeling that I had covered everything I needed to say: moreover I have a full time job, so I never had the time to fully revise the paper as a whole. As a consequence, four years have passed and the paper is still in draft status. This is a pity, since it documents issues that people encounter and that regularly come out on the Python newsgroups and forums. Keeping the draft sitting on my hard disk is doing a disservice to the community. Still, I lack to time to finish it properly. To come out from the impasse, I decided to split the long paper in a series of short blog posts, which I do have the time to review properly. Moreover people are free to post comments and corrections in case I am making mistakes (speaking about ``super`` this is always possible). Once I finish the series, I may integrate the corrections, put it together again and possibly publish it as whole on the Python website. In other words, in order to finish the task, I am trying the strategies of *divide et conquer* and *release early, release often*. We will see how it goes. """ It seems the strategy worked since I have finished the paper, by splitting it in three post. Actually, it it not really finished because it does not say anything about the new super in Python 3.0, but that could do in a separate essay. Here I am asking for feedback/corrections. Moreover if people think it is worthy idea, I can re-assemble the paper again, donate it to the PSF and publish it on the Python website. Michele Simionato
On Aug 24, 3:43 pm, "Matt Giuca" <matt.gi...@gmail.com> wrote:
Hi Michele,
Do you have a URL for this blog?
Sorry, here it is: http://www.artima.com/weblogs/index.jsp?blogger=micheles
[Michele Simionato]
Thanks for posting these blogs. I think they serve as a nice compilation of docs, original analysis, and various discussions on super(). What I would really like to see is a fourth blog entry that shows how to use super() reliably and correctly. In general, I opposed to creating documentation in the form of "danger, danger this could explode ...." IMO, there is not much point in dwelling on bugs that have already been fixed, nor is there an advantage to showing all the ways the tool can be misused. In applications without multiple inheritance, it is a straight-forward to use super() as a way to avoid directly naming an immediate parent class. This is useful in and of itself. For cooperative multiple inheritance, I take issue with the abstracted examples provided (i.e. inconsistent signatures). In a real app that actually needs cooperative multiple inheritance, it becomes self-evident what "cooperative" actually means -- the methods *have* to be designed to interoperate -- it is intrinsic to the problem at hand. Your reasoning is akin to saying that cooperative multitasking is intrinsically flawed because one of the tasks could be designed to not cooperate (never yield). Cooperative multiple inheritance is *not* about mixing two unrelated parents that just happen to use the same method name but have different semantics and were not designed to cooperate with each other. The A-B-C-D diagrams and foo/bar methods in the examples are deceptive because they silently drop the precondition of cooperation while attempting to demonstrate a supposed flaw in the tool. If I understand the problem correctly, in the rare cases where you do need cooperative multiple inheritance, then super() is the only workable solution short of designing some equivalent using composition instead of inheritance. I do agree with you that the docs can be improved. I'll work on patch that makes clear that super() returns a proxy-object (not an actual parent class) that dispatches explict method calls (not syntax driven) to one of multiple parents designed to interact cooperatively. Also, it may be controversial, but there may be some merit in de-documenting the "unbound" case. It seems to add more confusion than it's worth. Lastly, I take issue with one other part of the blogs. While they show a clear dislike for cooperative multiple inheritance, they take a potshot at multiple inheritance in general. I don't follow the logic here. IMO, mixin classes like DictMixin have proven themselves as being very useful. Plenty of frameworks share this approach. Likewise, the new ABCs offer mixin capabilities that are really nice. I think it is a non-sequiter to reason from "diamond diagrams are complicated" to "mixins should be disallowed". Instead, I think it better to simply recommend that a key to happiness is to keep various mixin classes completely orthogonal to one another (no overlapping method names). Lest I sound negative, let me thank you again for the blog entries and for compiling the most complete discussion of it in one place. Raymond
On Tue, Aug 26, 2008 at 5:32 PM, Raymond Hettinger <python@rcn.com> wrote:
What I would really like to see is a fourth blog entry that shows how to use super() reliably and correctly.
That could be arranged.
Yep. The parts about the bugs of super in 2.2 and 2.3 were written years ago, when they were relevant. Nowadays they are less relevant, but since they were already written and since there are still people using older versions of Python I decided to keep them. I would not keep them in a revised version intended as "semi-official" documentation of super. Still, I think they are fine as a blog post.
They just show that the tool is delicate and not easy to use.
In my experience, one can go a long way using composition instead of inheritance. I also think that Python would not lose much without cooperative multiple inheritance. This is however a personal opinion and in any case the point is moot because the language is the way it is. Still, in a blog post personal opinions and even rants have their place. That part could be removed in an "semi-official" document.
Also, it may be controversial, but there may be some merit in de-documenting the "unbound" case. It seems to add more confusion than it's worth.
Fine with me.
I not completely against multiple inheritance. I am against multiple inheritance as it is now. A restricted form of multiple inheritance in which mixins classes are guaranteed to be orthogonal would be fine with me (provided it is not abused). This concept exists already in other languages, the orthogonal mixins are called "traits". I have written a trilogy of papers about mixins: if you read them, you will see where I come from (Zope, which I do not like) and you will also see that I like DictMixin instead. I will publish the papers in the blog soon or later, but you can find the Italian version here: http://stacktrace.it/articoli/2008/06/i-pericoli-della-programmazione-con-i-... http://stacktrace.it/articoli/2008/07/i-pericoli-della-programmazione-con-i-... http://stacktrace.it/articoli/2008/08/i-pericoli-della-programmazione-con-i-... Michele Simionato
[Raymond]
[Michele]
They just show that the tool is delicate and not easy to use.
To me, they miss the point. Simply, if you don't have diamonds, then super() is easy to use and if you do have have diamonds, then super() is the *only* way to do it. Diamonds impose a set of design constraints (making the classes cooperative). The A-B-C-D diagrams ignore this point and make super() seem like an accident waiting to happen. In contrast, with a concrete example, accidental non-cooperativeness would be more self-evident. The problem isn't that super() is fragile. The problem is that a cooperative design pattern requires you to actually make the classes cooperate. This is no different than a visitor design pattern requiring that you attach a conformant visit() method for on each of the visited classes. [Michele]
In my experience, one can go a long way using composition instead of inheritance.
While that is often true, I don't think it applies to the case of cooperative multiple inheritence. To achieve substantially the same functionality using composition, you would likely have to re-invent much of what super() does for us automatically, and you would still be imposing constraits on the composed classes that are substantially the same as what you have with inheritance. [Michele]
I also think that Python would not lose much without cooperative multiple inheritance.
I would state this differently: "The use cases for cooperative multiple inheritence don't arise often in practice; so, if we dropped support for those cases, you probably wouldn't notice until you encountered one of the rare occasions where it was the right answer to your problem." There was some quote floating around that expressed the situation well -- it went something like: "Python makes most problems easy and hard problems possible". The use cases for cooperative multiple inheritance fall in the latter category. BTW, I really like your paper explaining the MRO. Excellent work. Raymond
On Tue, Aug 26, 2008 at 8:56 PM, Raymond Hettinger <python@rcn.com> wrote:
It is just a matter of how rare the use cases really are. Cooperative methods has been introduced 6+ years ago. In all this time surely they must have been used. How many compelling uses of cooperation we can find in real life code? For instance in the standard library or in some well known framework? This is a serious question I have been wanting to ask for years. I am sure people here can find some example, so just give me a pointer and we will see.
BTW, I really like your paper explaining the MRO. Excellent work.
The issue with that paper is that I wrote it when my Python experience was reduced to six month and my experience with real life large object oriented frameworks was zero. Nowadays I value simplicity more.
On Tue, Aug 26, 2008 at 6:16 PM, Michele Simionato <michele.simionato@gmail.com> wrote: ...
http://www.koders.com/default.aspx?s=super&btn=&la=Python&li=* finds over 5,000 hits, but it would take substantial work to sift through them (in particular because not all refer to the built-in super, as you'll see even in the first page!) -- and a random hit I found by going to p.7 is really bad...: """Mixin to enable reification.""" def __init__(self): super(ReificationStore, self).__init__() [there's *nothing else* in this __init__!]. Alex
On Wed, Aug 27, 2008 at 3:30 AM, Alex Martelli <aleaxit@gmail.com> wrote:
Yep. Notice (I am sure you understood the point correctly, but just to clarify) that I am not interested in random occurrences of super, but in code/frameworks expressly designed to leverage on cooperation and doing it in a compelling way. IOW, I want to see cases where using cooperation is really better than relying on other techniques. Guido gives an example in http://www.python.org/download/releases/2.2.3/descrintro/#cooperation with a .save method, so in theory there are good use cases, but I wonder in practice how common they are and if they are frequent enough to justify the added complication. M.S.
Michele Simionato wrote:
If you aren't aware of it you should take a look at Enthought's traits package. It's part of the Enthought Tool Suite (ETS). https://svn.enthought.com/enthought/wiki While I too appreciate your comments about super I believe you have perhaps overdone it. I do look forward to seeing the edited edition as a part of the documentation. [Hint: the *docs* aren't in feature freeze ;-)] regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On Tue, Aug 26, 2008 at 11:10 PM, Steve Holden <steve@holdenweb.com> wrote:
If you aren't aware of it you should take a look at Enthought's traits package. It's part of the Enthought Tool Suite (ETS).
I know of the existence of that framework, however it is quite large and I don't see the relation with the concept of traits I have in mind, which is more or less the one described here: http://www.iam.unibe.ch/%7Escg/Archive/Papers/Scha03aTraits.pdf Basically, these are the properties of traits: 1. the methods/attributes in a trait go logically together; 2. if a trait enhances a class, then all subclasses are enhanced too; 3. if a trait has methods in common with the class, then the methods defined in the class have the precedence; 4. the ordering of traits is not important, i.e. enhancing a class first with trait T1 and then with trait T2 or viceversa is the same; 5. if traits T1 and T2 have names in common, enhancing a class both with T1 and T2 raises an error unless there is an explicitoverriding; 6. if a trait has methods in common with the base class, then the trait methods have the precedence; Properties from 4 to 6 are the distinguishing properties of traits with respect to multiple inheritance and mixins.
On Tue, Aug 26, 2008 at 6:13 PM, Michele Simionato <michele.simionato@gmail.com> wrote:
I must correct myself here. Even if for practical purposes traits look like a restricted multiple inheritance, in principle it is better to think of them as of an enhanced single inheritance. With traits there is always a single superclass: traits are just single inheritance with a nice syntax to include methods (like in Ruby) and a guarantee that methods will not be overridden silently (this one is missing in Ruby). M.S.
On Aug 24, 3:43 pm, "Matt Giuca" <matt.gi...@gmail.com> wrote:
Hi Michele,
Do you have a URL for this blog?
Sorry, here it is: http://www.artima.com/weblogs/index.jsp?blogger=micheles
[Michele Simionato]
Thanks for posting these blogs. I think they serve as a nice compilation of docs, original analysis, and various discussions on super(). What I would really like to see is a fourth blog entry that shows how to use super() reliably and correctly. In general, I opposed to creating documentation in the form of "danger, danger this could explode ...." IMO, there is not much point in dwelling on bugs that have already been fixed, nor is there an advantage to showing all the ways the tool can be misused. In applications without multiple inheritance, it is a straight-forward to use super() as a way to avoid directly naming an immediate parent class. This is useful in and of itself. For cooperative multiple inheritance, I take issue with the abstracted examples provided (i.e. inconsistent signatures). In a real app that actually needs cooperative multiple inheritance, it becomes self-evident what "cooperative" actually means -- the methods *have* to be designed to interoperate -- it is intrinsic to the problem at hand. Your reasoning is akin to saying that cooperative multitasking is intrinsically flawed because one of the tasks could be designed to not cooperate (never yield). Cooperative multiple inheritance is *not* about mixing two unrelated parents that just happen to use the same method name but have different semantics and were not designed to cooperate with each other. The A-B-C-D diagrams and foo/bar methods in the examples are deceptive because they silently drop the precondition of cooperation while attempting to demonstrate a supposed flaw in the tool. If I understand the problem correctly, in the rare cases where you do need cooperative multiple inheritance, then super() is the only workable solution short of designing some equivalent using composition instead of inheritance. I do agree with you that the docs can be improved. I'll work on patch that makes clear that super() returns a proxy-object (not an actual parent class) that dispatches explict method calls (not syntax driven) to one of multiple parents designed to interact cooperatively. Also, it may be controversial, but there may be some merit in de-documenting the "unbound" case. It seems to add more confusion than it's worth. Lastly, I take issue with one other part of the blogs. While they show a clear dislike for cooperative multiple inheritance, they take a potshot at multiple inheritance in general. I don't follow the logic here. IMO, mixin classes like DictMixin have proven themselves as being very useful. Plenty of frameworks share this approach. Likewise, the new ABCs offer mixin capabilities that are really nice. I think it is a non-sequiter to reason from "diamond diagrams are complicated" to "mixins should be disallowed". Instead, I think it better to simply recommend that a key to happiness is to keep various mixin classes completely orthogonal to one another (no overlapping method names). Lest I sound negative, let me thank you again for the blog entries and for compiling the most complete discussion of it in one place. Raymond
On Tue, Aug 26, 2008 at 5:32 PM, Raymond Hettinger <python@rcn.com> wrote:
What I would really like to see is a fourth blog entry that shows how to use super() reliably and correctly.
That could be arranged.
Yep. The parts about the bugs of super in 2.2 and 2.3 were written years ago, when they were relevant. Nowadays they are less relevant, but since they were already written and since there are still people using older versions of Python I decided to keep them. I would not keep them in a revised version intended as "semi-official" documentation of super. Still, I think they are fine as a blog post.
They just show that the tool is delicate and not easy to use.
In my experience, one can go a long way using composition instead of inheritance. I also think that Python would not lose much without cooperative multiple inheritance. This is however a personal opinion and in any case the point is moot because the language is the way it is. Still, in a blog post personal opinions and even rants have their place. That part could be removed in an "semi-official" document.
Also, it may be controversial, but there may be some merit in de-documenting the "unbound" case. It seems to add more confusion than it's worth.
Fine with me.
I not completely against multiple inheritance. I am against multiple inheritance as it is now. A restricted form of multiple inheritance in which mixins classes are guaranteed to be orthogonal would be fine with me (provided it is not abused). This concept exists already in other languages, the orthogonal mixins are called "traits". I have written a trilogy of papers about mixins: if you read them, you will see where I come from (Zope, which I do not like) and you will also see that I like DictMixin instead. I will publish the papers in the blog soon or later, but you can find the Italian version here: http://stacktrace.it/articoli/2008/06/i-pericoli-della-programmazione-con-i-... http://stacktrace.it/articoli/2008/07/i-pericoli-della-programmazione-con-i-... http://stacktrace.it/articoli/2008/08/i-pericoli-della-programmazione-con-i-... Michele Simionato
[Raymond]
[Michele]
They just show that the tool is delicate and not easy to use.
To me, they miss the point. Simply, if you don't have diamonds, then super() is easy to use and if you do have have diamonds, then super() is the *only* way to do it. Diamonds impose a set of design constraints (making the classes cooperative). The A-B-C-D diagrams ignore this point and make super() seem like an accident waiting to happen. In contrast, with a concrete example, accidental non-cooperativeness would be more self-evident. The problem isn't that super() is fragile. The problem is that a cooperative design pattern requires you to actually make the classes cooperate. This is no different than a visitor design pattern requiring that you attach a conformant visit() method for on each of the visited classes. [Michele]
In my experience, one can go a long way using composition instead of inheritance.
While that is often true, I don't think it applies to the case of cooperative multiple inheritence. To achieve substantially the same functionality using composition, you would likely have to re-invent much of what super() does for us automatically, and you would still be imposing constraits on the composed classes that are substantially the same as what you have with inheritance. [Michele]
I also think that Python would not lose much without cooperative multiple inheritance.
I would state this differently: "The use cases for cooperative multiple inheritence don't arise often in practice; so, if we dropped support for those cases, you probably wouldn't notice until you encountered one of the rare occasions where it was the right answer to your problem." There was some quote floating around that expressed the situation well -- it went something like: "Python makes most problems easy and hard problems possible". The use cases for cooperative multiple inheritance fall in the latter category. BTW, I really like your paper explaining the MRO. Excellent work. Raymond
On Tue, Aug 26, 2008 at 8:56 PM, Raymond Hettinger <python@rcn.com> wrote:
It is just a matter of how rare the use cases really are. Cooperative methods has been introduced 6+ years ago. In all this time surely they must have been used. How many compelling uses of cooperation we can find in real life code? For instance in the standard library or in some well known framework? This is a serious question I have been wanting to ask for years. I am sure people here can find some example, so just give me a pointer and we will see.
BTW, I really like your paper explaining the MRO. Excellent work.
The issue with that paper is that I wrote it when my Python experience was reduced to six month and my experience with real life large object oriented frameworks was zero. Nowadays I value simplicity more.
On Tue, Aug 26, 2008 at 6:16 PM, Michele Simionato <michele.simionato@gmail.com> wrote: ...
http://www.koders.com/default.aspx?s=super&btn=&la=Python&li=* finds over 5,000 hits, but it would take substantial work to sift through them (in particular because not all refer to the built-in super, as you'll see even in the first page!) -- and a random hit I found by going to p.7 is really bad...: """Mixin to enable reification.""" def __init__(self): super(ReificationStore, self).__init__() [there's *nothing else* in this __init__!]. Alex
On Wed, Aug 27, 2008 at 3:30 AM, Alex Martelli <aleaxit@gmail.com> wrote:
Yep. Notice (I am sure you understood the point correctly, but just to clarify) that I am not interested in random occurrences of super, but in code/frameworks expressly designed to leverage on cooperation and doing it in a compelling way. IOW, I want to see cases where using cooperation is really better than relying on other techniques. Guido gives an example in http://www.python.org/download/releases/2.2.3/descrintro/#cooperation with a .save method, so in theory there are good use cases, but I wonder in practice how common they are and if they are frequent enough to justify the added complication. M.S.
Michele Simionato wrote:
If you aren't aware of it you should take a look at Enthought's traits package. It's part of the Enthought Tool Suite (ETS). https://svn.enthought.com/enthought/wiki While I too appreciate your comments about super I believe you have perhaps overdone it. I do look forward to seeing the edited edition as a part of the documentation. [Hint: the *docs* aren't in feature freeze ;-)] regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On Tue, Aug 26, 2008 at 11:10 PM, Steve Holden <steve@holdenweb.com> wrote:
If you aren't aware of it you should take a look at Enthought's traits package. It's part of the Enthought Tool Suite (ETS).
I know of the existence of that framework, however it is quite large and I don't see the relation with the concept of traits I have in mind, which is more or less the one described here: http://www.iam.unibe.ch/%7Escg/Archive/Papers/Scha03aTraits.pdf Basically, these are the properties of traits: 1. the methods/attributes in a trait go logically together; 2. if a trait enhances a class, then all subclasses are enhanced too; 3. if a trait has methods in common with the class, then the methods defined in the class have the precedence; 4. the ordering of traits is not important, i.e. enhancing a class first with trait T1 and then with trait T2 or viceversa is the same; 5. if traits T1 and T2 have names in common, enhancing a class both with T1 and T2 raises an error unless there is an explicitoverriding; 6. if a trait has methods in common with the base class, then the trait methods have the precedence; Properties from 4 to 6 are the distinguishing properties of traits with respect to multiple inheritance and mixins.
On Tue, Aug 26, 2008 at 6:13 PM, Michele Simionato <michele.simionato@gmail.com> wrote:
I must correct myself here. Even if for practical purposes traits look like a restricted multiple inheritance, in principle it is better to think of them as of an enhanced single inheritance. With traits there is always a single superclass: traits are just single inheritance with a nice syntax to include methods (like in Ruby) and a guarantee that methods will not be overridden silently (this one is missing in Ruby). M.S.
participants (6)
-
Alex Martelli -
Matt Giuca -
Michele Simionato -
michele.simionato＠gmail.com
-
Raymond Hettinger -
Steve Holden