[issue37176] super() docs don't say what super() does
New submission from Jeroen Demeyer <J.Demeyer@UGent.be>: The documentation for super() at https://docs.python.org/3.8/library/functions.html#super does not actually say what super() does. It only says "Return a proxy object that delegates method calls to a parent or sibling class of type" and then gives a bunch of use cases and examples. If there is one place where we should define exactly what super() does (as opposed to give guidance on how to use it), the stdlib reference should be it. ---------- assignee: docs@python components: Documentation messages: 344827 nosy: docs@python, jdemeyer priority: normal severity: normal status: open title: super() docs don't say what super() does type: enhancement versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment: The docs do say what super does: it returns "a proxy object that delegates method calls to a parent or sibling class of type", just as you quoted. That concise description is (almost) completely accurate and precise. (I say *almost* because it's not just method calls that it works with, but any attribute lookup.) What more do you want? That's not a rhetorical question. I'm not saying that the docs are perfect or cannot be improved, but they look pretty good to me: they tell you what super does, they tell you why you might use it, and show how to use it. What's missing? (Again, not a rhetorical question.) I understand that super is a very advanced corner of the language: there's a lot of necessary technical jargon in the docs: - One already needs to have a good understanding of what super *does* in order to make sense of the sentence describing what it *is*, so there's an element of circular reasoning needed. - The reader needs to understand that super isn't magical, it just returns an object like any other function, and understand what "proxy object" means, as well as delegation. - And have an understanding of how inheritance and the MRO work in Python, and the difference between bound and unbound methods/proxies. But to the experienced reader who knows these concepts, I'm having a hard time seeing what you believe is missing from the docs. Perhaps there ought to be a "gentle guide to super" somewhere, and the docs could link to that? ---------- nosy: +steven.daprano _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Change by Steven D'Aprano <steve+python@pearwood.info>: ---------- Removed message: https://bugs.python.org/msg344892 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment: The docs do say what super does: it returns "a proxy object that delegates method calls to a parent or sibling class of type", just as you quoted. That concise description is (almost) completely accurate and precise. (I say *almost* because it's not just method calls that it works with, but any attribute lookup.) What more do you want? That's not a rhetorical question. I'm not saying that the docs are perfect or cannot be improved, but they look pretty good to me: they tell you what super does, they tell you why you might use it, and show how to use it. What's missing? Again, not a rhetorical question. I understand that super is a very advanced corner of the language: there's a lot of necessary technical jargon in the docs, e.g.: - One already needs to have a good understanding of what super *does* in order to make sense of the sentence describing what it *is*, so there's an element of circular reasoning needed. - The reader needs to understand that super isn't magical, it just returns an object like any other function, and understand what "proxy object" means, as well as delegation. - And have an understanding of how inheritance and the MRO work in Python, and the difference between bound and unbound methods/proxies. I think that to the experienced reader who knows these concepts, the docs should be pretty clear and complete. I'm having a hard time seeing what you believe is missing from the docs. Can you explain further? Perhaps there ought to be a "gentle guide to super" somewhere, and the docs could link to that? ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:
It only says "Return a proxy object that delegates method calls to a parent or sibling class of type" and then gives a bunch of use cases and examples."
That wording seems very reasonable to me. ---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
What more do you want?
Mainly: it says "a parent or sibling class of *type*" but it doesn't explain which class it actually uses. And the sentence "The __mro__ attribute of the type lists the method resolution search order used by both getattr() and super()" is even wrong or at least confusing: what matters is not the MRO of the type (the first argument to super()) but the MRO of the object (the second argument to super()). The zero-argument form super() is not explained at all.
Perhaps there ought to be a "gentle guide to super" somewhere, and the docs could link to that?
There are plenty of guides like that and in fact that docs already link to https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment: On Fri, Jun 07, 2019 at 08:00:34AM +0000, Jeroen Demeyer wrote:
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
What more do you want?
Mainly: it says "a parent or sibling class of *type*" but it doesn't explain which class it actually uses.
Only one of the two arguments is called "type". The other is called "object-or-type".
And the sentence "The __mro__ attribute of the type lists the method resolution search order used by both getattr() and super()" is even wrong or at least confusing: what matters is not the MRO of the type (the first argument to super()) but the MRO of the object (the second argument to super()).
The sentence doesn't talk about the MRO of *type* (the first argument), it talks about the __mro__ attribute. It is correct. The __mro__ attribute is on the type, not the instance: py> int.__mro__ (<class 'int'>, <class 'object'>) py> int().__mro__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'int' object has no attribute '__mro__' super() look-ups never start at the instance, they delegate to the superclass (or superclasses), not back to the current class, or the instance itself.
The zero-argument form super() is not explained at all.
Yes it is. Look at the example given: super().method(arg) # This does the same thing as: # super(C, self).method(arg) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
Only one of the two arguments is called "type". The other is called "object-or-type".
I'm having problems with the first word of "a parent or sibling class of type". The most important part of super() is to *which* class that attribute lookups are delegated to and this is not explained.
The __mro__ attribute is on the type, not the instance:
Sorry for that. I meant to say: And the sentence "The __mro__ attribute of the type lists the method resolution search order used by both getattr() and super()" is even wrong or at least confusing: what matters is not the MRO of the type (the first argument to super()) but the MRO of ***the type of*** the object (the second argument to super()).
Yes it is. Look at the example given:
An example is not an explanation. But it's true, this is the least of my problems with this doc. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment: On Fri, Jun 07, 2019 at 09:57:11AM +0000, Jeroen Demeyer wrote:
I'm having problems with the first word of "a parent or sibling class of type".
The first word is "a". Did you mean something else or did you mean it literally? If the second case, we have to talk about *a* parent class rather than *the* parent class, since Python supports multiple inheritance and a class can have more than one parent class.
The most important part of super() is to *which* class that attribute lookups are delegated to and this is not explained.
Lookups are delegated to a parent or sibling class, exactly as the documentation says. Which class that is (whether a parent or sibling) can only be determined at runtime.
And the sentence "The __mro__ attribute of the type lists the method resolution search order used by both getattr() and super()" is even wrong or at least confusing: what matters is not the MRO of the type (the first argument to super()) but the MRO of ***the type of*** the object (the second argument to super()).
I believe that is still wrong. What matters is the __mro__ attribute of the first argument. It matters because that is how the MRO actually is searched. The MRO of the second argument is not relevant, except to the degree that it overlaps with the __mro__ attribute of the first. And it will always overlaps, because the second argument must be an instance or subclass of the first. But not necessarily *directly* of the first: object > Parent > A > B > C > D > E > F > instance = F() If the method is defined in class C, then the super call in C should be written: class C(B): def method(self, arg): super().method(arg) # like super(C, self).method(arg) and the lookup will start at C.__mro__[1], namely B, precisely as expected. If it started back at the type of the instance (self), namely class F, the C.method would recursively call itself over and over and over again. If you don't believe me, you can simulate super() working the way you suggest by doing this: # -----%<----- class A(object): def method(self): print("from class A") class B(A): def method(self): print("bad method") # This is bad, don't do it! super(type(self), self).method() class C(B): pass C().method() # -----%<----- For multiple inheritance: class C(B, X, Y, Z): ... we can't generally tell in advance which will be looked at next, since that depends on the precise details of the inheritance diagram. But whatever the class is, at this point, it is still the __mro__ of C that is used, not the MRO of the instance. The bottom line: what matters is <first argument>.__mro__, exactly as documented. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
The sentence doesn't talk about the MRO of *type* (the first argument), it talks about the __mro__ attribute.
If you have to explain in a bpo issue how the doc should be read, that proves exactly my point that it's confusing. The fact that it's technically correct if you read it the right way is irrelevant. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment:
If you have to explain in a bpo issue how the doc should be read, that proves exactly my point that it's confusing. The fact that it's technically correct if you read it the right way is irrelevant.
Do you expect the docs to be technically correct when read wrongly? How else should people read the docs, except the right way? If you have *concrete* suggestions for improvements, of course we will consider them. I'll start with two concrete improvements: 1. Explicitly document that the zero-argument version is equivalent to super(__class__, <first argument (usually self)>), and note that __class__ here is a special variable populated by the compiler. The first part of this is already documented in super.__doc__ and the second part used to be documented: https://docs.python.org/3.1/library/functions.html?#super and I don't think there's any harm in adding it back in. 2. Link to the essay on how the MRO is calculated. https://www.python.org/download/releases/2.3/mro/ ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
What matters is the __mro__ attribute of the first argument. It matters because that is how the MRO actually is searched.
I'm sorry to say that you're wrong here. super() looks at the MRO of the type of the object (the second argument) (*). It has to do that in order to support diamonds. Consider a diamond like D / \ B C \ / A (with A as common base class). Now super(B, D()).attr will look in the MRO of D (which is D, B, C, A) and therefore delegate to C.attr. In this case, C does not even appear in the MRO of B. (*) To be pedantic: in the special case that the second argument is a type itself, it looks at the MRO of the second argument. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment:
I'm sorry to say that you're wrong here.
I'm happy to be corrected. It is fair to say I failed to take the multiple inheritance case into account. Clearly super can't *only* look at the MRO of the first object since that will miss the multiple inheritance case, as you point out. Thank you. But neither can it *only* look at the MRO of the second class, because that would restart the search at the top of the hierarchy; also if type(second argument) was the only thing that mattered, that would make the first argument redundant and pointless. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment: And this last comment is precisely the kind of information which should be explained in the super() docs. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Géry <gery.ogam@gmail.com> added the comment: @Jeroen Demeyer
I'm sorry to say that you're wrong here. super() looks at the MRO of the type of the object (the second argument) (*).
Exactly! It is funny because I was about to open the same issue this weekend. The documentation of super() is wrong here. ---------- nosy: +maggyero _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Géry <gery.ogam@gmail.com> added the comment: @Steven D'Aprano
What matters is the __mro__ attribute of the first argument. It matters because that is how the MRO actually is searched.
By the way, if it was true (it is not), then what did you think was the purpose of the second parameter of super(type, obj-or-type)? ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment:
What matters is the __mro__ attribute of the first argument. It matters because that is how the MRO actually is searched.
By the way, if it was true (it is not),
Yes, I see that now.
then what did you think was the purpose of the second parameter of super(type, obj-or-type)?
Given super(T, obj).method the method object is bound to instance obj; how else could it return a bound method instead of an unbound one? ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: Please make a concrete proposal (a PR or somesuch). That will make it much easier to determine whether a particular bit of word-smithing is a improvement. Of the issues discussed so far, these are the most promising: * Document how the zero argument form of super() infers its arguments. * Be clear that it is the mro of type(self) that determines the search path and that the role of super() is to the next in the mro after the current class. ---------- versions: -Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Change by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- assignee: docs@python -> rhettinger _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Martin Panter <vadmium+py@gmail.com> added the comment: Some of the problems brought up here (which sibling or subclass, and which parameter’s MRO) also came up a few years ago in Issue 23674. ---------- nosy: +martin.panter _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Géry <gery.ogam@gmail.com> added the comment: @Steven D'Aprano
But neither can it *only* look at the MRO of the second class, because that would restart the search at the top of the hierarchy; also if type(second argument) was the only thing that mattered, that would make the first argument redundant and pointless.
Yes of course, but nobody states the opposite. Both parameters are necessary super(type1, obj-or-type2): — the first one for getting the start class in the MRO, which is the class following type1 in the MRO; — the second one for getting the MRO, which is type(obj).__mro__ if it is an instance of type1 or type2.__mro__ if it is a subclass of type1, and for binding the function into a method in a super(type1, obj).function expression (as you correctly said in your last message). Guido's Python equivalent implementation of super() given in his 2002 paper Unifying types and classes in Python 2.2 is very informative: https://www.python.org/download/releases/2.2.3/descrintro/#cooperation All this confusion around super() shows that there is room for improvement in the documentation. @Raymond Hettinger
Please make a concrete proposal (a PR or somesuch).
I will try to make a PR this weekend if I find some time. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Jeroen Demeyer <J.Demeyer@UGent.be> added the comment:
Some of the problems brought up here (which sibling or subclass, and which parameter’s MRO) also came up a few years ago in Issue 23674.
Indeed. I would actually say that these two issues are duplicates of each other. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: Yes, issue 23674 seems to be at least a partial duplicate. We should just get this fixed and close both issues. Ideally, the text can also be made more compact. Having eight paragraphs sends an implicit message that this is too complex to understand and that it should be avoided. One essential goal is that we need to overcome strong preconceptions about about what super() might do versus what it actually does. Those with experience of super() in other languages tend to presume that it would be a keyword rather than a builtin type that needs to be instantiated, that it can only work within a class, that it has lower computational overhead than it does, that it only calls parents rather than siblings, that search order is controlled by the method using super() rather than the child instance, that operators will work, or that the search order is something is something other than the C3 algorithm. The core challenge of documenting super() is that it is so different from what a person might imagine. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Steven D'Aprano <steve+python@pearwood.info> added the comment: On Sat, Jun 08, 2019 at 09:14:18PM +0000, Raymond Hettinger wrote:
Ideally, the text can also be made more compact. Having eight paragraphs sends an implicit message that this is too complex to understand and that it should be avoided.
But it is complex -- as you point out, there are *at least* seven potential misapprehensions here. super does a lot: there are at least four ways to call it (zero-argument form, unbound super, class+instance and class+subclass). And as the old "super considered harmful" versus "super considered super" debate shows, there are pitfalls in multiple inheritance that people run into. The challenge here is that for 95% of cases (plucking a number from thin air) using super is simple: just follow the example in the docs. But the remaining troublesome cases can be very troublesome, and people running into those cases do need to understand the gory details.
One essential goal is that we need to overcome strong preconceptions about about what super() might do versus what it actually does. [...]
Here's two more that you missed: - that you should call super with super(type(self), self); - that unbound super objects dispatch to unbound methods. At some point or another over the last decade I've misunderstood super to do almost all of those things. As this thread shows, I thought I had understood it but I still misunderstood the role of the class calling super. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Carlos André Dantas de Lima <eucandre@gmail.com> added the comment: The method says who you will use some recursion. ---------- nosy: +Carlos André Dantas de Lima _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: I've add PR 15564 to clarify some known ambiguities for the super() docs including a specific example "what it actually does". ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
Change by Raymond Hettinger <raymond.hettinger@gmail.com>: ---------- resolution: -> fixed stage: -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue37176> _______________________________________
participants (6)
-
Carlos André Dantas de Lima -
Géry
-
Jeroen Demeyer
-
Martin Panter
-
Raymond Hettinger
-
Steven D'Aprano