Problems with the new super()
Hi all, I blogged about that topic today which turned out to be a very bad idea, so I summarize it for the mailinglist here to hopefully start a discussion about the topic, which I think is rather important. In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to `super`. That's interesting because it means that the language shifts into a completely different direction. `super` was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don't know their parent but confusing for many. I can see that a replacement is a good idea, but I'm not so sure if the current implementation is the way to go. The main problem with replacing `super(Foo, self).bar()` with something like `super.bar()` is obviously that self is explicit and the class (in that case Foo) can't be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions `vars`, `locals`, `gloabal`, `vars` and some functions in the inspect module. And all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that. The implementation of the automatic super currently has two ugly details that I think violate the Python Zen: The bytecode generated is differently if the name "super" is used in the function. `__class__` is only added as cell to the code if `super` or `__class__` is referenced. That and the fact that `f_localsplus` is completely unavailable from within python makes the whole process appear magical. This is way more magical than anything we've had in Python in the past and just doesn't fit into the language in my opinion. We do have an explicit self in methods and methods are more or less just functions. Python's methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That's an incredible cool thing to have and makes things very simple and non-magical. Breaking that principle by introducing an automatic super seems to harm the concept. Another odd thing is that Python 3 starts keeping information on the C layer we can't access from within Python. Super is one example, another good one are methods. They don't have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective: | >>> class C: | ... normal = lambda x: None | ... static = staticmethod(lambda x: None) | ... | >>> type(C.normal) is type(C.static) | True | >>> C.normal | <function <lambda> at 0x4da150> As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance: | >>> type(C().normal) is type(C().static) | False | >>> C().normal | <bound method C.<lambda> of <__main__.C object at 0x4dbcf0>> | >>> C().static | <function <lambda> at 0x4da198> While I don't knwo about the method thing, I think an automatic super should at least be implementable from within Python. I could imagine that by adding __class__ and __self__ to scopes automatically a lot of that magic could be removed. Regards, Armin
The staticmethod thing isn't new; that's also the case in 2.x. The super() thing is a case of practicality beats purity. Note that you pay a small but measurable cost for the implicit __class__ (it's implemented as a "cell variable", the same mechanism used for nested scopes) so we wouldn't want to introduce it unless it is used. On Wed, Apr 30, 2008 at 2:46 PM, Armin Ronacher <armin.ronacher@active-4.com> wrote:
Hi all,
I blogged about that topic today which turned out to be a very bad idea, so I summarize it for the mailinglist here to hopefully start a discussion about the topic, which I think is rather important.
In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to `super`. That's interesting because it means that the language shifts into a completely different direction.
`super` was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don't know their parent but confusing for many. I can see that a replacement is a good idea, but I'm not so sure if the current implementation is the way to go.
The main problem with replacing `super(Foo, self).bar()` with something like `super.bar()` is obviously that self is explicit and the class (in that case Foo) can't be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions `vars`, `locals`, `gloabal`, `vars` and some functions in the inspect module. And all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that.
The implementation of the automatic super currently has two ugly details that I think violate the Python Zen: The bytecode generated is differently if the name "super" is used in the function. `__class__` is only added as cell to the code if `super` or `__class__` is referenced. That and the fact that `f_localsplus` is completely unavailable from within python makes the whole process appear magical.
This is way more magical than anything we've had in Python in the past and just doesn't fit into the language in my opinion. We do have an explicit self in methods and methods are more or less just functions. Python's methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That's an incredible cool thing to have and makes things very simple and non-magical. Breaking that principle by introducing an automatic super seems to harm the concept.
Another odd thing is that Python 3 starts keeping information on the C layer we can't access from within Python. Super is one example, another good one are methods. They don't have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective:
| >>> class C: | ... normal = lambda x: None | ... static = staticmethod(lambda x: None) | ... | >>> type(C.normal) is type(C.static) | True | >>> C.normal | <function <lambda> at 0x4da150>
As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance:
| >>> type(C().normal) is type(C().static) | False | >>> C().normal | <bound method C.<lambda> of <__main__.C object at 0x4dbcf0>> | >>> C().static | <function <lambda> at 0x4da198>
While I don't knwo about the method thing, I think an automatic super should at least be implementable from within Python. I could imagine that by adding __class__ and __self__ to scopes automatically a lot of that magic could be removed.
Regards, Armin
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Hi, Guido van Rossum <guido <at> python.org> writes:
The staticmethod thing isn't new; that's also the case in 2.x. staticmethod hasn't changed, method has. In the past Class.method gave you a unbound method, now you get a function back as if it was a static method.
The super() thing is a case of practicality beats purity. Note that you pay a small but measurable cost for the implicit __class__ (it's implemented as a "cell variable", the same mechanism used for nested scopes) so we wouldn't want to introduce it unless it is used. I do agree that super() is a lot easier to work with than regular way to call it. But the fact that it breaks if i do `_super = super` or that it's impossible to emulate it from within Python.
Regards, Armin
Armin Ronacher schrieb:
The super() thing is a case of practicality beats purity. Note that you pay a small but measurable cost for the implicit __class__ (it's implemented as a "cell variable", the same mechanism used for nested scopes) so we wouldn't want to introduce it unless it is used. I do agree that super() is a lot easier to work with than regular way to call it. But the fact that it breaks if i do `_super = super` or that it's impossible to emulate it from within Python.
That it isn't emulatable from Python doesn't bother me -- several functions have that property. But the other two magical things about super() really bother me too. I haven't looked at the new super in detail so far (and I don't know how many others have), and two things are really strikingly unpythonic in my view: * super() only works when named "super" [1]. It shouldn't be a function if it has that property; no other Python function has that. * "__class__" is magical in classes. If you define a local called "__class__" super won't work anymore in that function. Also, you can access "__class__" from any method, without the "self." qualifier -- another magical name. There may be more implications and surprising behavior surrounding this. I know that the implementation is a compromise, but I'd rather see a super() whose full semantics can be explained to programmers without using to "cell variable", "f_localsplus" and "symtable". cheers, Georg [1] Actually, it only works if a name "super" is accessed somewhere in the function, but this is what someone trying to alias "super" will perceive. -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
2008/5/1, Georg Brandl <g.brandl@gmx.net>:
There may be more implications and surprising behavior surrounding this.
I know that the implementation is a compromise, but I'd rather see a super() whose full semantics can be explained to programmers without using to "cell variable", "f_localsplus" and "symtable".
In consideration of what's been said about super() in the past, and what is handled here regarding its Py3 implementation, I want to make a step back, and ask: Has super() proved more useful than harmful? Which is the value for Py3 to keep it? Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/
2008/5/1 Facundo Batista <facundobatista@gmail.com>:
2008/5/1, Georg Brandl <g.brandl@gmx.net>:
There may be more implications and surprising behavior surrounding this.
I know that the implementation is a compromise, but I'd rather see a super() whose full semantics can be explained to programmers without using to "cell variable", "f_localsplus" and "symtable".
In consideration of what's been said about super() in the past, and what is handled here regarding its Py3 implementation, I want to make a step back, and ask:
Has super() proved more useful than harmful? Which is the value for Py3 to keep it?
Since Python supports multiple inheritance, you can't get away from something like super. Alternatives for methods chaining to parent classes are 1. implicit/automatic (like C++, I think), 2. explicit/semi-automatic (python's super, on one form or another, 3. explicit manual (programmer has to manually keep track of everything). Of all these alternatives I prefer Python's super (in one form or another). 1 is too "magic", and 3 is too error prone and cumbersome. A better question would be, is multiple inheritance good or bad for programs? :-) I think many people's answer to the above would be, generally bad, but there are exceptions where it helps and can be justified.
Regards,
-- . Facundo
Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
-- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert
Gustavo Carneiro wrote:
A better question would be, is multiple inheritance good or bad for programs? :-)
I would say there are good ways of using it, and bad ways of using it. In my experience, the good ways occur when the classes being mixed together are completely independent -- there is no overlap in method or instance variable names, and each class brings its own independent bundle of functionality to the party. If the classes being mixed clash or overlap in functionality somehow, the inheriting class needs to override all of the clashing methods and properties and resolve matters by delegating to one or another of the inherited classes (using explicit inherited method calls, not super!). If it's not feasible to do that for some reason, then you're better off forgetting about multiple inheritance and finding some other solution to the problem. -- Greg
On May 1, 2008, at 9:21 PM, Greg Ewing wrote:
If the classes being mixed clash or overlap in functionality somehow, the inheriting class needs to override all of the clashing methods and properties and resolve matters by delegating to one or another of the inherited classes (using explicit inherited method calls, not super!).
Sorry but thats just not fair. It's not about convenience for the classes you're inheriting from, its so the inheriting class and superclasses can cooperate without requiring implementation details of one another. I agree that if your methods are 'clashing' then you are probably misinheriting, but cooperative methods can be the most natural way to model certain situations. Not to be dull but when building up a method from smaller reusable bits of functionality this is the case. Once you have explicitly stated the classes you want to inherit from, it isn't magic to expect some of their methods to chain without explicitly restating how exactly they should do that. In fact, doing so may make you more likely to make an error when you change which bits you want to include. If the method resolution order is there (and explicit somewhere) you might as well make use of it.
If it's not feasible to do that for some reason, then you're better off forgetting about multiple inheritance and finding some other solution to the problem.
Is it an issue of feasibility, or of what is the 'most obvious' solution? jared
Jared Flatow wrote:
I agree that if your methods are 'clashing' then you are probably misinheriting, but cooperative methods can be the most natural way to model certain situations.
I'm not saying that nobody should ever use super, only that it's not the right thing for the situation I was talking about there. What it essentially comes down to is that classes mix well if they were designed to mix well. Keeping their features independent is one way to achieve that. Designing their methods to fit together in a super call chain is another, if you can manage to pull it off. All I was really trying to say is that stating that "multiple inheritance is bad" is far too simplistic.
Is it an issue of feasibility, or of what is the 'most obvious' solution?
I can only speak from my own experience, which is that whenever I've had a problem involving multiple inheritance, super() didn't solve it. What did solve it was either refactoring so that the classes being mixed were more independent, or finding another solution that didn't require multiple inheritance. Usually the new solution turned out to be better in other ways as well, so I've come to regard multiple inheritance issues as a code smell suggesting that I need to rethink something. -- Greg
On May 2, 7:54 am, Greg Ewing <greg.ew...@canterbury.ac.nz> wrote:
I can only speak from my own experience, which is that whenever I've had a problem involving multiple inheritance, super() didn't solve it. What did solve it was either refactoring so that the classes being mixed were more independent, or finding another solution that didn't require multiple inheritance.
Usually the new solution turned out to be better in other ways as well, so I've come to regard multiple inheritance issues as a code smell suggesting that I need to rethink something.
This is my experience as well. I have not found a real life problem yet that I could not solve with single inheritance + composition/delegation in a better and more maintainable way than using multiple inheritance. Also, I have come to believe that cooperative methods are a wart (it is too difficult to reason about them, they are fragile, and overall I see them as an unneeded complication in everyday coding). But we are stuck with multiple inheritance now and there is already a lot of code out there using it, so in the present situation we have to cope with it and to make it more usable, so I welcome the new super. I think that it should be made a keyword tough (it is too magic now not to be one). Michele Simionato
Facundo Batista wrote:
Has super() proved more useful than harmful? Which is the value for Py3 to keep it?
Personally I've found exactly zero use cases for super() so far in my own code. A couple of times I thought I'd found one, but it turned out not to do quite what I wanted, and I ended up finding better solutions. So if it were up to me, I wouldn't be putting any effort into making it easier to use. I'm actually worried that making it too easy to use could lead people into using it when it's not appropriate. -- Greg
On Thu, May 1, 2008 at 11:20 AM, Georg Brandl <g.brandl@gmx.net> wrote:
But the other two magical things about super() really bother me too. I haven't looked at the new super in detail so far (and I don't know how many others have), and two things are really strikingly unpythonic in my view:
* super() only works when named "super" [1]. It shouldn't be a function if it has that property; no other Python function has that.
Actually, I believe IronPython and/or Jython have to use this trick in certain cases -- at least I recall Jim Hugunin talking about generating different code when the use of locals() was detected. I'm not proud of this, but I don't see a way around it. The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not). There were long discussions about various possible ways to implement something like this, and they all had their downsides. (The PEP still isn't fixed to describe the status quo.)
* "__class__" is magical in classes. If you define a local called "__class__" super won't work anymore in that function.
Also, you can access "__class__" from any method, without the "self." qualifier -- another magical name.
I don't mind this at all -- it's a name starting and ending with double underscores, so you shouldn't use it except for its defined semantics, which happen to be exactly what super needs. (I would have proposed __super__() to get an exception for that too, except that it's excessively ugly.)
There may be more implications and surprising behavior surrounding this.
I know that the implementation is a compromise, but I'd rather see a super() whose full semantics can be explained to programmers without using to "cell variable", "f_localsplus" and "symtable".
You don't have to explain it that way at all. First you explain how the 2.x super(C, self) works. Then you explain that if you call it with no arguments, the arguments default to the current class object and the first argument of the current function. Only people wanting to write their own interpreter need to know more.
cheers, Georg
[1] Actually, it only works if a name "super" is accessed somewhere in the function, but this is what someone trying to alias "super" will perceive.
To Facundo, who asks if we need super() at all: yes, we need it. You can't write decent multiple-inheritance code without it. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, May 01, 2008 at 12:55:22PM -0700, Guido van Rossum wrote:
I'm not proud of this, but I don't see a way around it. The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not). There were long discussions about various possible ways to implement something like this, and they all had their downsides. (The PEP still isn't fixed to describe the status quo.)
I remember some brainstorms about treating more like self. I'm not sure if these were thought through all the way, but I remember seeing something like: class MyClass(Super1, Super2): # This method requires super: @requires_super def __init__(self, super, **kwds): super(**kwds) # This method doesn't require super: def some_method(self): pass I'm sure there are drawbacks, but it fits in my head. Using super in Python 2.0 is verbose but simple. However, I'm a little scared of super in Python 3.0. I guess I'm probably just a wimp. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On May 1, 2008, at 6:33 PM, Andrew McNabb wrote:
On Thu, May 01, 2008 at 12:55:22PM -0700, Guido van Rossum wrote:
I'm not proud of this, but I don't see a way around it. The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not). There were long discussions about various possible ways to implement something like this, and they all had their downsides. (The PEP still isn't fixed to describe the status quo.)
I remember some brainstorms about treating more like self. I'm not sure if these were thought through all the way, but I remember seeing something like:
class MyClass(Super1, Super2): # This method requires super: @requires_super def __init__(self, super, **kwds): super(**kwds)
# This method doesn't require super: def some_method(self): pass
I'm sure there are drawbacks, but it fits in my head. Using super in Python 2.0 is verbose but simple. However, I'm a little scared of super in Python 3.0. I guess I'm probably just a wimp.
It certainly makes me uncomfortable too. I think of all the alternatives in PEP 3135, I'd probably prefer self.__super__.foo(), except that I'd call it self.super.foo(). Although I don't mind reserving a non-underscore-adorned name for Python 3.0, I could see adopting self.__super__ and using super(self).foo() as a shortcut. To me, that addresses the main rationale of the PEP without the magic (i.e. no need to repeat the class). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSBpWLHEjvBPtnXfVAQJmOAP+NW1tj67Ls+m6PCbF9wYpPRQhT2RJ1210 0QdYxyYz8akY5+I1QJTp3BN5erDLw1sAWGcKVP2phw7Rvb3pXf8FGh/Yg8du7KAg ZAm96xdaNLPiATVDaZZHuoWZ3+S6zUbmx6QtpjU//EAOXhwQCoTdhDme9QyPDI/2 kA+oldSXr+M= =bBRP -----END PGP SIGNATURE-----
Barry Warsaw wrote:
I think of all the alternatives in PEP 3135, I'd probably prefer self.__super__.foo(), except that I'd call it self.super.foo().
I'm not sure that's sufficient. You need to be able to specify a class when using MI. I'd prefer self.super().foo(). self.super() would take one argument, the base class, optional iff self's class has only one base class. Not that it makes a lick of difference, /larry/
Guido van Rossum wrote:
The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not).
If it's really such a useful thing as to warrant so much magic to support it, then I think it deserves to have a keyword. Conversely, I would say that if it doesn't deserve a keyword, it also doesn't deserve that much magic. -- Greg
On Fri, 2 May 2008, Greg Ewing wrote:
Guido van Rossum wrote:
The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not).
If it's really such a useful thing as to warrant so much magic to support it, then I think it deserves to have a keyword.
Conversely, I would say that if it doesn't deserve a keyword, it also doesn't deserve that much magic.
One might even go further and say that if it's that magic, then it's a keyword by definition. Certainly I would want a syntax-highlighting editor to highlight it somehow, and if the editor has a "rename" feature to rename a variable and replace all references to it (but not other uses of the same identifiers in different scope), then it probably should at least alert the programmer before allowing a rename to or from the magic name. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist
Guido van Rossum schrieb:
On Thu, May 1, 2008 at 11:20 AM, Georg Brandl <g.brandl@gmx.net> wrote:
But the other two magical things about super() really bother me too. I haven't looked at the new super in detail so far (and I don't know how many others have), and two things are really strikingly unpythonic in my view:
* super() only works when named "super" [1]. It shouldn't be a function if it has that property; no other Python function has that.
Actually, I believe IronPython and/or Jython have to use this trick in certain cases -- at least I recall Jim Hugunin talking about generating different code when the use of locals() was detected.
I don't know if it's possible in Jython to have "locals" referring to something else. For CPython, the name "super" in a function can refer to anything -- local, global or builtin -- and it just feels wrong for the compiler to make assumptions based on the mere mention of a non-reserved name.
I'm not proud of this, but I don't see a way around it. The alternative would be to make it a keyword, which seemed excessive (plus, it would be odd if super() were a keyword when self is not).
I don't find it odd. In fact, IMO the whole magic needed for the runtime implementation of "super()" justifies super becoming a keyword. Georg [Moving this to the Python-3000 list] -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.
participants (12)
-
Andrew McNabb -
Armin Ronacher -
Barry Warsaw -
Facundo Batista -
Georg Brandl -
Greg Ewing -
Guido van Rossum -
Gustavo Carneiro -
Isaac Morland -
Jared Flatow -
Larry Hastings -
michele.simionato＠gmail.com