Re: [Python-ideas] [Python-Dev] python and super
On 16/04/2011 00:38, Greg Ewing wrote:
Michael Foord wrote:
consider the "recently" introduced problem caused by object.__init__ not taking arguments. This makes it impossible to use super correctly in various circumstances.
...
It is impossible to inherit from both C and A and have all parent __init__ methods called correctly. Changing the semantics of super as described would fix this problem.
I don't see how, because auto-super-calling would eventually end up trying to call object.__init__ with arguments and fail.
No, not as I described. Where a method does not call up to its parent class then super would *not* auto call the parent class (I'm *not* suggesting a fully auto-call super as the original poster did), but a method not calling super still wouldn't halt the chain of calls. To achieve this a call into a method that doesn't call super (and would normally end the chain) instead removes itself and its unique parents [1] from the mro for this call. It would make the semantics more complex, but it would solve this problem and the original posters problem. For this specific example, if no classes call up to object.__init__ then it won't be called. It would permit you to have methods that can participate in multiple inheritance whilst still blocking calls (overriding) up to their parent classes. All the best, Michael [1] i.e. classes that aren't also the parent of another class still in the mro.
You might think to "fix" this by making a special case of object.__init__ and refraining from calling it. But the same problem arises in a more general way whenever some class in the mix has a method with the right name but the wrong signature, which is likely to happen if you try to mix classes that weren't designed to be mixed together.
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
On Sun, Apr 17, 2011 at 3:22 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
No, not as I described. Where a method does not call up to its parent class then super would *not* auto call the parent class (I'm *not* suggesting a fully auto-call super as the original poster did), but a method not calling super still wouldn't halt the chain of calls. To achieve this a call into a method that doesn't call super (and would normally end the chain) instead removes itself and its unique parents [1] from the mro for this call.
It would make the semantics more complex, but it would solve this problem and the original posters problem. For this specific example, if no classes call up to object.__init__ then it won't be called. It would permit you to have methods that can participate in multiple inheritance whilst still blocking calls (overriding) up to their parent classes.
Could you elaborate on the MRO data structure and super() implementation changes you would propose in order to actually make it possible to implement this idea? The current MRO is calculated as a linear list at class definition time (using the C3 algorithm), throwing away a lot of the original tree information regarding the actual structure of the classes. For example, the MRO "C, B, A, object" could come from a class hierarchy that looked like: class A: ... class B(A): ... class C(B): ... Or one that looked like: class A: ... class B(A): ... class C(B, A): ... Or one that looked like: class A: ... class B: ... class C(B, A): ... If you add a "class D(C, B)" to the bottom of that hierarchy, then any one of those 3 structures could apply across the parent classes, but D would have the same MRO (i.e. "D, C, B, A, object"). And, of course, there are now even *more* scenarios that could produce the same MRO for D. So if the only information you have available is D's MRO (which is the current situation for super()), then you *don't know* whether B is C's parent, sibling or both - the calculation of the MRO discards too much detail about the class hierarchy. Creating a dynamic MRO for every single method call just so people can fail to think correctly about cooperative super() designs is a huge price to pay to gain something that probably won't help in practice. Auto-chaining of super calls just opens up a whole can of worms, and if it can even be done at all, I definitely don't see how it could be done in a backwards compatible way. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 17/04/2011 14:09, Nick Coghlan wrote:
No, not as I described. Where a method does not call up to its parent class then super would *not* auto call the parent class (I'm *not* suggesting a fully auto-call super as the original poster did), but a method not calling super still wouldn't halt the chain of calls. To achieve this a call into a method that doesn't call super (and would normally end the chain) instead removes itself and its unique parents [1] from the mro for this call.
It would make the semantics more complex, but it would solve this problem and the original posters problem. For this specific example, if no classes call up to object.__init__ then it won't be called. It would permit you to have methods that can participate in multiple inheritance whilst still blocking calls (overriding) up to their parent classes. Could you elaborate on the MRO data structure and super() implementation changes you would propose in order to actually make it
On Sun, Apr 17, 2011 at 3:22 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: possible to implement this idea?
The current MRO is calculated as a linear list at class definition time (using the C3 algorithm), throwing away a lot of the original tree information regarding the actual structure of the classes.
Well, it isn't going to happen (no-one else is remotely in favour), so I'm not going to spend a lot more time on the topic. :-)
For example, the MRO "C, B, A, object" could come from a class hierarchy that looked like:
class A: ... class B(A): ... class C(B): ...
Or one that looked like:
class A: ... class B(A): ... class C(B, A): ...
Or one that looked like: class A: ... class B: ... class C(B, A): ...
If you add a "class D(C, B)" to the bottom of that hierarchy, then any one of those 3 structures could apply across the parent classes, but D would have the same MRO (i.e. "D, C, B, A, object"). And, of course, there are now even *more* scenarios that could produce the same MRO for D.
So if the only information you have available is D's MRO (which is the current situation for super()), then you *don't know* whether B is C's parent, sibling or both - the calculation of the MRO discards too much detail about the class hierarchy.
But given that the mro contains the actual classes, *all* the information is available at runtime. (If a call to a method of B terminates the chain then super would be able to know what the base classes of B are by looking at B - even if that isn't expressed directly in the mro.)
Creating a dynamic MRO for every single method call
It is *only* needed where you are mixing super calls with non super calls in the presence of multiple inheritance. So not for every method call.
just so people can fail to think correctly about cooperative super() designs
I disagree with this categorisation. It is particularly for where you *are* thinking about cooperative super calls. The current algorithm has no way to express "don't call my parent class but continue the cooperative calls for other base classes". This means that there are some multiple inheritance scenarios (like __init__ where more than one base class inherits directly from object) that you just can't do and use super. The alternative is not to use super at all, and for diamond inheritance that makes it hard to avoid calling up twice to some methods. These are all *particularly* problems where you are mixing in classes that are from a third party library that you don't control. Even if these classes would be *perfectly amenable* for use as mixins via multiple inheritance, the current super semantics make that impossible. Using composition to get round this can be very messy and require custom code to do your dispatch, (again in diamond inheritance situations trying to avoid methods being called twice) when inheritance *could* just solve the problem (the problem from the original poster being one example of this). Both of these are real problems, and a lot of the task of *explaining* super to Python programmers goes to explaining these caveats and when you can't use super. Yes I'm suggesting adding complexity to the implementation, but it should remove some of the complexity (and especially some of the painful caveats) when actually using it.
is a huge price to pay to gain something that probably won't help in practice.
Auto-chaining of super calls just opens up a whole can of worms, and if it can even be done at all, I definitely don't see how it could be done in a backwards compatible way.
I'm not suggesting full auto-chaining. And especially given that the change only has any effect where you are mixing super calls with non-super calls, which you really can't do at the moment, I don't see any genuine backwards compatibility issues. If by backwards compatibility issues you mean preserving the current structure of __mro__, well it *could* be done without changing that - or by creating an alternative structure for multiple inheritance. Doesn't look like it will happen though. :-) I'll probably blog about the idea for posterity, and then move on. All the best, Michael Foord
Cheers, Nick.
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
Michael Foord wrote:
On 17/04/2011 14:09, Nick Coghlan wrote:
just so people can fail to think correctly about cooperative super() designs
I disagree with this categorisation. It is particularly for where you *are* thinking about cooperative super calls.
I think what Nick means is that, although *you* might be thinking about super calls, the people who wrote the classes you're using did *not* (otherwise they would have included the required super calls). So you're trying to mix classes that weren't designed to be mixed together, which is likely to lead to many more problems than just missing super calls.
The current algorithm has no way to express "don't call my parent class but continue the cooperative calls for other base classes".
That's not quite what you're proposing. What you're proposing is more like "call my parent class, but don't stop if my parent class doesn't call *its* parent class." Which still leaves the question of what happens if your parent's parent turns up later in your own MRO. Your parent says not to call it, but your MRO says to call it. Who wins, and why? There's also the question of what to do with return values. Without an answer to that, this feature would be restricted to methods that don't return any useful value. -- Greg
On Mon, Apr 18, 2011 at 9:35 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Michael Foord wrote:
On 17/04/2011 14:09, Nick Coghlan wrote:
just so people can fail to think correctly about cooperative super() designs
I disagree with this categorisation. It is particularly for where you *are* thinking about cooperative super calls.
I think what Nick means is that, although *you* might be thinking about super calls, the people who wrote the classes you're using did *not* (otherwise they would have included the required super calls). So you're trying to mix classes that weren't designed to be mixed together, which is likely to lead to many more problems than just missing super calls.
Yep, that's what I meant. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Apr 18, 2011 at 1:25 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
On 17/04/2011 14:09, Nick Coghlan wrote:
So if the only information you have available is D's MRO (which is the current situation for super()), then you *don't know* whether B is C's parent, sibling or both - the calculation of the MRO discards too much detail about the class hierarchy.
But given that the mro contains the actual classes, *all* the information is available at runtime. (If a call to a method of B terminates the chain then super would be able to know what the base classes of B are by looking at B - even if that isn't expressed directly in the mro.)
But super() is out of the picture by the time B returns - it's just an object, it has no control over what happens when its methods return. There's no overarching object with a view of the whole method chain, just a series of object constructions and method calls: obj.__mro__ == D, C, B, A, object obj.method() --> D.method(obj) ----> super(D, obj).method() # (aka C.method(obj)) ------> super(C, obj).method() # (aka B.method(obj)) --------> super(B, obj).method() # (aka A.method(obj)) ----------> super(A, obj).method() # (aka object.method(obj)) How does the caller of super(D, obj).method() know whether or not super(C, obj).method() was called any more than the caller of C.method(obj) knows whether or not B.method(obj) was called?
Creating a dynamic MRO for every single method call
It is *only* needed where you are mixing super calls with non super calls in the presence of multiple inheritance. So not for every method call.
Then why build it into super() at all? Prove the concept by going back to old-school super style and pass in the class reference explicitly and create a dynamic MRO based on removal of your parent classes but not your siblings. I just don't think it is possible to solve this as simply as you appear to believe. If you can produce a working prototype that actually has acceptable performance, is backwards compatible with current cooperative super() code then I'll be happily proven wrong, but the entire concept currently seems to be based on an idea of iterating over the MRO in a way that *doesn't actually happen*. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Greg Ewing -
Michael Foord -
Nick Coghlan