MRO local precedence ordering revisited
Dear all, I just pushed a very specific question which as I now recognized is far more general: It concerns python MRO (method resolution order). As reference I take https://www.python.org/download/releases/2.3/mro/ There the section "Bad Method Resolution Orders" starts with an example which behaviour in the current python 2.7 is rather unintuitive to me. I rename the example slightly to illustrate its usecase. class Mixin(object):
pass
class A(Mixin):
this unfortunately throws "TypeError: Error when calling the metaclass bases Cannot create a consistent method resolution order (MRO) for bases A, Mixin" The reference name above comments this case similar to the following (adapted to the easier example above):
...
While it might be the case that in Python 2.2 things where different, I cannot agree that the expected order of Method resolution is ambiguous (at least as far I see at this stage). The reference itself says that we would expect B to be inherited first by Mixin and then by A. There is no ambiguity any longer. *Therefore I would like to propose to make this MRO again a valid one.* The usecase should be obvious: If you want a Mixin to be the first thing overwriting functions, but still want to inherit as normal. (I myself want for instance a Mixin to overwrite the __init__ method, which is simply not impossible when choosing class B(A, Mixin)) best, Stephan
On Fri, Dec 11, 2015 at 3:36 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Just to make something quite clear: "current" Python, as discussed on this list, is 3.5 or 3.6, not 2.7. There won't be a 2.8, and future releases of 2.7.x won't be changing anything like this. What you're describing is also the case in current Python 3.x, but any change to how the MRO is derived would happen in 3.6 at the very soonest. (At least, I don't _think_ the current behaviour could be considered a bug. I could be wrong.)
What you're proposing, in effect, is that it should be possible for super() to go *down* the class hierarchy. Under your proposal, B.__mro__ would be (Mixin,A,object), which means that super().method() inside A will go directly to object, and that the same call inside Mixin will go to A - reversing the order of the calls compared to the way they would be if the object were of type A. Now it is known that the super doesn't mean "call my base class", but something more like "call the next class"; but it's currently guaranteed that the MRO for any derived class is a *merge* of the MROs of its parents, without ever reordering them. Under your rules, what would be the MRO here? class A(object): # explicitly stating the default def __new__(cls): print("Constructing an A") return super().__new__(cls) def method(self): print("Calling method on A") super().method() class B(object, A): pass If you answered (B,object,A), then you've just made it possible to have some other class than object at the end of the chain. Where do super() calls go if there *is no next class*? What should happen? Can that be made equally intuitive to your proposal about mixins? ChrisA
I wholeheartedly agree with Chris on this one. This isn't broken, nor is it something we want to change. If you're looking for a case where object is not the last of its hierarchy, Python 2 has classic classes, but they are remaining artefacts there. The MRO needs to be consistent across all classes along the MRO, with object last.
On Fri, Dec 11, 2015 at 3:36 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
It's unambiguous to you because you *already know* your "expected" semantics. The spam-eggs example in Michele Simionato's paper that was cited earlier (https://www.python.org/download/releases/2.3/mro/) makes it clear why this is ambiguous from the point of view of the interpreter. The interpreter can only see the syntax, and there are two semantic principles it can follow to determine what various users might expect: use the most specific base class first (which enforces monotonicity IIUC), or use the order specified in the class first (local precedence). In this case those two principles conflict, and therefore MRO is ambiguous. In cases where you know the "expected" semantics, probably you can use a metaclass to enforce them (don't ask me how, I've never programmed a metaclass). BTW, multiple inheritance != mixin, and I certainly wouldn't call your Liftable class (from the original thread) a mixin, as it has complex and unobvious semantics of instance initialization in the presence of multiple inheritance. YMMV, as there doesn't seem to be an official definition of mixin for Python. Chris Angelico writes:
(At least, I don't _think_ the current behaviour could be considered a bug. I could be wrong.)
Given that this was discussed extensively for 2.3, and Guido found himself compelled to change his mind to support the C3 algorithm by Samuele Pedroni's examples at that time, and that the C3 algorithm has been widely adopted by other languages (intuition portability!), I would say, no, this is *not* a bug. There may be an algorithm which satisfies the C3 conditions and generates a order in more cases, in which case changing to that algorithm would be a reasonable feature request. But C3 still excludes the OP's desired MRO algorithm.
On Dec 10, 2015, at 08:36, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
First off, Python 2.7 isn't going to change, and there isn't going to be a 2.8. If you want to propose changes to Python, you first need to learn what's changed up to 3.5, and then propose your change for 3.6. Second, the fact that it's unintuitive may be a problem with the description, rather than the algorithm. Once you understand what it's trying to do, and what you're trying to do, and why they're not compatible, it's intuitive that it doesn't work. Maybe the docs need to make it easier to understand what it's trying to do, and maybe you can help point out what's confusing you. Since Python just borrowed the C3 algorithm from Dylan to solve the same problems, and other languages have borrowed it as well, you might want to search for more generic documentation on it, too. Anyway, if you understand what C3 is solving, and you think it could be modified to still solve those problems while also allowing your use, or that one of those problems can't be solved but you don't think it's actually a problem, then explain that. Otherwise, I think you're either asking for something inconsistent, or asking for something consistent but broken, like the C++ rules (which would be even more broken in Python without first adding notions like virtual base class and auto-pre-supering constructors).
Mixin has to appear before A so that its methods appear before A's. But Mixin has to appear between A and object so that A's methods can count on inheriting from it. How do you fit both those constraints?
That's not what mixins are for. Mixins add behavior; something that changes the behavior of a real superclass is something different.
Dear all, this is a highly sensitive topic, isn't it? =) Thank you all for your extensive responses. Of course I was a bit provocative with the sentence in bold, but I haven't said this would be a bug, and if I have, I am sorry for that. I am in fact not familiar with what precisely are all the problems the C3 algorithm tries to solve, however my own intuitiv MRO would not be anything proposed so far, but: (B, mixin, A, mixin, object) or in the simpler, more trivial version (B, object, A, object) As you haven't mentioned this as a possibility at all, I guess having a class twice in this list produces some weird behaviour I do not know about yet - if someone can point out, that would be great. cheers, Stephan P.S.: I haven't said that I want a python 2.8, nevertheless thanks for pointing out that there won't be a python 2.8 (this I in fact have not known before)! I am familar with python 3.5 as well, however I just wanted to make clear where I am working and in which language the example code is runnable. On 10 December 2015 at 19:58, Andrew Barnert <abarnert@yahoo.com> wrote:
On Fri, Dec 11, 2015 at 7:03 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
In terms of people's expectations of super(), I think it would be intolerably surprising for it to be able to call a method from *the same* class. If your mixin has "def method(self): super().method()", it'll call itself (once) if called from B.
https://www.python.org/dev/peps/pep-0404/ http://blog.startifact.com/guido_no.jpg There are some special exemptions in 2.7 (for instance, some changes went through recently that improve security, even at the expense of some backward compatibility), but the bar is extremely high; behaviour has to either be recognizably buggy, or have significant security implications (cf hash randomization and certificate checking), to be changed in 2.7. I don't think the MRO fits either category, so any change would be 3.x only. As mentioned, though, your code works on 3.5/3.6 without changes and with the same semantics. ChrisA
@Chris thanks for pointing out this self-reference, I am still not sure whether this is really suprising to me, but at least I am still not done with thinking about it, so probably it is @Ned As I understood it, the MRO is not only for searching attributes - there it is indeed impressively redundant to put the same class twice into the MRO, thanks for pointing that out - but also for the hierarchy of the super() command @all thank you all for your comments and help. My current conclusion is that I will read about the C3 algorithm in crucially more detail and what it in fact is trying to solve ... and eventually may come back best, Stephan On 10 December 2015 at 21:43, Ned Batchelder <ned@nedbatchelder.com> wrote:
On Thu, Dec 10, 2015 at 1:51 PM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
In that situation including the class twice results in an infinite loop. If your MRO is (B, mixin, A, mixin, object), then the chain of super calls looks like this: super(B, self).method() super(mixin, self).method() super(A, self).method() super(mixin, self).method() super(A, self).method() ... This happens because the second super call from the mixin class is identical to the first one. super doesn't know which appearance of mixin based on the arguments passed it, and if it naively assumes the first appearance, then nothing after the second appearance of mixin will ever be reached.
On Dec 10, 2015, at 12:03, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Dear all,
this is a highly sensitive topic, isn't it?
Well, it's an _interesting_ topic. If there really is something new to be done here that could improve the way inheritance works, that's a big deal, so people want to think about it, which means asking you challenging questions.
First, think about what kind of algorithm could get mixin to appear twice in the first list without also getting object to appear twice. I don't think there's any obvious way to do it. (Well, you could special-case object, but that doesn't help the more general diamond problem.) If you have a non-obvious answer to that, even though that still might not be a complete solution to what you really wanted, it would be a major contribution on its own.
As you haven't mentioned this as a possibility at all, I guess having a class twice in this list produces some weird behaviour I do not know about yet - if someone can point out, that would be great.
For an obvious problem: how could any cooperative inheritance tree ever super anything if object, which obviously doesn't cooperate, might end up on the MRO between one class and the next? Plus, if a class can get called twice on the super chain, how does it know which of the two times it's being called? Imagine a trivial mixin that just counts how many instances of its descendant get created. How do you avoid counting twice when a C gets constructed. (If you're thinking of using __mangled names to help, notice that _mixin__mangled and _mixin__mangled are the same name.) More generally: the code in a class generally assumes that it's not going to somehow super itself. Breaking that assumption, especially for code that adds attributes, makes the code much harder to write, and reason about. You probably can come up with a manual solution to these problems that doesn't work fully generally, but does work for your use case. (By adding specific rules on how classes in your tree interact, you can simplify the problem as far as you want.) Which you'd implement in a metaclass. But once you're writing a metaclass that interferes with the MRO mechanism, why do you care what the default MRO mechanism is? (That last might not be a rhetorical question--maybe the answer is "I wouldn't care, but it's too hard to override the setup of __mro__ in a metaclass __new__ method" or something, in which case there may be something to improve here.)
Stephan Sahm wrote:
I'm not sure what that would do to super calls. It's possible you would end up in a loop from Mixin -> A -> Mixin -> A -> ... But in any case, this MRO doesn't solve the fundamental problem that the MROs of B and A are inherently contradictory. *You* know what MRO you want in this particular case, but there's no way for Python to know that. -- Greg
On Dec 10, 2015, at 15:19, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
After thinking about this a bit: the MRO isn't really contradictory; what's contradictory is what he's expecting super() to do with his MRO. When called from within mixin.spam(), he wants it to give him A, but when calling from within mixin.spam(), he wants it to give him object. Since those two "whens" are identical, there's his problem. But if you never call super(), there's nothing wrong with it; any attribute provided by Mixin the first time means you'll never get to Mixin the second time. (I suppose you could hack that with quantum attributes that have an x% chance of being there each time you call __getattr__, but just don't do that...) Of course he wants to call super. Or, rather, he wants to call something that's kind of like super, but that handles his MRO in a non-contradictory way. A similar MI next-method protocol that went by index rather than by name would work: within mro[1].spam() it gives you mro[2]; within mro[3].spam() it gives you mro[4]. Python doesn't come with a function that does that, but you can write one, with a bit of kludgery. And you can easily write a metaclass that sets up the MRO you want. Which means you can build exactly what the OP is asking for. It still won't do what he wants. For one thing, like super, it depends on the entire class hierarchy cooperating, which includes the root not supering but everyone else doing so--but mixin is the root, and also elsewhere on the tree, so it has to both super and not super. And there are a couple other problems. But tightening up the requirements a bit, I think you can make something coherent out of this. It's definitely not the best way to solve his problem, and it's probably not a good way to solve _any_ problem--but it does work. See http://stupidpythonideas.blogspot.com/2015/12/can-you-customize-method-resol... for a fully worked through example.
Stephan Sahm wrote:
The reason Python disallows this is that B is saying that methods of Mixin should override those of A, whereas A is saying that its own methods should override those of Mixin. So there is a potential for B to break A's functionality. It may be that nothing gets broken in your particular case, but Python can't know that in general, so it errs on the safe side. -- Greg
On Fri, Dec 11, 2015 at 3:36 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Just to make something quite clear: "current" Python, as discussed on this list, is 3.5 or 3.6, not 2.7. There won't be a 2.8, and future releases of 2.7.x won't be changing anything like this. What you're describing is also the case in current Python 3.x, but any change to how the MRO is derived would happen in 3.6 at the very soonest. (At least, I don't _think_ the current behaviour could be considered a bug. I could be wrong.)
What you're proposing, in effect, is that it should be possible for super() to go *down* the class hierarchy. Under your proposal, B.__mro__ would be (Mixin,A,object), which means that super().method() inside A will go directly to object, and that the same call inside Mixin will go to A - reversing the order of the calls compared to the way they would be if the object were of type A. Now it is known that the super doesn't mean "call my base class", but something more like "call the next class"; but it's currently guaranteed that the MRO for any derived class is a *merge* of the MROs of its parents, without ever reordering them. Under your rules, what would be the MRO here? class A(object): # explicitly stating the default def __new__(cls): print("Constructing an A") return super().__new__(cls) def method(self): print("Calling method on A") super().method() class B(object, A): pass If you answered (B,object,A), then you've just made it possible to have some other class than object at the end of the chain. Where do super() calls go if there *is no next class*? What should happen? Can that be made equally intuitive to your proposal about mixins? ChrisA
I wholeheartedly agree with Chris on this one. This isn't broken, nor is it something we want to change. If you're looking for a case where object is not the last of its hierarchy, Python 2 has classic classes, but they are remaining artefacts there. The MRO needs to be consistent across all classes along the MRO, with object last.
On Fri, Dec 11, 2015 at 3:36 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
It's unambiguous to you because you *already know* your "expected" semantics. The spam-eggs example in Michele Simionato's paper that was cited earlier (https://www.python.org/download/releases/2.3/mro/) makes it clear why this is ambiguous from the point of view of the interpreter. The interpreter can only see the syntax, and there are two semantic principles it can follow to determine what various users might expect: use the most specific base class first (which enforces monotonicity IIUC), or use the order specified in the class first (local precedence). In this case those two principles conflict, and therefore MRO is ambiguous. In cases where you know the "expected" semantics, probably you can use a metaclass to enforce them (don't ask me how, I've never programmed a metaclass). BTW, multiple inheritance != mixin, and I certainly wouldn't call your Liftable class (from the original thread) a mixin, as it has complex and unobvious semantics of instance initialization in the presence of multiple inheritance. YMMV, as there doesn't seem to be an official definition of mixin for Python. Chris Angelico writes:
(At least, I don't _think_ the current behaviour could be considered a bug. I could be wrong.)
Given that this was discussed extensively for 2.3, and Guido found himself compelled to change his mind to support the C3 algorithm by Samuele Pedroni's examples at that time, and that the C3 algorithm has been widely adopted by other languages (intuition portability!), I would say, no, this is *not* a bug. There may be an algorithm which satisfies the C3 conditions and generates a order in more cases, in which case changing to that algorithm would be a reasonable feature request. But C3 still excludes the OP's desired MRO algorithm.
On Dec 10, 2015, at 08:36, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
First off, Python 2.7 isn't going to change, and there isn't going to be a 2.8. If you want to propose changes to Python, you first need to learn what's changed up to 3.5, and then propose your change for 3.6. Second, the fact that it's unintuitive may be a problem with the description, rather than the algorithm. Once you understand what it's trying to do, and what you're trying to do, and why they're not compatible, it's intuitive that it doesn't work. Maybe the docs need to make it easier to understand what it's trying to do, and maybe you can help point out what's confusing you. Since Python just borrowed the C3 algorithm from Dylan to solve the same problems, and other languages have borrowed it as well, you might want to search for more generic documentation on it, too. Anyway, if you understand what C3 is solving, and you think it could be modified to still solve those problems while also allowing your use, or that one of those problems can't be solved but you don't think it's actually a problem, then explain that. Otherwise, I think you're either asking for something inconsistent, or asking for something consistent but broken, like the C++ rules (which would be even more broken in Python without first adding notions like virtual base class and auto-pre-supering constructors).
Mixin has to appear before A so that its methods appear before A's. But Mixin has to appear between A and object so that A's methods can count on inheriting from it. How do you fit both those constraints?
That's not what mixins are for. Mixins add behavior; something that changes the behavior of a real superclass is something different.
Dear all, this is a highly sensitive topic, isn't it? =) Thank you all for your extensive responses. Of course I was a bit provocative with the sentence in bold, but I haven't said this would be a bug, and if I have, I am sorry for that. I am in fact not familiar with what precisely are all the problems the C3 algorithm tries to solve, however my own intuitiv MRO would not be anything proposed so far, but: (B, mixin, A, mixin, object) or in the simpler, more trivial version (B, object, A, object) As you haven't mentioned this as a possibility at all, I guess having a class twice in this list produces some weird behaviour I do not know about yet - if someone can point out, that would be great. cheers, Stephan P.S.: I haven't said that I want a python 2.8, nevertheless thanks for pointing out that there won't be a python 2.8 (this I in fact have not known before)! I am familar with python 3.5 as well, however I just wanted to make clear where I am working and in which language the example code is runnable. On 10 December 2015 at 19:58, Andrew Barnert <abarnert@yahoo.com> wrote:
On Fri, Dec 11, 2015 at 7:03 AM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
In terms of people's expectations of super(), I think it would be intolerably surprising for it to be able to call a method from *the same* class. If your mixin has "def method(self): super().method()", it'll call itself (once) if called from B.
https://www.python.org/dev/peps/pep-0404/ http://blog.startifact.com/guido_no.jpg There are some special exemptions in 2.7 (for instance, some changes went through recently that improve security, even at the expense of some backward compatibility), but the bar is extremely high; behaviour has to either be recognizably buggy, or have significant security implications (cf hash randomization and certificate checking), to be changed in 2.7. I don't think the MRO fits either category, so any change would be 3.x only. As mentioned, though, your code works on 3.5/3.6 without changes and with the same semantics. ChrisA
@Chris thanks for pointing out this self-reference, I am still not sure whether this is really suprising to me, but at least I am still not done with thinking about it, so probably it is @Ned As I understood it, the MRO is not only for searching attributes - there it is indeed impressively redundant to put the same class twice into the MRO, thanks for pointing that out - but also for the hierarchy of the super() command @all thank you all for your comments and help. My current conclusion is that I will read about the C3 algorithm in crucially more detail and what it in fact is trying to solve ... and eventually may come back best, Stephan On 10 December 2015 at 21:43, Ned Batchelder <ned@nedbatchelder.com> wrote:
On Thu, Dec 10, 2015 at 1:51 PM, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
In that situation including the class twice results in an infinite loop. If your MRO is (B, mixin, A, mixin, object), then the chain of super calls looks like this: super(B, self).method() super(mixin, self).method() super(A, self).method() super(mixin, self).method() super(A, self).method() ... This happens because the second super call from the mixin class is identical to the first one. super doesn't know which appearance of mixin based on the arguments passed it, and if it naively assumes the first appearance, then nothing after the second appearance of mixin will ever be reached.
On Dec 10, 2015, at 12:03, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Dear all,
this is a highly sensitive topic, isn't it?
Well, it's an _interesting_ topic. If there really is something new to be done here that could improve the way inheritance works, that's a big deal, so people want to think about it, which means asking you challenging questions.
First, think about what kind of algorithm could get mixin to appear twice in the first list without also getting object to appear twice. I don't think there's any obvious way to do it. (Well, you could special-case object, but that doesn't help the more general diamond problem.) If you have a non-obvious answer to that, even though that still might not be a complete solution to what you really wanted, it would be a major contribution on its own.
As you haven't mentioned this as a possibility at all, I guess having a class twice in this list produces some weird behaviour I do not know about yet - if someone can point out, that would be great.
For an obvious problem: how could any cooperative inheritance tree ever super anything if object, which obviously doesn't cooperate, might end up on the MRO between one class and the next? Plus, if a class can get called twice on the super chain, how does it know which of the two times it's being called? Imagine a trivial mixin that just counts how many instances of its descendant get created. How do you avoid counting twice when a C gets constructed. (If you're thinking of using __mangled names to help, notice that _mixin__mangled and _mixin__mangled are the same name.) More generally: the code in a class generally assumes that it's not going to somehow super itself. Breaking that assumption, especially for code that adds attributes, makes the code much harder to write, and reason about. You probably can come up with a manual solution to these problems that doesn't work fully generally, but does work for your use case. (By adding specific rules on how classes in your tree interact, you can simplify the problem as far as you want.) Which you'd implement in a metaclass. But once you're writing a metaclass that interferes with the MRO mechanism, why do you care what the default MRO mechanism is? (That last might not be a rhetorical question--maybe the answer is "I wouldn't care, but it's too hard to override the setup of __mro__ in a metaclass __new__ method" or something, in which case there may be something to improve here.)
Stephan Sahm wrote:
I'm not sure what that would do to super calls. It's possible you would end up in a loop from Mixin -> A -> Mixin -> A -> ... But in any case, this MRO doesn't solve the fundamental problem that the MROs of B and A are inherently contradictory. *You* know what MRO you want in this particular case, but there's no way for Python to know that. -- Greg
On Dec 10, 2015, at 15:19, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
After thinking about this a bit: the MRO isn't really contradictory; what's contradictory is what he's expecting super() to do with his MRO. When called from within mixin.spam(), he wants it to give him A, but when calling from within mixin.spam(), he wants it to give him object. Since those two "whens" are identical, there's his problem. But if you never call super(), there's nothing wrong with it; any attribute provided by Mixin the first time means you'll never get to Mixin the second time. (I suppose you could hack that with quantum attributes that have an x% chance of being there each time you call __getattr__, but just don't do that...) Of course he wants to call super. Or, rather, he wants to call something that's kind of like super, but that handles his MRO in a non-contradictory way. A similar MI next-method protocol that went by index rather than by name would work: within mro[1].spam() it gives you mro[2]; within mro[3].spam() it gives you mro[4]. Python doesn't come with a function that does that, but you can write one, with a bit of kludgery. And you can easily write a metaclass that sets up the MRO you want. Which means you can build exactly what the OP is asking for. It still won't do what he wants. For one thing, like super, it depends on the entire class hierarchy cooperating, which includes the root not supering but everyone else doing so--but mixin is the root, and also elsewhere on the tree, so it has to both super and not super. And there are a couple other problems. But tightening up the requirements a bit, I think you can make something coherent out of this. It's definitely not the best way to solve his problem, and it's probably not a good way to solve _any_ problem--but it does work. See http://stupidpythonideas.blogspot.com/2015/12/can-you-customize-method-resol... for a fully worked through example.
Stephan Sahm wrote:
The reason Python disallows this is that B is saying that methods of Mixin should override those of A, whereas A is saying that its own methods should override those of Mixin. So there is a potential for B to break A's functionality. It may be that nothing gets broken in your particular case, but Python can't know that in general, so it errs on the safe side. -- Greg
participants (8)
-
Andrew Barnert -
Chris Angelico -
Emanuel Barry -
Greg Ewing -
Ian Kelly -
Ned Batchelder -
Stephan Sahm -
Stephen J. Turnbull