:-) 2011/4/14 Antoine Pitrou <solipsis@pitrou.net>

On Thu, 14 Apr 2011 08:15:10 -0500 Benjamin Peterson <benjamin@python.org> wrote:

...

2011/4/14 Ricardo Kirkner <ricardokirkner@gmail.com>:

...

Hi all,

I recently stumbled upon an issue with a class in the mro chain not calling super, therefore breaking the chain (ie, further base classes along the chain didn't get called). I understand it is currently a requirement that all classes that are part of the mro chain behave and always call super. My question is, shouldn't/wouldn't it be better, if python took ownership of that part, and ensured all classes get called, even if some class misbehaved?

For example, if using a stack-like structure, pushing super calls and popping until the stack was empty, couldn't this restriction be removed?

No. See line 2 of the Zen of Python.

You could have quoted it explicitly :) FWIW, line 2 is: Explicit is better than implicit.

Regards

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com

At 03:55 PM 4/14/2011 +0100, Michael Foord wrote:

Ricardo isn't suggesting that Python should always call super for you, but when you *start* the chain by calling super then Python could ensure that all the methods are called for you. If an individual method doesn't call super then a theoretical implementation could skip the parents methods (unless another child calls super).

That would break classes that deliberately don't call super. I can think of examples in my own code that would break, especially in __init__() cases. It's perfectly sensible and useful for there to be classes that intentionally fail to call super(), and yet have a subclass that wants to use super(). So, this change would expose an internal implementation detail of a class to its subclasses, and make "fragile base class" problems worse. (i.e., where an internal change to a base class breaks a previously-working subclass).

Exactly what Michael said. Stopping the chain going upwards is one thing. Stopping it going sideways is another. On Thu, Apr 14, 2011 at 12:37 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

On 14/04/2011 16:34, P.J. Eby wrote:

...

At 03:55 PM 4/14/2011 +0100, Michael Foord wrote:

...

Ricardo isn't suggesting that Python should always call super for you, but when you *start* the chain by calling super then Python could ensure that all the methods are called for you. If an individual method doesn't call super then a theoretical implementation could skip the parents methods (unless another child calls super).

That would break classes that deliberately don't call super.  I can think of examples in my own code that would break, especially in __init__() cases.

It's perfectly sensible and useful for there to be classes that intentionally fail to call super(), and yet have a subclass that wants to use super().  So, this change would expose an internal implementation detail of a class to its subclasses, and make "fragile base class" problems worse.  (i.e., where an internal change to a base class breaks a previously-working subclass).

It shouldn't do. What I was suggesting is that a method not calling super shouldn't stop a *sibling* method being called, but could still prevent the *parent* method being called.

Michael

-- http://www.voidspace.org.uk/

May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 15/04/2011 02:02, Greg Ewing wrote:

Michael Foord wrote:

...

What I was suggesting is that a method not calling super shouldn't stop a *sibling* method being called, but could still prevent the *parent* method being called.

There isn't necessarily a clear distinction between parents and siblings.

class A: ...

class B(A): ...

class C(A, B): ...

In C, is A a parent of B or a sibling of B?

For a super call in C, B is a sibling to A. For a super call in B, A is a parent. With the semantics I was suggesting if C calls super, but A doesn't then B would still get called. All the best, Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Mark Shannon wrote:

...

...

...

class A: pass class B(A): pass class C(A,B):pass

Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Cannot create a consistent method resolution order (MRO) for bases B, A

All right, but this is okay: class C(B, A): pass

Michael Foord wrote:

...

For a super call in C, B is a sibling to A. For a super call in B, A is a parent.

With the semantics I was suggesting if C calls super, but A doesn't then B would still get called.

which is contradicted by:

"Siblings", in the context of a single MRO are thus classes between which there is no sub-class/super-class relation.

So I maintain that the situation is far from clear. :-) -- Greg

On Apr 14, 2011, at 8:34 AM, P.J. Eby wrote:

At 03:55 PM 4/14/2011 +0100, Michael Foord wrote:

...

Ricardo isn't suggesting that Python should always call super for you, but when you *start* the chain by calling super then Python could ensure that all the methods are called for you. If an individual method doesn't call super then a theoretical implementation could skip the parents methods (unless another child calls super).

That would break classes that deliberately don't call super. I can think of examples in my own code that would break, especially in __init__() cases.

It's perfectly sensible and useful for there to be classes that intentionally fail to call super(), and yet have a subclass that wants to use super(). So, this change would expose an internal implementation detail of a class to its subclasses, and make "fragile base class" problems worse. (i.e., where an internal change to a base class breaks a previously-working subclass).

I agree. Better for someone to submit a recipe for a variant of super and see if there is any uptake. Raymond

On 14 Apr, 2011, at 18:10, Michael Foord wrote:

On 14/04/2011 17:02, Raymond Hettinger wrote:

...

On Apr 14, 2011, at 8:34 AM, P.J. Eby wrote:

...

At 03:55 PM 4/14/2011 +0100, Michael Foord wrote:

...

Ricardo isn't suggesting that Python should always call super for you, but when you *start* the chain by calling super then Python could ensure that all the methods are called for you. If an individual method doesn't call super then a theoretical implementation could skip the parents methods (unless another child calls super). That would break classes that deliberately don't call super. I can think of examples in my own code that would break, especially in __init__() cases.

It's perfectly sensible and useful for there to be classes that intentionally fail to call super(), and yet have a subclass that wants to use super(). So, this change would expose an internal implementation detail of a class to its subclasses, and make "fragile base class" problems worse. (i.e., where an internal change to a base class breaks a previously-working subclass). I agree. Better for someone to submit a recipe for a variant of super and see if there is any uptake.

In Python 3 super is treated specially by the compiler, so an alternative implementation that behaves similarly to the built-in one modulo this change is not possible.

Two use cases for the suggested alternative behaviour have been presented. What is the use case for a method not wanting to prevent its *sibling* methods in a multiple inheritance situation being called?

I believe the use case Phillip (and others) have presented is for methods preventing their *parent* methods being called.

What would the semantics be of a super that intentially calls all siblings? In particular what is the return value of such a call? The implementation can't know how to combine the implementations in the inheritance chain and should refuse the tempation to guess. Ronald

What would the semantics be of a super that intentially calls all siblings? In particular what is the return value of such a call? The implementation can't know how to combine the implementations in the inheritance chain and should refuse the tempation to guess.

I'll give you the example I came upon: I have a TestCase class, which inherits from both Django's TestCase and from some custom TestCases that act as mixin classes. So I have something like class MyTestCase(TestCase, Mixin1, Mixin2): ... now django's TestCase class inherits from unittest2.TestCase, which we found was not calling super. Even if this is a bug and should be fixed in unittest2, this is an example where I, as a consumer of django, shouldn't have to be worried about how django's TestCase class is implemented. Since I explicitely base off 3 classes, I expected all 3 classes to be initialized, and I expect the setUp method to be called on all of them. If I'm assuming/expecting unreasonable things, please enlighten me. Otherwise, there you have a real-world use case for when you'd want the sibling classes to be called even if one class breaks the mro chain (in this case TestCase). Thanks, Ricardo

Ethan Furman <ethan@stoneleaf.us> writes:

Here we have django's TestCase that does *not* want to call unittest2.TestCase (assuming that's not a bug), but it gets called anyway because the Mixin3 sibling has it as a base class. So does this mean that TestCase and Mixin3 just don't play well together?

Maybe composition instead of inheritance is the answer (in this case, anyway ;).

TestCase subclasses is a multiple-inheritance use case that I share. The mix-ins add test cases (methods named ‘test_’ on the mix-in class) to the TestCase subclass. I would prefer not to use multiple inheritance for this if it can be achieved in a better way. How can composition add test cases detectable by Python 2's ‘unittest’ and Python 3's ‘unittest2’? -- \ “The userbase for strong cryptography declines by half with | `\ every additional keystroke or mouseclick required to make it | _o__) work.” —Carl Ellison | Ben Finney

Ben Finney wrote:

Ethan Furman <ethan@stoneleaf.us> writes:

...

Here we have django's TestCase that does *not* want to call unittest2.TestCase (assuming that's not a bug), but it gets called anyway because the Mixin3 sibling has it as a base class. So does this mean that TestCase and Mixin3 just don't play well together?

Maybe composition instead of inheritance is the answer (in this case, anyway ;).

TestCase subclasses is a multiple-inheritance use case that I share. The mix-ins add test cases (methods named ‘test_’ on the mix-in class) to the TestCase subclass. I would prefer not to use multiple inheritance for this if it can be achieved in a better way.

How can composition add test cases detectable by Python's ‘unittest’?

Metaclasses, if's that an option... 8<------------------------------------------------------------- import unittest from composite import Composite # python 3 only class Spam(): def test_spam_01(self): print('testing spam_01') def test_spam_02(self): print('testing spam_02') class Eggs(): def test_eggs_01(self): print('testing eggs_01') def test_eggs_02(self): print('testing eggs_02') class TestAll( unittest.TestCase, metaclass=Composite, parts=(Spam, Eggs)): def setUp(self): print('Setting up...') def tearDown(self): print('Tearing down...') def test_something(self): print('testing something') if __name__ == '__main__': unittest.main() 8<------------------------------------------------------------- or a class decorator 8<------------------------------------------------------------- class Compose(object): # python 3 only def __init__(self, *parts): self.parts = parts def __call__(self, func): for part in self.parts: for attr in dir(part): if attr[:2] == attr[-2:] == '__': continue setattr(func, attr, getattr(part, attr)) return func import unittest class Spam(): def test_spam_01(self): print('testing spam_01') def test_spam_02(self): print('testing spam_02') class Eggs(): def test_eggs_01(self): print('testing eggs_01') def test_eggs_02(self): print('testing eggs_02') @Compose(Spam, Eggs) class TestAll(unittest.TestCase): def setUp(self): print('Setting up...') def tearDown(self): print('Tearing down...') def test_something(self): print('testing something') if __name__ == '__main__': unittest.main() 8<------------------------------------------------------------- The decorator, as written, doesn't work on py2, and doesn't do any error checking (so overwrites methods in the final class) -- but I'm sure it could be spiffed up. ~Ethan~

Ben Finney wrote:

Ethan Furman <ethan@stoneleaf.us> writes:

...

Ben Finney wrote:

...

How can composition add test cases detectable by Python's ‘unittest’?

Metaclasses, if's that an option... […] or a class decorator […]

Both interesting, thank you. But Python 3 isn't an option for several projects where I'd like to use this.

Well, I'm sure there's a way to do it -- alas, I lack the time to find it either in the docs, archives, or by experimentation. What I did find is that if you have your functions in modules, instead of in classes, it works fine in Python 2.6+. 8<---spam.py------------------------------------------------------- def test_spam_01(self): print('testing spam_01') def test_spam_02(self): print('testing spam_02') 8<----------------------------------------------------------------- 8<---eggs.py------------------------------------------------------- def test_eggs_01(self): print('testing eggs_01') def test_eggs_02(self): print('testing eggs_02') 8<----------------------------------------------------------------- 8<---test_compose.py----------------------------------------------- import unittest class Compose(object): # 2.6-2.7, functions must be in modules def __init__(self, *parts): self.parts = parts def __call__(self, func): for part in self.parts: for attr in dir(part): if attr[:2] == attr[-2:] == '__': continue if getattr(cls, attr, None): raise AttributeError( "%s already exists in %s" % (attr, cls)) setattr(func, attr, getattr(part, attr)) return func @Compose(spam, eggs) class TestAll(unittest.TestCase): def setUp(self): print('Setting up...') def tearDown(self): print('Tearing down...') def test_something(self): print('testing something') if __name__ == '__main__': unittest.main() 8<---test_compose.py----------------------------------------------- Compose now has rudimentary error checking, and if can live with your extras living in their own .py files, this might work for you. ~Ethan~

Ben Finney wrote:

Both interesting, thank you. But Python 3 isn't an option for several projects where I'd like to use this.

Okay, I took some time to try and figure this out (have I mentioned how much I love Python 3's clean-up?), and I have something -- lightly tested with methods, properties, and attributes, with the objects being kept in classes instead of modules. Posted to ActiveState. http://code.activestate.com/recipes/577658-composition-of-classes-instead-of... Hope this helps! ~Ethan~

Raymond Hettinger wrote:

If an external non-cooperative class needs to be used, then it should be wrapped in a class that makes an explicit __init__ call to the external class and then calls super().__init__() to continue the forwarding.

I don't think it's as simple as that. Isn't that super() call going to call the __init__() method that you just explicitly called *again*? Seems like you would at least need to use super(BaseClass)... to skip the one you just called. But it's not immediately obvious to me that this won't ever skip other classes that you *do* want to call. -- Greg

On 15/04/2011 02:23, Steven D'Aprano wrote:

Ricardo Kirkner wrote:

...

I have a TestCase class, which inherits from both Django's TestCase and from some custom TestCases that act as mixin classes. So I have something like

class MyTestCase(TestCase, Mixin1, Mixin2): ...

now django's TestCase class inherits from unittest2.TestCase, which we found was not calling super. Even if this is a bug and should be fixed in unittest2, this is an example where I, as a consumer of django, shouldn't have to be worried about how django's TestCase class is implemented. Since I explicitely base off 3 classes, I expected all 3 classes to be initialized, and I expect the setUp method to be called on all of them.

If I'm assuming/expecting unreasonable things, please enlighten me.

If we treat django's failure to use super as a bug, you want the Python language to work-around that bug so that:

What you say (that this particular circumstance could be treated as a bug in django) is true, however consider the "recently" introduced problem caused by object.__init__ not taking arguments. This makes it impossible to use super correctly in various circumstances. http://freshfoo.com/blog/object__init__takes_no_parameters Given the following classes (Python 3): class A: def __init__(self, a): print ('A') class B: def __init__(self, a): print ('B') class C(B): def __init__(self, a): print ('C') super().__init__(a) It is impossible to inherit from both C and A and have all parent __init__ methods called correctly. Changing the semantics of super as described would fix this problem. For: class D(C, A): def __init__(self, a): super().__init__(a) D(1) This is printed: C B (A __init__ is not called). For this: class D(A, C): def __init__(self, a): super().__init__(a) D(1) The following is printed: A (B and C __init__ methods are not called.) All the best, Michael Foord

"I, as a consumer of django, shouldn't have to be worried about bugs in django". (For at least one class of bug.)

If we *don't* treat django's failure to use super as a bug, but as a deliberate design choice, then you are trying to do something which django doesn't support. Possibly *deliberately* doesn't support. You want the Python language to add that support so that:

"I, as a consumer of django, shouldn't have to be worried about whether django supports what I want to do or not".

Either way you look at it, I think it's extremely unreasonable to expect the language to work-around bugs in third-party applications, or to add features to them that the third-party developers either didn't consider or don't want.

Multiple inheritance is tricky enough to get right without adding "Do What I Mean" black magic to it. I'd rather work around bugs in third-party classes than try to deal with Python actively subverting the code I read and write by mysteriously calling superclass methods where there is no call to a superclass method.

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

On 15/04/2011 16:18, Carl Meyer wrote:

On 04/15/2011 08:53 AM, Michael Foord wrote:

...

...

If we treat django's failure to use super as a bug, you want the Python language to work-around that bug so that: What you say (that this particular circumstance could be treated as a bug in django) is true, Just as a side note: if there is a bug demonstrated here, it is in unittest2, not Django. Django's TestCase subclasses don't even override __init__ or setUp, so there is no opportunity for them to call or fail to call super() in either case.

If you re-read Ricardo's original presentation of the case, he correctly noted that it is unittest2's TestCase which does not call super() and thus prevents cooperative multiple inheritance. I'm not sure who in this thread first mis-read his post and called it a possible bug in Django, but it was a mis-reading which now appears to be self-propagating ;-)

Well yes, but it is also a bug in the copy of unittest2 embedded in django - so whilst it can be fixed in unittest2 (simply deleting the setUp and tearDown methods which do nothing but override unittest.TestCase.setUp and tearDown) it *also* needs to be fixed in django. This particular issue does illustrate the problem well though - the methods in unittest2 don't call up to their parent class (which is fine because those methods are empty), but in not calling up also they prevent sibling methods being called in a multiple inheritance situation. So for those who have been saying that not wanting to call up to parents is a valid use case, yes I quite agree. But you have to be aware that because of the semantics of super, not calling up to your parents basically prevents those methods being used in the presence of multiple inheritance. All the best, Michael Foord

Carl _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html

Michael Foord wrote:

consider the "recently" introduced problem caused by object.__init__ not taking arguments. This makes it impossible to use super correctly in various circumstances.

...

It is impossible to inherit from both C and A and have all parent __init__ methods called correctly. Changing the semantics of super as described would fix this problem.

I don't see how, because auto-super-calling would eventually end up trying to call object.__init__ with arguments and fail. You might think to "fix" this by making a special case of object.__init__ and refraining from calling it. But the same problem arises in a more general way whenever some class in the mix has a method with the right name but the wrong signature, which is likely to happen if you try to mix classes that weren't designed to be mixed together. -- Greg

Michael Foord wrote:

On 15/04/2011 02:23, Steven D'Aprano wrote: [...]

...

If we treat django's failure to use super as a bug, you want the Python language to work-around that bug so that:

What you say (that this particular circumstance could be treated as a bug in django) is true, however consider the "recently" introduced problem caused by object.__init__ not taking arguments. This makes it impossible to use super correctly in various circumstances. [...] It is impossible to inherit from both C and A and have all parent __init__ methods called correctly. Changing the semantics of super as described would fix this problem.

So you say. I don't have an an opinion on whether or not you are technically correct, but adding DWIM black-magic to super scares me. It scares me even if it were guaranteed to *only* apply to __init__, but if it applied to arbitrary methods, it frankly terrifies me. If it were limited to only apply to __init__, there would be a constant stream of requests that we loosen the restriction and "make super just work" for all methods, despite the dangers of DWIM code. -- Steven

Ricardo Kirkner wrote:

I'll give you the example I came upon:

I have a TestCase class, which inherits from both Django's TestCase and from some custom TestCases that act as mixin classes. So I have something like

class MyTestCase(TestCase, Mixin1, Mixin2): ...

now django's TestCase class inherits from unittest2.TestCase, which we found was not calling super. Even if this is a bug and should be fixed in unittest2, this is an example where I, as a consumer of django, shouldn't have to be worried about how django's TestCase class is implemented.

I have to disagree -- anytime you are using somebody else's code you need to be aware of what it's supposed to do -- especially when playing with multiple inheritance. This response to the decorator I wrote for this situation may be helpful: Carl Banks wrote (on Python-List):

The problem is that he was doing mixins wrong. Way wrong.

Here is my advice on mixins:

Mixins should almost always be listed first in the bases. (The only exception is to work around a technicality. Otherwise mixins go first.)

If a mixin defines __init__, it should always accept self, *args and **kwargs (and no other arguments), and pass those on to super().__init__. Same deal with any other function that different sister classes might define in varied ways (such as __call__).

A mixin should not accept arguments in __init__. Instead, it should burden the derived class to accept arguments on its behalf, and set attributes before calling super().__init__, which the mixin can access.

If you insist on a mixin that accepts arguments in __init__, then it should should pop them off kwargs. Avoid using positional arguments, and never use named arguments. Always go through args and kwargs.

If mixins follow these rules, they'll be reasonably safe to use on a variety of classes. (Maybe even safe enough to use in Django classes.)

~Ethan~

P.J. Eby wrote:

It's perfectly sensible and useful for there to be classes that intentionally fail to call super(), and yet have a subclass that wants to use super().

One such case is where someone is using super() in a single-inheritance environment as a way of not having to write the base class name explicitly into calls to base methods. (I wouldn't recommend using super() that way myself, but some people do.) In that situation, any failure to call super() is almost certainly deliberate. -- Greg

R. David Murray wrote:

Why not? It seems more useful than using it for chaining, especially given the compiler hack in Python3.

Because it's prone to doing the wrong thing if the class using it is ever involved in multiple inheritance. If you're expecting the call to go to a particular class, it's safer to explicitly name that class. -- Greg

Argh! Sorry list. I meant to discard the post that was just sent. Please accept my humblest apologies... Mark