Re: [Python-ideas] Keyword for direct pass through of kwargs to super

[Steven D'Aprano]
The whole point I was trying to make is: If it doesn’t make any sense to init a class with **kwargs: why write down that it would (or even **could**) accept them? Shouldn’t the init tell you something like 'If you instantiate this class then this is everything you can give me'? Well, right now in addition it says 'Just give me anything with keywords‘. I would think that something like 'Oh, and if I am in the middle of an MRO: I will pass everything down the line‘ would be a much better description of it’s true intentions instead. [Carl Smith]
I think that (as Raymond Hettinger once said) 'super is super', but can’t you make it a bit smarter with telling it: 'Hey - If you don’t expect 'eggs', keep calm, it probably (or rather certainly) wasn’t meant for you so just pass it on to your super'. Best, Michael

On Sat, May 26, 2018 at 5:39 PM, Michael Lohmann <mial.lohmann@gmail.com> wrote:
Let's get some somewhat-more-plausible examples. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", **kw): print("This pizza has %s pineapple." % pineapple) super().__init__(**kw) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" menu = [ BestPizza(size="large", price=12.50), BestPizza(size="personal", price=8.50), ] Okay. Now what are the possible keyword args for a BestPizza? The *entire set* of args for the hierarchy. You can set size, price, pineapple, crust_cheese, and surcharge. The fact that some of them (or even ALL of them, at the leaf node) are simply passed on up the chain doesn't change the fact that they're all valid. Does that make sense? ChrisA

Let me put it this way: class A(object): def __init__(self, a_value, **kwargs): print("This is a value:", a_value) super().__init__(**kwargs) Which parameters does `A` take when being initialized? Whenever you give any kwargs when directly instantiating `A` they will be passed down to super which in this case is `object`. And now to the follow-up question: Can you tell me which kwargs object takes as an input for it’s __init__? So does it EVER make ANY sense to specify them if you DIRECTLY create an instance of `A`? But now let’s say your MRO is `SuperClass, A, B`, then A should better be able to forward the kwargs, so currently you need to directly hand them to `A` and rely on the fact that it passes them on to `B`. Why should `A` even get those kwargs in the first place? They could just be "passed around it" as soon as the interpreter sees that they are in the what currently would be **kwargs and later reunited with the arguments that the super().__init__ call has. Of course this can’t be done by default. So why not tell the interpreter with something like "@pass_through_kwargs" that it should do this for you? [Chris Angelico]
class HaveYouEverTriedThis(Pizza, Lasagna): """Well, this is just bizarre""" Now suddenly `Pizza` would have to accept kwargs. Why????????? It didn’t "learn anything new". It can’t do anything with them on it’s own. Shouldn’t you expect from a function that in the brackets you are shown what is possible to actually call this function on? You could never actually call Pizza with kwargs, so 1) Why should you be able to do so in the first place? and 2) Why show the programmer that you could?. Just to make the MRO work. And I would suggest that to handle this it would be a lot more elegant to say: "If you are in the middle of an MRO: just pass everything unexpected down“ [Steven D'Aprano]

On Sat, May 26, 2018 at 7:22 PM, Michael Lohmann <mial.lohmann@gmail.com> wrote:
Right, which means that Pizza and Lasagna are not compatible classes in that way. If you were to try to concoct some sort of, I don't know, layered conglomerate with meat, tomato, and pizza bases, you'd have to write an init method that does that for you. class Pizzagna(Pizza, Lasagna): def __init__(self, *, number_of_layers, **kw): Lasagna.__init__(self, number_of_layers=number_of_layers) Pizza.__init__(self, **kw) The Pizza class was deliberately designed to be an apex class - one that is the head of a hierarchy. The Lasagna class isn't part of that hierarchy, so merging it in takes some extra work - a pizza can't automatically subsume a lasagna. We had the family over here for dinner tonight, and we ate two pizzas. Now I'm wondering if a Pizzagna would have been a better choice... Anyhow. Further discussion about this should probably go onto python-list. ChrisA

Right, which means that Pizza and Lasagna are not compatible classes in that way.
Okay, let me try it one final time with the original pizza example. Let’s assume that your restaurant has a special offer on all Hawaiian Pizzas where you can get all sizes for 10$. Now the only reasonable thing to pass into **kw is size, so let’s say that for the readability of the class we decide to replace it. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): # No **kw here since no longer needed print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" BestPizza(crust_cheese="gouda“) # Doesn’t work since Hawaii doesn’t bypass it But now the HawaiianPizza gets annoyed because it doesn’t know what to do with the crust_cheese. So just to make BestPizza work I will have to add **kw to HawaiianPizza again. But now let’s say we have a hungry programmer that just takes a short look and sees that HawaiianPizza is a subclass of Pizza and he thinks "Okay then, just get some HawaiianPizza(price=8)". In fact if he directly orders any HawaiianPizza there is nothing he can pass in into **kw that wouldn’t result in an error. I am sorry to annoy you all with this. Maybe this problem just isn’t as common as I thought it was… Michael

On 2018-05-26 02:22, Michael Lohmann wrote:
If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). But you can't, because if you pass on extra **kwargs, that will fail if the class inherits directly from object, but if you don't pass on extra **kwargs, that will fail if the class doesn't inherit directly from object. Is that correct? I agree that it is somewhat somewhat awkward that "is this the top class in the hierarchy" is something that has to be known when writing a class. I think this would be ameliorated by having "object" accept and ignore extra arguments. (I seem to recall that was decided to be a bad idea at some time in the past, though.) But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). Well, it does not necessarily inherit from object, but from any class, that does not accept any more kwargs. E.g. if you have a diamond structure as your class-hierachy then one branch could forward information to the second (if you understand what I mean?).
But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem.
I intended the following text for the python-list mailing-list, but it I think I might have structured my ideas a bit better than in my previous messages and the summary in __Reason__ might tackle why this could be a nice idea (spoiler: make super even more super by doing things if it is NOT there in a clever manner). Let us start with a simple class: class Aardvark: def __init__(self, quantity): print("There is some quantity:", quantity) Well, that works nicely and we can go and send our Aardvark to survey some quantities. But if we now want to get the multi-inheritance to work properly we need to change it to: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever“ % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) But if you now look at the declaration of the Aardvark .__init__, it seems like you could instantiate it with **kwargs. This in fact is not true. As soon as you create a direct instance of Aardvark, `object` as the super() doesn’t accept any kwargs. So basically I think that the parameters for the init should just say `quantity ` while still preserving the functionality. Now that obviously doesn’t work until now. But could you add something that lets this class tell the interpreter instead: "Hey, could you just forward anything that this init doesn’t need to super().__init__" ? I have something like this in mind: class Aardvark: @bypass_kwargs_to_super def __init__(self, *, quantity): print("There is some quantity:", quantity) super().__init__() This would collect everything "behind the scenes" that usually **kwargs would have collected and "append" it to the super call. Question: If Aardvark knows that he really does not need the kwargs himself: why give them to him in the first place? I mean, I trust him - he seems like a very nice guy, but he might have accidentally forgotten to forward them unintentionally. You would obviously still be able to use **kwargs the usual way but then you couldn’t use this technique and would need to take care of passing down all the information to super() yourself as usual. __Reason__: With something like this it is immediately obvious that Aardvark ONLY takes `quantity` as an input if you instantiate it directly but if subclassed it is able to hand information down the MRO to something from a different "Branch". And in addition to the better human readability: any form of automated docstring-genaration now can be certain that you don’t do anything with (or to) the **kwargs (since the init doesn’t get them in the first place). In addition it could make super even more super by doing something if it is NOT there as proposed in the second of the the following problems. Of course you need to be quite clever to do this properly, for example 1) What to do if there are collisions between the bypassed kwargs and the ones from the init call? - Probably keep the ones you bypassed since they come from the top of the MRO 2) What do you do if super().__init__ was not called? The most clever thing would be to go „up and to the next branch“ of the inheritance diagram. As in: if Aardvark is a Subclass of Animal, don’t call its init but directly Clevers - (you would have to look up the MRO of the super of Aardvark and skip them in the Ethel MRO before calling the next init automatically). In theory you could also add something that forwards *args as well but the usage of that is probably much more limited... On thing that might look a bit strange is that you actually could actually pass in additional kwargs despite the init saying otherwise: `Aardvark(quantity='some spam', something="Object will throw an error now unexpected kwarg")` and this would then throw an error that "object (instead of Aardvark) does not expect something". But I guess that would be okay since the init now is much better readable of what it actually expects. Michael

Brendan Barnwell wrote:
This shouldn't be a problem if each method along the way absorbs all the arguments it knows about, and only passes on the ones it doesn't. E.g. class A: def __init__(self, alpha, **kwds): print("alpha =", alpha) super().__init__(**kwds) class B(A): def __init__(self, beta, **kwds): print("beta =", beta) super().__init__(**kwds) b = B(alpha = 17, beta = 42) Both A and B here pass on a **kwds argument, but by the time it get to object, there are no arguments left, so it's fine. -- Greg

I realized that bypassing kwargs is probably the least important thing of this idea - so if implemented it definitely had to get a better name. Just look at the following example: class Magic: magic_number = 42 def __init__(self): A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes class Foo(Magic): def __init__(self): print("foo") # Let's say we want magic_number==42 in Foo # So you can’t call super().__init__ here if it is not in the middle of an MRO # Technically you could get it working by going to the first init in the MRO of self after Foo \ # that isn’t in the MRO of Foo but you can see that this gets quite ugly to write (and read!). # And then you would still have the problem of indicating that Foo seems to accepted kwargs as input class Bar: def __init__(self, bar): print("bar:", bar) class FooBar(Foo, Bar): def __init__(self): # There is no easy way right now to avoid writing Foo.__init__() Bar.__init__(bar="bar“) But if Foo adopted this protocol of "automated MRO handling"/"cooperativity" (or however you want to call it) Bar.__init__ could be called automatically. What do I mean by that? Well, basically do automatically what I described in the comment of Foo if no super().__init__ call was registered in any cooperative class. Okay, this example might be a bit far-fetched, but it shows, that you could easily get the MRO working as expected with a simple super().__init__(bar="bar") in FooBar.
What bugs me is that in my example from yesterday ( class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever“ % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) ) if you want to instantiate an Aardvark directly there is NO WAY EVER that you could give him any kwargs. So why should the __init__ indicate something else? Well, just to make the MRO work. All I want is to make it as obvious as possible that an Aardvark ONLY takes `quantity` as input, but is fully "cooperative" with other classes if it is in the middle of the MRO (by which I mean that it will automatically call the __init__ and hand on any kwargs it didn’t expect to a class from a different branch of the class hierarchy).

Everything my idea has to offer really is just reasonable if you don’t have single inheritance only
I would like to correct myself immediately on that one: In the Pizza-example (from yesterday as well) it would be possible to overwrite the default price of the HawaiianPizza with the merging of bypassed kwargs with the ones from the call itself. Let us assume that the exemption of the 10$ price for all HawaiianPizzas would be a Family-sized pizza for 20$: class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) @cooperative class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class FamilyHawaiianPizza(HawaiianPizza): def __init__(self, *, pineapple="chunked"): super().__init__(price=20, size=20, pineapple=pineapple) #This would overwrite the price of the HawaiianPizza But this would not be the real intention of the usage, but just a side effect of correctly handling a simple diamond-like hierarchy. The reason that bypassed kwargs should overwrite them is that they come from up the MRO and so obviously should have priority. Sorry to spam you all, Have a nice Sunday! Michael

Sorry for that. Yes, it should have been Magic (I renamed the class after writing it and didn’t pay attention). I just wanted to override the class-variable `magic_number` to give a reason why I don’t ever want to call Magic.__init__ in Foo. If you want, you can have this class instead: class Magic: def __init__(self): raise RuntimeError("Do not initialize this class") but I figured that this might look a bit artificial...

I'd say NOT wanting to call an __init__ method of a superclass is a rather uncommon occurence. It's generally a huge error. So I think it's worth not accomodating that. 2018-05-28 9:27 GMT+02:00 Michael Lohmann <mial.lohmann@gmail.com>:

I will give you an example then where I am absolutely fine with calling super().__init__ in all classes and describe why I am not really satisfied with the current status. (It is from my email from yesterday 17:19 GMT):

2018-05-28 9:44 GMT+02:00 Michael Lohmann <mial.lohmann@gmail.com>:
This is more an issue with your Ethel class. In *normal* subclassing, it's init would look like: class Ethel(Aardvark, Clever): def __init__(self, **kwargs): super().__init__(**kwargs) and you'd instantiate it like: e = Ethel(quantity="some spam", cleverness=1000) or even (because keyword argument order doesn't matter): e = Ethel(cleverness=1000, quantity="some spam") Because don't forget: assert isinstance(e, Ethel) assert isinstance(e, Aardvark) assert isinstance(e, Clever) will all pass perfectly fine. It's not just an Ethel or an Aardvark, it's also a Clever. (which doesn't sound like it makes sense...perhaps clever should be an attribute on an Aardvark/Ethel instead ?). That all aside, for a moment. You actually CAN call object.__init__(**kwargs) no problem - as long as kwargs is empty. I'd have written your classes like this: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1, **kwargs): print("You are %d clever" % cleverness) super().__init__(**kwargs) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self, **kwargs): super().__init__(**kwargs) Just try it - it works perfectly fine. Each constructor will consume the keywords meant for it, therefore, the last super() call will call the object constructor without keyword arguments. **kwargs is the price we have to pay for good multiple inheritance - and IMO, it's a low one. If we're talking about signalling what the arguments mean, just ensure you have a good docstring. class Aardvark: def __init__(self, quantity, **kwargs): """ Aardvarks are gentle creatures, and therefore cooperate with multiple inheritance. :param quantity: The quantity of Aardvark(s). """

Michael Lohmann wrote:
You seem to want the signature of Ethel.__init__ to indicate exactly what arguments it can accept. But even with your proposed feature, that isn't going to happen. It will only show arguments that appear explicitly in Ethel.__init__'s argument list. If it didn't override the default value for cleverness, you would never know that it was a possible parameter. In situations like this there's no substitute for hand-written documentation, and that can include explaining what is allowed for **kwds. -- Greg

Michael Lohmann wrote:
But your original example looks just as artificial. Skipping the initialisation of a class you're inheriting from is an extremely weird thing to do, and in any real-life situation there's almost certainly a better design. In any case, I don't see how this has anything to do with invisible passing of **kwds. In short, I don't understand what you're saying at all. -- Greg

On Sat, May 26, 2018 at 09:39:14AM +0200, Michael Lohmann wrote:
What makes you say that? Since the kwargs are passed on to the super().__init__ call, the eggs argument is passed onto the super class of Aardvark. What makes you so certain that Aardvark's parent (or grandparent, or great-grandparent...) doesn't take a parameter "eggs"? Considering that *I* made this example up, it is a bit rich for you to tell me that "eggs" isn't used. Of course it is used, by one of the superclasses. That's why it was provided.
(or at some point **kwargs are being accepted and not passed on to super which would be terrible on its own).
No, it is a standard technique to collect keyword arguments, process those your class cares about, and pass the rest on to super(). Eventually the class second from the top (the class which inherits directly from object) MUST NOT pass those arguments to super, since object doesn't accept any arguments: you can't keep passing arguments up the MRO all the way to the top. At one point or another, each argument must be used and discarded.
If your class doesn't accept **kwargs, then don't put **kwargs in the parameter list.
Shouldn’t the init tell you something like 'If you instantiate this class then this is everything you can give me'?
The way to do that is to write the __init__ of the class that only takes the parameters you want.
Well, right now in addition it says 'Just give me anything with keywords‘.
Only if you intentionally add **kwargs to the parameter list. Why are you putting **kwargs in the parameter list if you don't want to accept them?
"Somebody"? Who? It's my class. If "somebody" used kwargs.pop('eggs') it must have been me. If I did it, it means that my class wants to consume the eggs parameter and NOT pass it on.
I don't understand this comment one bit.
The way we do that is by explicitly collecting **kwargs and explicitly passing it on to the super call. I don't think I understand the scenario you have in mind. Here is what I have in mind: class Foo: def __init__(self, spam): self.spam = spam # this is a null-op, so it could be left out super().__init__() # calls object.__init__ which does nothing class Bar(Foo): def __init__(self, eggs, **kwargs): self.eggs = eggs super().__init__(**kwargs) class Baz(Bar): def __init__(self, cheese, **kwargs): self.cheese = cheese super().__init__(**kwargs) class Aardvark(Baz): def __init__(self, myarg, **kwargs): self.myarg = myarg super().__init__(**kwargs) obj = Aardvark(27, spam=3, eggs=5, cheese=True) What situation do you have in mind? -- Steve

On Sat, May 26, 2018 at 5:39 PM, Michael Lohmann <mial.lohmann@gmail.com> wrote:
Let's get some somewhat-more-plausible examples. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", **kw): print("This pizza has %s pineapple." % pineapple) super().__init__(**kw) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" menu = [ BestPizza(size="large", price=12.50), BestPizza(size="personal", price=8.50), ] Okay. Now what are the possible keyword args for a BestPizza? The *entire set* of args for the hierarchy. You can set size, price, pineapple, crust_cheese, and surcharge. The fact that some of them (or even ALL of them, at the leaf node) are simply passed on up the chain doesn't change the fact that they're all valid. Does that make sense? ChrisA

Let me put it this way: class A(object): def __init__(self, a_value, **kwargs): print("This is a value:", a_value) super().__init__(**kwargs) Which parameters does `A` take when being initialized? Whenever you give any kwargs when directly instantiating `A` they will be passed down to super which in this case is `object`. And now to the follow-up question: Can you tell me which kwargs object takes as an input for it’s __init__? So does it EVER make ANY sense to specify them if you DIRECTLY create an instance of `A`? But now let’s say your MRO is `SuperClass, A, B`, then A should better be able to forward the kwargs, so currently you need to directly hand them to `A` and rely on the fact that it passes them on to `B`. Why should `A` even get those kwargs in the first place? They could just be "passed around it" as soon as the interpreter sees that they are in the what currently would be **kwargs and later reunited with the arguments that the super().__init__ call has. Of course this can’t be done by default. So why not tell the interpreter with something like "@pass_through_kwargs" that it should do this for you? [Chris Angelico]
class HaveYouEverTriedThis(Pizza, Lasagna): """Well, this is just bizarre""" Now suddenly `Pizza` would have to accept kwargs. Why????????? It didn’t "learn anything new". It can’t do anything with them on it’s own. Shouldn’t you expect from a function that in the brackets you are shown what is possible to actually call this function on? You could never actually call Pizza with kwargs, so 1) Why should you be able to do so in the first place? and 2) Why show the programmer that you could?. Just to make the MRO work. And I would suggest that to handle this it would be a lot more elegant to say: "If you are in the middle of an MRO: just pass everything unexpected down“ [Steven D'Aprano]

On Sat, May 26, 2018 at 7:22 PM, Michael Lohmann <mial.lohmann@gmail.com> wrote:
Right, which means that Pizza and Lasagna are not compatible classes in that way. If you were to try to concoct some sort of, I don't know, layered conglomerate with meat, tomato, and pizza bases, you'd have to write an init method that does that for you. class Pizzagna(Pizza, Lasagna): def __init__(self, *, number_of_layers, **kw): Lasagna.__init__(self, number_of_layers=number_of_layers) Pizza.__init__(self, **kw) The Pizza class was deliberately designed to be an apex class - one that is the head of a hierarchy. The Lasagna class isn't part of that hierarchy, so merging it in takes some extra work - a pizza can't automatically subsume a lasagna. We had the family over here for dinner tonight, and we ate two pizzas. Now I'm wondering if a Pizzagna would have been a better choice... Anyhow. Further discussion about this should probably go onto python-list. ChrisA

Right, which means that Pizza and Lasagna are not compatible classes in that way.
Okay, let me try it one final time with the original pizza example. Let’s assume that your restaurant has a special offer on all Hawaiian Pizzas where you can get all sizes for 10$. Now the only reasonable thing to pass into **kw is size, so let’s say that for the readability of the class we decide to replace it. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): # No **kw here since no longer needed print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" BestPizza(crust_cheese="gouda“) # Doesn’t work since Hawaii doesn’t bypass it But now the HawaiianPizza gets annoyed because it doesn’t know what to do with the crust_cheese. So just to make BestPizza work I will have to add **kw to HawaiianPizza again. But now let’s say we have a hungry programmer that just takes a short look and sees that HawaiianPizza is a subclass of Pizza and he thinks "Okay then, just get some HawaiianPizza(price=8)". In fact if he directly orders any HawaiianPizza there is nothing he can pass in into **kw that wouldn’t result in an error. I am sorry to annoy you all with this. Maybe this problem just isn’t as common as I thought it was… Michael

On 2018-05-26 02:22, Michael Lohmann wrote:
If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). But you can't, because if you pass on extra **kwargs, that will fail if the class inherits directly from object, but if you don't pass on extra **kwargs, that will fail if the class doesn't inherit directly from object. Is that correct? I agree that it is somewhat somewhat awkward that "is this the top class in the hierarchy" is something that has to be known when writing a class. I think this would be ameliorated by having "object" accept and ignore extra arguments. (I seem to recall that was decided to be a bad idea at some time in the past, though.) But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). Well, it does not necessarily inherit from object, but from any class, that does not accept any more kwargs. E.g. if you have a diamond structure as your class-hierachy then one branch could forward information to the second (if you understand what I mean?).
But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem.
I intended the following text for the python-list mailing-list, but it I think I might have structured my ideas a bit better than in my previous messages and the summary in __Reason__ might tackle why this could be a nice idea (spoiler: make super even more super by doing things if it is NOT there in a clever manner). Let us start with a simple class: class Aardvark: def __init__(self, quantity): print("There is some quantity:", quantity) Well, that works nicely and we can go and send our Aardvark to survey some quantities. But if we now want to get the multi-inheritance to work properly we need to change it to: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever“ % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) But if you now look at the declaration of the Aardvark .__init__, it seems like you could instantiate it with **kwargs. This in fact is not true. As soon as you create a direct instance of Aardvark, `object` as the super() doesn’t accept any kwargs. So basically I think that the parameters for the init should just say `quantity ` while still preserving the functionality. Now that obviously doesn’t work until now. But could you add something that lets this class tell the interpreter instead: "Hey, could you just forward anything that this init doesn’t need to super().__init__" ? I have something like this in mind: class Aardvark: @bypass_kwargs_to_super def __init__(self, *, quantity): print("There is some quantity:", quantity) super().__init__() This would collect everything "behind the scenes" that usually **kwargs would have collected and "append" it to the super call. Question: If Aardvark knows that he really does not need the kwargs himself: why give them to him in the first place? I mean, I trust him - he seems like a very nice guy, but he might have accidentally forgotten to forward them unintentionally. You would obviously still be able to use **kwargs the usual way but then you couldn’t use this technique and would need to take care of passing down all the information to super() yourself as usual. __Reason__: With something like this it is immediately obvious that Aardvark ONLY takes `quantity` as an input if you instantiate it directly but if subclassed it is able to hand information down the MRO to something from a different "Branch". And in addition to the better human readability: any form of automated docstring-genaration now can be certain that you don’t do anything with (or to) the **kwargs (since the init doesn’t get them in the first place). In addition it could make super even more super by doing something if it is NOT there as proposed in the second of the the following problems. Of course you need to be quite clever to do this properly, for example 1) What to do if there are collisions between the bypassed kwargs and the ones from the init call? - Probably keep the ones you bypassed since they come from the top of the MRO 2) What do you do if super().__init__ was not called? The most clever thing would be to go „up and to the next branch“ of the inheritance diagram. As in: if Aardvark is a Subclass of Animal, don’t call its init but directly Clevers - (you would have to look up the MRO of the super of Aardvark and skip them in the Ethel MRO before calling the next init automatically). In theory you could also add something that forwards *args as well but the usage of that is probably much more limited... On thing that might look a bit strange is that you actually could actually pass in additional kwargs despite the init saying otherwise: `Aardvark(quantity='some spam', something="Object will throw an error now unexpected kwarg")` and this would then throw an error that "object (instead of Aardvark) does not expect something". But I guess that would be okay since the init now is much better readable of what it actually expects. Michael

Brendan Barnwell wrote:
This shouldn't be a problem if each method along the way absorbs all the arguments it knows about, and only passes on the ones it doesn't. E.g. class A: def __init__(self, alpha, **kwds): print("alpha =", alpha) super().__init__(**kwds) class B(A): def __init__(self, beta, **kwds): print("beta =", beta) super().__init__(**kwds) b = B(alpha = 17, beta = 42) Both A and B here pass on a **kwds argument, but by the time it get to object, there are no arguments left, so it's fine. -- Greg

I realized that bypassing kwargs is probably the least important thing of this idea - so if implemented it definitely had to get a better name. Just look at the following example: class Magic: magic_number = 42 def __init__(self): A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes class Foo(Magic): def __init__(self): print("foo") # Let's say we want magic_number==42 in Foo # So you can’t call super().__init__ here if it is not in the middle of an MRO # Technically you could get it working by going to the first init in the MRO of self after Foo \ # that isn’t in the MRO of Foo but you can see that this gets quite ugly to write (and read!). # And then you would still have the problem of indicating that Foo seems to accepted kwargs as input class Bar: def __init__(self, bar): print("bar:", bar) class FooBar(Foo, Bar): def __init__(self): # There is no easy way right now to avoid writing Foo.__init__() Bar.__init__(bar="bar“) But if Foo adopted this protocol of "automated MRO handling"/"cooperativity" (or however you want to call it) Bar.__init__ could be called automatically. What do I mean by that? Well, basically do automatically what I described in the comment of Foo if no super().__init__ call was registered in any cooperative class. Okay, this example might be a bit far-fetched, but it shows, that you could easily get the MRO working as expected with a simple super().__init__(bar="bar") in FooBar.
What bugs me is that in my example from yesterday ( class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever“ % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) ) if you want to instantiate an Aardvark directly there is NO WAY EVER that you could give him any kwargs. So why should the __init__ indicate something else? Well, just to make the MRO work. All I want is to make it as obvious as possible that an Aardvark ONLY takes `quantity` as input, but is fully "cooperative" with other classes if it is in the middle of the MRO (by which I mean that it will automatically call the __init__ and hand on any kwargs it didn’t expect to a class from a different branch of the class hierarchy).

Everything my idea has to offer really is just reasonable if you don’t have single inheritance only
I would like to correct myself immediately on that one: In the Pizza-example (from yesterday as well) it would be possible to overwrite the default price of the HawaiianPizza with the merging of bypassed kwargs with the ones from the call itself. Let us assume that the exemption of the 10$ price for all HawaiianPizzas would be a Family-sized pizza for 20$: class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) @cooperative class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class FamilyHawaiianPizza(HawaiianPizza): def __init__(self, *, pineapple="chunked"): super().__init__(price=20, size=20, pineapple=pineapple) #This would overwrite the price of the HawaiianPizza But this would not be the real intention of the usage, but just a side effect of correctly handling a simple diamond-like hierarchy. The reason that bypassed kwargs should overwrite them is that they come from up the MRO and so obviously should have priority. Sorry to spam you all, Have a nice Sunday! Michael

Sorry for that. Yes, it should have been Magic (I renamed the class after writing it and didn’t pay attention). I just wanted to override the class-variable `magic_number` to give a reason why I don’t ever want to call Magic.__init__ in Foo. If you want, you can have this class instead: class Magic: def __init__(self): raise RuntimeError("Do not initialize this class") but I figured that this might look a bit artificial...

I'd say NOT wanting to call an __init__ method of a superclass is a rather uncommon occurence. It's generally a huge error. So I think it's worth not accomodating that. 2018-05-28 9:27 GMT+02:00 Michael Lohmann <mial.lohmann@gmail.com>:

I will give you an example then where I am absolutely fine with calling super().__init__ in all classes and describe why I am not really satisfied with the current status. (It is from my email from yesterday 17:19 GMT):

2018-05-28 9:44 GMT+02:00 Michael Lohmann <mial.lohmann@gmail.com>:
This is more an issue with your Ethel class. In *normal* subclassing, it's init would look like: class Ethel(Aardvark, Clever): def __init__(self, **kwargs): super().__init__(**kwargs) and you'd instantiate it like: e = Ethel(quantity="some spam", cleverness=1000) or even (because keyword argument order doesn't matter): e = Ethel(cleverness=1000, quantity="some spam") Because don't forget: assert isinstance(e, Ethel) assert isinstance(e, Aardvark) assert isinstance(e, Clever) will all pass perfectly fine. It's not just an Ethel or an Aardvark, it's also a Clever. (which doesn't sound like it makes sense...perhaps clever should be an attribute on an Aardvark/Ethel instead ?). That all aside, for a moment. You actually CAN call object.__init__(**kwargs) no problem - as long as kwargs is empty. I'd have written your classes like this: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don’t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1, **kwargs): print("You are %d clever" % cleverness) super().__init__(**kwargs) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self, **kwargs): super().__init__(**kwargs) Just try it - it works perfectly fine. Each constructor will consume the keywords meant for it, therefore, the last super() call will call the object constructor without keyword arguments. **kwargs is the price we have to pay for good multiple inheritance - and IMO, it's a low one. If we're talking about signalling what the arguments mean, just ensure you have a good docstring. class Aardvark: def __init__(self, quantity, **kwargs): """ Aardvarks are gentle creatures, and therefore cooperate with multiple inheritance. :param quantity: The quantity of Aardvark(s). """

Michael Lohmann wrote:
You seem to want the signature of Ethel.__init__ to indicate exactly what arguments it can accept. But even with your proposed feature, that isn't going to happen. It will only show arguments that appear explicitly in Ethel.__init__'s argument list. If it didn't override the default value for cleverness, you would never know that it was a possible parameter. In situations like this there's no substitute for hand-written documentation, and that can include explaining what is allowed for **kwds. -- Greg

Michael Lohmann wrote:
But your original example looks just as artificial. Skipping the initialisation of a class you're inheriting from is an extremely weird thing to do, and in any real-life situation there's almost certainly a better design. In any case, I don't see how this has anything to do with invisible passing of **kwds. In short, I don't understand what you're saying at all. -- Greg

On Sat, May 26, 2018 at 09:39:14AM +0200, Michael Lohmann wrote:
What makes you say that? Since the kwargs are passed on to the super().__init__ call, the eggs argument is passed onto the super class of Aardvark. What makes you so certain that Aardvark's parent (or grandparent, or great-grandparent...) doesn't take a parameter "eggs"? Considering that *I* made this example up, it is a bit rich for you to tell me that "eggs" isn't used. Of course it is used, by one of the superclasses. That's why it was provided.
(or at some point **kwargs are being accepted and not passed on to super which would be terrible on its own).
No, it is a standard technique to collect keyword arguments, process those your class cares about, and pass the rest on to super(). Eventually the class second from the top (the class which inherits directly from object) MUST NOT pass those arguments to super, since object doesn't accept any arguments: you can't keep passing arguments up the MRO all the way to the top. At one point or another, each argument must be used and discarded.
If your class doesn't accept **kwargs, then don't put **kwargs in the parameter list.
Shouldn’t the init tell you something like 'If you instantiate this class then this is everything you can give me'?
The way to do that is to write the __init__ of the class that only takes the parameters you want.
Well, right now in addition it says 'Just give me anything with keywords‘.
Only if you intentionally add **kwargs to the parameter list. Why are you putting **kwargs in the parameter list if you don't want to accept them?
"Somebody"? Who? It's my class. If "somebody" used kwargs.pop('eggs') it must have been me. If I did it, it means that my class wants to consume the eggs parameter and NOT pass it on.
I don't understand this comment one bit.
The way we do that is by explicitly collecting **kwargs and explicitly passing it on to the super call. I don't think I understand the scenario you have in mind. Here is what I have in mind: class Foo: def __init__(self, spam): self.spam = spam # this is a null-op, so it could be left out super().__init__() # calls object.__init__ which does nothing class Bar(Foo): def __init__(self, eggs, **kwargs): self.eggs = eggs super().__init__(**kwargs) class Baz(Bar): def __init__(self, cheese, **kwargs): self.cheese = cheese super().__init__(**kwargs) class Aardvark(Baz): def __init__(self, myarg, **kwargs): self.myarg = myarg super().__init__(**kwargs) obj = Aardvark(27, spam=3, eggs=5, cheese=True) What situation do you have in mind? -- Steve
participants (6)
-
Brendan Barnwell
-
Chris Angelico
-
Greg Ewing
-
Jacco van Dorp
-
Michael Lohmann
-
Steven D'Aprano