Multiple inheritance from builtin (C) types [still] supported in Python3?

Hello, It's more or less known fact (ref: google) that you can't inherit from multiple generic builtin (as in "coded in C") types: class C(dict, list): pass TypeError: multiple bases have instance lay-out conflict However, more detailed googling led me to this page of a book: http://books.google.com.ua/books?id=JnR9hQA3SncC&pg=PA104&lpg=PA104 , which suggest that there might be adhoc support for some native types to serve as multiple bases together, but it doesn't provide an example. The book is apparently focused on Python2. I tried to look at typeobject.c:best_base(), which throws the quoted error above, and the only special rules I could quickly decipher from it is that it's possible to multi-inherit from a class and its subclass. Intuitively, it can be understood - it makes no sense to do that on the same inheritance level, but can happen with recursive inheritance, and then there's no need for conflict - both classes can be "merged" into subclass. Indeed, following works: import _collections class Foo(_collections.defaultdict, dict): pass (opposite order gives MRO conflict) So, is that it, or disjoint native types are supported as bases somehow? Also, would someone know if a class-subclass case happens for example in stdlib? As the previous question, https://mail.python.org/pipermail/python-dev/2014-April/134342.html this one is to help set adequate requirements for implementing multiple inheritance in MicroPython. Thanks, Paul mailto:pmiscml@gmail.com

Antoine's example works because list inherits from object. The more general rule "compatible layout" only allows the rarest of cases to work -- basically the two classes involved must have a common base class and one of the classes must not add any C-level fields to that base class's layout. I would never count on this except in cases where this possibility is documented (e.g. when one class is designed to be a mix-in for the other). On Mon, Apr 28, 2014 at 11:24 AM, Antoine Pitrou <solipsis@pitrou.net>wrote:
-- --Guido van Rossum (python.org/~guido)

Hello, On Mon, 28 Apr 2014 20:24:58 +0200 Antoine Pitrou <solipsis@pitrou.net> wrote:
Well, it's easy to treat "object" class as a special-case, "null" class. So, let's re-formulate questions above with "where such native base classes are not 'object'".
Basically, if two classes have compatible layouts, you can inherit from both at once.
How is "compatible layout" defined? Or "layout" for that matter at all?
Regards
Antoine.
-- Best regards, Paul mailto:pmiscml@gmail.com

Hi Paul, On Mon, 28 Apr 2014 21:42:02 +0300 Paul Sokolovsky <pmiscml@gmail.com> wrote:
See Guido's answer. I don't think it's documented anywhere, but you can find the relevant code somewhere in Objects/typeobject.c (it's quite a mouthful, though :-)). (IIRC, "layout" is determined by tp_basicsize, tp_itemsize, the number of __slots__, and other things perhaps) Regards Antoine.

On Mon, Apr 28, 2014 at 12:02 PM, Antoine Pitrou <solipsis@pitrou.net>wrote:
IIRC the actual inheritance pattern also goes into it. Two structs that each add an identical new field to a common base class's struct should *not* be considered compatible. -- --Guido van Rossum (python.org/~guido)

On Mon, Apr 28, 2014 at 11:42 AM, Paul Sokolovsky <pmiscml@gmail.com> wrote: Well, it's easy to treat "object" class as a special-case, "null"
class.
But the implementation doesn't, at least not for the question you're asking.
The layout is what the C struct defining the object looks like. These are typically defined in headers in the Include directory (e.g. listobject.h). The definition of compatible layout is defined by the C code that gives the error message when it's incompatible. Sorry, I don't recall exactly where that is, so I recommend that you just look at CPython's source tree. (I know you have a personal desire not to look at CPython, but I can't help you without referring to it anyway.) -- --Guido van Rossum (python.org/~guido)

Hello, On Mon, 28 Apr 2014 12:08:40 -0700 Guido van Rossum <guido@python.org> wrote: []
Well, sure I did, as I mentioned, but as that's first time I see that code (that specific piece is in typeobject.c:extra_ivars()), it would take quite some time be sure I understand all aspects of it. Thanks for confirming that it's governed essentially by CPython implementation details and not some language-level semantics like metaclasses (I mentioned them because error message in Python2 did so, though Python3 doesn't refer to metaclasses). An example would really help me to get a feel of the issue, but I assume lack of them means that there's no well-known idiom where such inheritance is used, and that's good enough on its own. I also tried to figure how it's important to support such multi-base cases, so the code I write didn't require complete rewrite if it hits one day, but everything seems to turn out to be pretty extensible.
-- --Guido van Rossum (python.org/~guido)
Thanks! -- Best regards, Paul mailto:pmiscml@gmail.com

On Mon, Apr 28, 2014 at 7:26 PM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
track of the offsets. (The slot descriptors do.) But I don't think there's anything in principle that requires this, it's just the implementation. You could in theory relocate __slots__ defined from Python code in order to make a merged subclass. It's just that the effective "__slots__" of C code can't be moved, because C code is expecting to find them at specific offsets. Therefore, if two types define their own struct fields, they can't be inherited from unless one is a subtype of the other. In the C code (again if I recall correctly), this is done using the __base__ attribute of the type, which indicates what struct layout the object will use. A type can have a larger struct than its base type, adding its own fields after the base type's struct fields. (The dict and weakref fields are added -- if they are added -- *after* the base struct fields. If your __base__ already has them, those offsets within the existing layout are used, which is why them being in another base class's __slots__ isn't a problem.) When you create a new type, CPython looks at your bases to find a suitable __base__. If two of your bases inherit from each other, the ancestor can be ignored, keeping the more-derived one as a candidate __base__. If a base adds only __dict__ and/or __weakref__ (or neither) to its __base__, then its __base__ is a candidate (not recursively, though). If at the end there is more than one base left standing, then it's an error, since you have bases with incompatible layouts. That is not a precise description of the algorithm, but that's the gist of how it works. __base__ is a slot on the type object and is tracked at the C level in order to sort out layouts like this.

Hello, On Tue, 29 Apr 2014 10:47:26 -0400 PJ Eby <pje@telecommunity.com> wrote: []
Ok, so one can gather from this that __slot__'ed class can't be used as multiple bases, as in this example: class B1: __slots__ = ('a', 'b') class B2: __slots__ = ('a', 'b', 'c') class C(B2, B1): pass
Well, here it itches to ask if C++-like offsetting of subclass to base class "this" pointer was considered, but I already feel like a child asking all those stupid questions. One obvious answer would be that there's little point to complicate the matter, as it's just emulation of inheritance via enclosing, and explicit enclosing offers more flexibility anyway (but then it's less efficient). []
Thanks much for the detailed description, will serve as a good reference when in MicroPython we'll hit actual case when CPython accepts some inheritance patterns and uPy doesn't. -- Best regards, Paul mailto:pmiscml@gmail.com

When I redesigned and reimplemented this part of Python inheritance (somewhere in the 2.1 - 2.3 time frame, I forget the exact timing) I was well aware of the C++ approach and decided against it, preferring an approach requiring less compiler assistance that was easier for C programmers to use and understand. If as a Python programmer you want a more general multiple inheritance, you just shouldn't use slots. As a C programmer writing extension modules, the single-inheritance model (which this effectively is, at that level) is much easier to deal with. On Tue, Apr 29, 2014 at 6:03 PM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

On 29 April 2014 21:38, Guido van Rossum <guido@python.org> wrote:
Also related to the "single inheritance is easier to deal with when writing C extensions" aspect, C extension code is also far more likely than Python code to call the base class method implementations directly - getting hold of super() objects from C to do cooperative multiple inheritance isn't straightforward (I actually don't know how to do it myself - I'd have to go trawling through the C API docs to figure it out). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Sokolovsky wrote:
Well, here it itches to ask if C++-like offsetting of subclass to base class "this" pointer was considered,
I suppose in theory it would be possible to build a new set of __slot__ descriptors for the subclass. It mightn't even be all that difficult. My guess would be that it wasn't considered worth the effort, if it was considered at all. -- Greg

It would be a disaster if the base class's slot descriptors would be broken by that though, so the implementation of slot descriptors would have to become more complicated. (It's worth understanding how __slots__ works. the interpreter first finds the slot on the class and then calls its __get__ method to retrieve the actual value from the instance. Each slot descriptor, in the current implementation, knows the offset of its slot within the instance. On Tue, Apr 29, 2014 at 11:48 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
-- --Guido van Rossum (python.org/~guido)

Antoine's example works because list inherits from object. The more general rule "compatible layout" only allows the rarest of cases to work -- basically the two classes involved must have a common base class and one of the classes must not add any C-level fields to that base class's layout. I would never count on this except in cases where this possibility is documented (e.g. when one class is designed to be a mix-in for the other). On Mon, Apr 28, 2014 at 11:24 AM, Antoine Pitrou <solipsis@pitrou.net>wrote:
-- --Guido van Rossum (python.org/~guido)

Hello, On Mon, 28 Apr 2014 20:24:58 +0200 Antoine Pitrou <solipsis@pitrou.net> wrote:
Well, it's easy to treat "object" class as a special-case, "null" class. So, let's re-formulate questions above with "where such native base classes are not 'object'".
Basically, if two classes have compatible layouts, you can inherit from both at once.
How is "compatible layout" defined? Or "layout" for that matter at all?
Regards
Antoine.
-- Best regards, Paul mailto:pmiscml@gmail.com

Hi Paul, On Mon, 28 Apr 2014 21:42:02 +0300 Paul Sokolovsky <pmiscml@gmail.com> wrote:
See Guido's answer. I don't think it's documented anywhere, but you can find the relevant code somewhere in Objects/typeobject.c (it's quite a mouthful, though :-)). (IIRC, "layout" is determined by tp_basicsize, tp_itemsize, the number of __slots__, and other things perhaps) Regards Antoine.

On Mon, Apr 28, 2014 at 12:02 PM, Antoine Pitrou <solipsis@pitrou.net>wrote:
IIRC the actual inheritance pattern also goes into it. Two structs that each add an identical new field to a common base class's struct should *not* be considered compatible. -- --Guido van Rossum (python.org/~guido)

On Mon, Apr 28, 2014 at 11:42 AM, Paul Sokolovsky <pmiscml@gmail.com> wrote: Well, it's easy to treat "object" class as a special-case, "null"
class.
But the implementation doesn't, at least not for the question you're asking.
The layout is what the C struct defining the object looks like. These are typically defined in headers in the Include directory (e.g. listobject.h). The definition of compatible layout is defined by the C code that gives the error message when it's incompatible. Sorry, I don't recall exactly where that is, so I recommend that you just look at CPython's source tree. (I know you have a personal desire not to look at CPython, but I can't help you without referring to it anyway.) -- --Guido van Rossum (python.org/~guido)

Hello, On Mon, 28 Apr 2014 12:08:40 -0700 Guido van Rossum <guido@python.org> wrote: []
Well, sure I did, as I mentioned, but as that's first time I see that code (that specific piece is in typeobject.c:extra_ivars()), it would take quite some time be sure I understand all aspects of it. Thanks for confirming that it's governed essentially by CPython implementation details and not some language-level semantics like metaclasses (I mentioned them because error message in Python2 did so, though Python3 doesn't refer to metaclasses). An example would really help me to get a feel of the issue, but I assume lack of them means that there's no well-known idiom where such inheritance is used, and that's good enough on its own. I also tried to figure how it's important to support such multi-base cases, so the code I write didn't require complete rewrite if it hits one day, but everything seems to turn out to be pretty extensible.
-- --Guido van Rossum (python.org/~guido)
Thanks! -- Best regards, Paul mailto:pmiscml@gmail.com

On Mon, Apr 28, 2014 at 7:26 PM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
track of the offsets. (The slot descriptors do.) But I don't think there's anything in principle that requires this, it's just the implementation. You could in theory relocate __slots__ defined from Python code in order to make a merged subclass. It's just that the effective "__slots__" of C code can't be moved, because C code is expecting to find them at specific offsets. Therefore, if two types define their own struct fields, they can't be inherited from unless one is a subtype of the other. In the C code (again if I recall correctly), this is done using the __base__ attribute of the type, which indicates what struct layout the object will use. A type can have a larger struct than its base type, adding its own fields after the base type's struct fields. (The dict and weakref fields are added -- if they are added -- *after* the base struct fields. If your __base__ already has them, those offsets within the existing layout are used, which is why them being in another base class's __slots__ isn't a problem.) When you create a new type, CPython looks at your bases to find a suitable __base__. If two of your bases inherit from each other, the ancestor can be ignored, keeping the more-derived one as a candidate __base__. If a base adds only __dict__ and/or __weakref__ (or neither) to its __base__, then its __base__ is a candidate (not recursively, though). If at the end there is more than one base left standing, then it's an error, since you have bases with incompatible layouts. That is not a precise description of the algorithm, but that's the gist of how it works. __base__ is a slot on the type object and is tracked at the C level in order to sort out layouts like this.

Hello, On Tue, 29 Apr 2014 10:47:26 -0400 PJ Eby <pje@telecommunity.com> wrote: []
Ok, so one can gather from this that __slot__'ed class can't be used as multiple bases, as in this example: class B1: __slots__ = ('a', 'b') class B2: __slots__ = ('a', 'b', 'c') class C(B2, B1): pass
Well, here it itches to ask if C++-like offsetting of subclass to base class "this" pointer was considered, but I already feel like a child asking all those stupid questions. One obvious answer would be that there's little point to complicate the matter, as it's just emulation of inheritance via enclosing, and explicit enclosing offers more flexibility anyway (but then it's less efficient). []
Thanks much for the detailed description, will serve as a good reference when in MicroPython we'll hit actual case when CPython accepts some inheritance patterns and uPy doesn't. -- Best regards, Paul mailto:pmiscml@gmail.com

When I redesigned and reimplemented this part of Python inheritance (somewhere in the 2.1 - 2.3 time frame, I forget the exact timing) I was well aware of the C++ approach and decided against it, preferring an approach requiring less compiler assistance that was easier for C programmers to use and understand. If as a Python programmer you want a more general multiple inheritance, you just shouldn't use slots. As a C programmer writing extension modules, the single-inheritance model (which this effectively is, at that level) is much easier to deal with. On Tue, Apr 29, 2014 at 6:03 PM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

On 29 April 2014 21:38, Guido van Rossum <guido@python.org> wrote:
Also related to the "single inheritance is easier to deal with when writing C extensions" aspect, C extension code is also far more likely than Python code to call the base class method implementations directly - getting hold of super() objects from C to do cooperative multiple inheritance isn't straightforward (I actually don't know how to do it myself - I'd have to go trawling through the C API docs to figure it out). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Sokolovsky wrote:
Well, here it itches to ask if C++-like offsetting of subclass to base class "this" pointer was considered,
I suppose in theory it would be possible to build a new set of __slot__ descriptors for the subclass. It mightn't even be all that difficult. My guess would be that it wasn't considered worth the effort, if it was considered at all. -- Greg

It would be a disaster if the base class's slot descriptors would be broken by that though, so the implementation of slot descriptors would have to become more complicated. (It's worth understanding how __slots__ works. the interpreter first finds the slot on the class and then calls its __get__ method to retrieve the actual value from the instance. Each slot descriptor, in the current implementation, knows the offset of its slot within the instance. On Tue, Apr 29, 2014 at 11:48 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
-- --Guido van Rossum (python.org/~guido)
participants (6)
-
Antoine Pitrou
-
Greg Ewing
-
Guido van Rossum
-
Nick Coghlan
-
Paul Sokolovsky
-
PJ Eby