[Python-Dev] Breaking calls to object.__init__/__new__

[Thomas Wouters]

On 3/22/07, Adam Olsen <rhamph at gmail.com> wrote:
>
> On 3/21/07, Guido van Rossum <guido at python.org> wrote:
> > On 3/21/07, Adam Olsen <rhamph at gmail.com> wrote:
> > > super() has always felt strange to me.
> >
> > When used in __init__? Or in general? If the former, that's because
> > it's a unique Python wart to even be able to use super for __init__.
>
> In general.  Too many things could fail without errors, so it wasn't
> obvious how to use it correctly.  None of the articles I've read
> helped either.


I've been thinking about writing an article that explains how to use
super(), so let's start here :) This is a long post that I'll probably
eventually copy-paste-and-edit into an article of some sort, when I get the
time. Please do comment, except with 'MI is insane' -- I already know that.
Nevertheless, I think MI has its uses.

super() is actually extremely straight-forward (except for the call syntax)
and the only sane way (that I can think of) of doing co-operative multiple
inheritance. Note 'co-operative'. It's not the same thing as using MI for
'mixins'. Co-operative MI isn't always (or often) useful, but it's the only
way to make complexer-than-mixins MI not make mush of your brain when users
do something unexpected. If you can do with composition or refactoring to
make the desire for MI go away, that is probably a better solution. I'm not
advocating MI over simpler solutions, but I am advocating *correct* MI over
*incorrect* MI  :) (And if Python 3.0 is indeed going to have abstract
baseclasses, MI is going to be a lot more common, so getting it right is
getting more and more important.)

Bottom line is, for any given method (with as specific signature and
semantics), you need a baseclass that implements that method but does not
call its superclass. (It can be an empty implementation, if you want, as
long as it at least has it and does not raise an exception. I believe
object.__init__'s odd behaviour was a (possibly unconcious) attempt to make
it such a baseclass for __init__.) Any class that wants to provide
implementation of that method (in a co-operative manner) has at least derive
from that baseclass. No class that does not derive from that baseclass must
implement the method(s) (at least, not if it is to be part of the MI
hierarchy.) Each method should use super() to call the baseclass version.

In order to change the signature of a method in part of the MI tree, you
need to extract the changed signature in its own hierarchy: provide a new
baseclass (which inherits from the original baseclass) with the changed
signature, but rather than not call the baseclass method, it calls the
baseclass method with the original signature. All classes that implement the
changed signature should then derive from that second baseclass. Classes
multiply-inheriting from two classes where one of the two's methods are
changed with respect to the common baseclass, should put the most-specific
class first (this is generally good advice when doing multiple inheritance
:)

Mixing baseclasses with different signatures but similar semantics is not
much easier this way, although IMHO the semantics are less surprising. (This
goes for mixing baseclasses that change the signature of *different
methods*, too: both classes are 'more specific' than the other class.) Say
you have:

  class A:
    def spam(self, msg):
      print msg

  class B(A):
    def spam(self, msg, times):
      for i in range(times):
        super(B, self).spam(msg)

  class C(A):
    def spam(self, msg, finish):
      super(C, self).spam(msg)
      print finish

There is no way straightforward way to combine them into a class D(B, C).
You could make each subclass take arbitrary keyword(-only) arguments and
'consume' them (as suggested in this thread) only in the
baseclasses-for-that-signature:

  class B(A):
    def spam(self, msg, times, **kwargs):
      for i in range(times):
        super(B, self).spam(msg, **kwargs)

  class C(A):
    def spam(self, msg, finish, **kwargs):
      super(C, self).spam(msg, **kwargs)
      print finish

... but that means adding **kwargs to *all* changed-in-derived-class methods
that *might* want to co-operate with changed-differently methods. Notice,
though, that the baseclass does *not* take arbitrary keywords (they should
all have been consumed by the derived-baseclasses), so you still get
checking for unused/misspelled keyword arguments. It happens in a slightly
confusing place (the basemostest class for a method) but the message is just
as clear.

Another way to do it is to insert a new class between B and C, which
implements the baseclass signature (not B's) but calls its superclass with
C's signature. It has to either invent the extra argument to C's signature
on the spot, or fetch it from someplace else, though:

  # 'helper' class to insert inbetween B and C
  class BtoC(C):
    def spam(self, msg):
      super(BtoC, self).spam(msg, self.finish)

  # New signature-baseclass for B and C combined
  class BandC(B, BtoC, C):
    def spam(self, msg, times, finish):
      # This obviously doesn't do the right thing for nested calls or calls
from multiple threads.
      self.finish = finish
      super(BandC, self).spam(msg, times)

However, the main problem with this solution is that subclasses of BandC no
longer satisfy the Liskov substitution principle with respect to positional
arguments to 'spam'. So, here, too, you would have to make them keyword-only
arguments (although since that would be a method signature change, you could
do it in a subclass of B and a subclass of C; you wouldn't have Liskov
substitutability from BandC to B or C, but you would have it from BandC to
KeywordOnlyB or KeywordOnlyC.)

Then there's the case where you want a derived class's method to implement
the functionality from a baseclass in a different way, and *not* call the
baseclass method (so not entirely co-operative). That gets even trickier if
you do want others to co-operate with your class, and do want co-operation
with other methods. The simple solution is to put your overruling class at
the end of the baseclass list, and only have it inherit from the basemost
class. The problem is that anyone multiply-inheriting from your overruling
class must put it at the end of its baseclass list (or the other classes'
methods may get silently ignored.) So if you want to be able to mix well
with other classes (in arbitrary order), you have to extract the
overruling-methods into a separate baseclass with *only* the overruling
bits, and make sure that one is always last in the inheritance tree right
before the baseclass (or at least not before any methods you want to
co-operate with.) At that point (and possibly way before) your inheritance
tree is getting so complicated, you should probably give up and use
composition, or refactor code so you don't have so many conflicts.

However, the main point of all of this is simple: the singly-inheriting
classes inbetween all these wacky glue-jobs *don't care* about any of this.
They only care about their direct baseclass method signature(s), and about
using super() to call them. You only have to get wacky if you want to do
wacky things. Multiply-inheriting classes that don't want to change the
signature of any methods don't need to care all that much either: all they
need to check is whether all direct baseclasses have the same signatures.
And use super() to call them, of course.

Most of the sanity-checks one would want in such a co-operative MI tree (do
signatures match, is there a single baseclass for a changed signature, if
there is an 'overriding class' is it in the right place in the inheritance
tree, etc) can pretty easily be done in a metaclass, possibly with the help
of some decorators. I've been meaning to write such a metaclass, but haven't
had the time. Likewise, I've been meaning to write all this down in an
article. Comments, corrections, questions and blatant accusations of
perversion of the fragile MI-unaware programmer's mind all welcome.

Omg-look-at-the-long-post-I-feel-like-pje-now'ly y'rs,
-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20070322/6d107e5a/attachment.htm