[Python-3000] Adaptation vs. Generic Functions

[Guido van Rossum]

On 4/12/06, Tim Hochberg <tim.hochberg at ieee.org> wrote:
>      What would happen if 'a+b' was just syntactic sugar for
>      'operator.add(a,b)', where operator.add was a generic
>      function, instead of the current magic dance involving
>      __add__ and __radd__.

Funny, exactly the same thought occured to me while flying across the
Atlantic. (But actually I believe I've heard someone else mention it
in this thread.)

Around the same time I also had this scary thought:

What if, instead of adding registries mapping types to functions left
and right (in pickle, copy, pprint, who knows where else), we had a
convention of adding __foo_bar__ methods directly to the class dict?
This would require giving up the effective immutability of built-in
types, which has widespread repercussions (mostly in the area of
multiple interpreters) but nevertheless if this was officially
endorsed, we could clean up a lot of type registries... (I notice that
I haven't really found any use for *multiple* dispatch in my few
explorations of @overloaded; that is, until the operator.add idea came
along.)

> Neglecting the fact that it would break all Python code now in
> existence, would this be a good thing, a bad thing or can we even tell?

We could make it break less code if it fell back on the old way of
doing things (looking for __add__ and __radd__). I'm not convinced
that that is a particularly *bad* convention; it's just not *enough*
in case you want to make an X and a Y addable where you control
neither X nor Y.

One problem (which isn't limited to this use case) is that there's no
guarantee that you're the only personl who came up with the idea of
making an X addable to a Y, but you may have a different
implementation in mind than someone else. If there are conflicting
registrations, what happens? (My current implementation just lets the
last one to register win, but that's not very user-friendly.)

What if the desired effect depends on who caused the call to be made?
I'm not saying "who made the call" because if you control the call
site you don't need operator overloading; you can code whatever add()
function you like. But if the + call itself is inside the package
defining X or Y, and the other argument happens to be obtained
indirectly, the ability to define what X()+Y() means without modifying
X or Y is the most desirable. But this is also when conflicting
definitions are most likely to happen. So maybe a better strategy
would be to wrap your Y instance in a wrapper that knows how to add
itself to an X? That way someone else with a different need to add an
X to a Y can provide their own wrapper and both will live happily
independent from each other, at least until the two needs meet.

> To get the current effect of __add__ and  __radd__, class definitions
> would look something like:
>
>      class Weeble:
>          #....
>      @operator.add.register(Weeble, object)
>      def add(a, b): #...
>      @operator.add.register(object, Weeble)
>      def radd(b, a): #...
>
> That's not too different from today, although it is a little weird that
> add and radd are outside of Weeble.

It doesn't look attractive as long as you control Weeble (hence my
proposal not to drop the old way but to grandfather it in as a last
resort).

> One would also need to need to be
> more flexible about strict dominance, allowing some sorts of ties and
> trying the tied functions one at a time until one worked. (Hmmm...
> perhaps this is the kind of thing Guido was referring to in his blog. If
> so, it bounced off my thick head ;-)

I kind of doubt it; I don't know exactly which passage you're
referring to but I've been kind of skeptical of the whole "next
method" concept in general here because of the possibility of
conflicts at all. But yes, the conventional way of treating __add__
etc. has a different need for resolving conflicts than Phillip Eby's
idea of requiring strict dominance of the selected solution over all
other candidates. So now that you mention it I wonder if the whole
strict dominance requirement isn't a red herring? And perhaps we
should in general use "returns NotImplemented" as a signal to try the
next best candidate, strict dominance be damned... (I guess we're
entering a different application domain than where Phillip first
developed his ideas.)

> I suppose one could add some metaclass magic to type so that __add__ and
> __radd__ were picked up and registered as above. That would make things
> look superficially the same as now and would be more or less backwards
> compatible.

Or do this in the default method.

> On the plus side, there's a lot more flexibility. You could teach two
> types that don't know anything about each other how to play nice
> together simply by registering a couple more adapters to operator.add.

Right. That's the clear advantage.

> On the minus side, there's perhaps too much flexibility. Any function
> call or module importation or even attribute access could suddenly
> change the behaviour of an unrelated type.

Well, all those operations could have undesirable side effects anyway;
this sounds like an overly broad fear of side effects. I worry more
about the specific issue of two different libraries trying to add the
same overloading with different semantics.

>  >>> a = 1
>  >>> b = 2
>  >>> # this calls operater.add.register(int,int)(lambda a,b:str(a+b))
>  >>> c = frobulate()
>  >>> a + b
> "3"
>
> We're all consenting adults here, so should I even worry about this. I
> don't know.

Not about this particular issue. frobulate() could use sys._getframe()
and assign new values to a and b.

> I suspect this falls into the "wild ideas' category, but it's an
> interesting thought experiment anyway. Perhaps with relevance to other
> uses of generic functions.

Definitely wild. Reader beware. Now is the time to generate lots of
wild ideas and let them sink in. If it still seems a good idea 3
months from now we may select it for a code experiment (as opposed to
a thought experiment). BTW I want to encourage lots of code
experiments for Python 3.0 -- though not directly in the p3yk (sic)
branch.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)