[Python-ideas] Fix that broken callable builtin

[Steven D'Aprano]

On Fri, Apr 17, 2015 at 09:04:56PM -0400, Terry Reedy wrote:

> It works because CPython apparently implements c1 + c2, at least for 
> user class instances, as c1.__add__(c2), instead of the presumed 
> equivalent C.__add__(c1, c2).  c1.__add__ is normally a bound method 
> with c1 and C.__add__ as attributes.  c1.__add__(c2) then calls 
> C.__add__(c1, c2).

I believe that analysis is wrong. In Python 3, and with new-style 
classes in Python 2, c1+c2 is implemented as type(c1).__add__(c1, c2) 
and not c1.__add__(c2). (Actually, it is more complex than that, since 
the + operator also has to consider __radd__, and various other 
complications. But let's ignore those complications.)

We can test this by giving instances an __add__ attribute in the 
instance dict, and see what happens. First I confirm that classic 
classes do implement a+b as a.__add__(b):

py> class Classic:
...     def __add__(self, other):
...             return 23
...
py> classic = Classic()
py> classic + None
23
py> from types import MethodType
py> classic.__add__ = MethodType(lambda self, other: 42, classic)
py> classic + None
42


But the same is not the case with a new-style class:


py> class Newstyle(object):
...     def __add__(self, other):
...             return 23
...
py> newstyle = Newstyle()
py> newstyle + None
23
py> newstyle.__add__ = MethodType(lambda self, other: 42, newstyle)
py> newstyle + None  # ignores the __add__ method on the instance
23
py> newstyle.__add__(None)
42


This demonstrates that when it comes to newstyle classes, dunder methods 
on the instance are not used.

I don't think that using __add__ as a property is intended, but it 
works, and I don't think it is an accident that it works. It is simply a 
consequence of how descriptors work in Python. Here is an example:

class Demo(object):
    @property
    def __add__(self):  # Note that there is only a self argument.
        # Return a function of one argument. If we use a closure, 
        # we can access self.
        return lambda other: [self, other]



py> x = Demo()
py> x + None
[<__main__.Demo object at 0xb7c2f5ec>, None]


Let's break that process down. As I show above, + is implemented as 
type(x).__add__ which returns a property object, a descriptor. The 
descriptor protocol says that the __get__ method will be called. So we 
have:

x + None
-> looks up type(x).__add__
-> which calls __get__
-> and finally calls the result of that

py> type(x).__add__
<property object at 0x935c0a4>
py> type(x).__add__.__get__(x, type(x))
<function <lambda> at 0xb7c29a04>
py> type(x).__add__.__get__(x, type(x))(None)
[<__main__.Demo object at 0xb7c2f5ec>, None]



-- 
Steve