[Python-ideas] Missing Core Feature: + - * / | & do not call __getattr__

[Andrew Barnert]

This is explained in the documentation (https://docs.python.org/3/reference/datamodel.html#special-method-lookup for 3.x; for 2.x, it's largely the same except that old-style classes exist and have a different rule).

There is also at least one StackOverflow answer that explains this (I know, because I wrote one...), but it's not even on the first page of a search, while the docs answer is the first result that shows up. Plus, the docs are written by the Python dev team in an open community process, while my SO answer is only as good as one user's understanding and writing ability; even if it weren't thrown in with a dozen answers that are wrong or just say "Python does this because it's dumb" or whatever, I'd go to the docs first.


On Friday, December 4, 2015 6:22 AM, Stephan Sahm <Stephan.Sahm at gmx.de> wrote:


>the wrapt link is on my future todo-list, will need some more time than I currently have

The wrapt link has the sample code to do exactly what you want. Pop it off your todo list.

If you have more time in the future, you may want to look at some bridging libraries like PyObjC; once you can understand how to map between Python's and ObjC's different notions of method lookup, you'll understand this well enough to teach a class on it. :) But for now, wrapt is more than enough.


>You both mentioned that the operators might be better tackleable via class-interfaces (__slots__ e.g.?)

Not __slots__. The terminology gets a bit confusing here. What they're referring to is the C-API notion of slots. In particular, every builtin (or C-extension) is defined by a C struct which contains, among other things, members like "np_add", which is a pointer to an adding function. For example, the int type's np_add member points to a function that adds integers to other things. Slightly oversimplified, the np_add slot of every class implemented in Python just points to a function that does a stripped-down lookup for '__add__' instead of the usual __getattribute__ mechanism. (Which means you get no __getattr__, __slots__, @properties from the metaclass, etc.)

So, why are these two things both called "slots"? Well, the point of __slots__ is to give you the space savings, and static collection of members that exist on every instance, that builtin types' instances get by using slots in C structs. So, the intuitive notion of C struct layout rather than dict lookup is central in both cases, just in different ways. (If you really want to, you could think about np_add as being a member of the __slots__ of a special metaclass that all builtin classes use. But that's probably more misleading than helpful, because CPython isn't actually implemented that way.)

>My current usecase is to implement a Mixin abc class which shall combine all the available __iadd__ __ior__ and so on with a copy() function (this one is then the abstractmethod) to produce automatically the respective __add__, __or__ and so on


OK, so if you can't handle __add__ dynamically at method lookup time, how do you deal with that?

Of course you can just write lots of boilerplate, but presumably you're using Python instead of Java for a reason. You could also write Python code that generates the boilerplate (as a module, or as code to exec), but presumably you're using Python instead of Tcl for a reason. If you need methods that are dynamically generated at class creation time, just dynamically generate your methods at class creation time:

    def mathify(cls): 
        for name in ('add', 'sub', 'mul', 'div'): 
            ifunc = getattr(cls, '__i{}__'.format(name)) 
            def wrapper(self, other): 
                self_copy = self.copy() 
                ifunc(self_copy, other) 
                return self_copy 
            setattr(cls, '__{}__'.format(name), wrapper)
        return cls

    @mathify
    class Quaternion:
        def __iadd__(self, other):
            self.x += other.x 

            self.y += other.y 

            self.z += other.z 
            self.w += other.w
            return self
        # etc.

Obviously in real life you'll want to fix up the name, docstring, etc. of wrapper (see functools.wraps for how to do this). And you'll want to use tested code rather than something I wrote in an email. You may also want to use a metaclass rather than a decorator (which can be easily hidden from the end-user, because they just need to inherit a mixin that uses that metaclass). Also, if you only ever need to do this dynamic stuff for exactly one class (your mixin), you may not want to use a decorator _or_ a metaclass; just munge the class up in module-level code right after the class definition. But you get the idea.

If you really need the lookup to be dynamic (but I don't think you do here, in which case you'd just be adding extra complexity and inefficiency for no reason), you just need to write your own protocol for this and dynamically generate the methods that bounce to that protocol. For example:

    def mathify(cls):

        for name in ('add', 'sub', 'mul', 'div'):

            def wrapper(self, other):

                self._get_math_method('{}'.format(name))(other)
            setattr(cls, '__{}__'.format(name), wrapper)
        return cls

    @mathify
    class Quaternion:
        def _get_math_method(self, name):
            def wrapper(self, other):
                # see previous example
            return wrapper

In fact, if you really wanted to, you could even abuse __getattr__. I think that would be more likely to confuse people than to help them, but...

    def mathify(cls): 
        for name in ('add', 'sub', 'mul', 'div'): 
            def wrapper(self, other): 
                self.__getattr__('__{}__'.format(name))(other) 
            setattr(cls, '__{}__'.format(name), wrapper) 
        return cls 

    @mathify 
    class Quaternion: 
        def __getattr__(self, name):
            if name.startswith('__') and name.endswith('__'):
                name = name.strip('_')
                def wrapper(self, other): 
                    # see previous example 
                return wrapper 


Again, don't do this last one. I think the first example (or even the simpler version where you just dynamically generate your mixin's members with simple module-level code) is all you need.