[Python-Dev] Should instances really be able to dictate the "existence" of special methods?

[Eric Snow]

On Mon, Apr 20, 2015 at 2:20 AM, Guido van Rossum <guido at python.org> wrote:
> (I suppose this new thread is a result of some research you did regarding
> the thread complaining about callable()?)

Yep. :)

> On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow <ericsnowcurrently at gmail.com>
> wrote:
>>
>> _PyObject_LookupSpecial is used in place of obj.__getattribute__ for
>> looking up special methods.  (As far as I recall it is not exposed in
>> the stdlib, e.g. inspect.getattr_special.)  Correct me if I'm wrong
>> (please!), but there are two key reasons:
>>
>>  * access to special methods in spite of obj.__getattribute__
>>  * speed
>
> Good question! I don't have an easy pointer to the original discussion, but
> I do recall that this was introduced in response to some issues with the
> original behavior, which looked up dunder methods on the instance and relied
> on the general mechanism for binding it to the instance. I don't think the
> reason was to circumvent __getattribute__, but your second bullet rings
> true: for every +, -, * etc. there would be a (usually failing) lookup in
> the instance dict before searching the class dict and then the base classes
> etc. There may also have been some confusion where people would e.g. assign
> a function of two arguments to x.__add__ and would be disappointed to find
> out it was called with only one argument. I think there were some folks who
> wanted to fix this by somehow "binding" such calls to the instance (since
> there's no easy way otherwise to get the first argument) but I thought the
> use case was sufficiently odd that it was better to avoid it altogether.
>
> In any case, it's not just an optimization -- it's an intentional (though
> obscure) feature.

Thanks for explaining.

>> While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or
>> call obj.__getattr__, it does resolve descriptors.  This is important
>> particularly since special methods will nearly always be some kind of
>> descriptor.  However, one consequence of this is that instances can
>> influence whether or not some capability, as relates to the special
>> method, is available.  This is accomplished by the descriptor's
>> __get__ raising AttributeError.
>
> Well, it's not really the instance that raises AttributeError -- it's the
> descriptor, which is a separate class (usually but not always a builtin
> class, such as property or classmethod). And the descriptor is "owned" by
> the class.

Sure.  That's what I meant. :)  The instance can influence what the
descriptor returns.

>> My question is: was this intentional?  Considering the obscure bugs
>> that can result (e.g. where did the AttributeError come from?), it
>> seems more likely that it is an oversight of an obscure corner case.
>
> I'm not sure what you would do to avoid this. You can't very well declare
> that a descriptor's __get__ method must not raise AttributeError. It could
> be implemented in Python and it could just hit a bug or something.

Right.  And such a bug will be misinterpreted and obscured and hard to
unravel.  I ran into this a while back with pickle (which still does
lookup for special methods on the instance).  Ultimately it's the same
old problem of not knowing how to interpret an exception that may have
bubbled up from some other layer.

Like I said, I don't think there's anything to be done about it either
way.  I just got the feeling that in the case of special methods, the
descriptor part of lookup should not expect AttributeError to come out
of the getter.  So I wanted to see if my intuition was correct even if
the point is essentially irrelevant. :)  At this point, though, I
think my intuition wasn't quite right, though I still don't think a
descriptor's getter is the right place to raise AttributeError.

> But
> perhaps I'm misunderstanding the situation you're describing?
>
>>
>> If that is the case then it would be nice if we could fix
>> _PyObject_LookupSpecial to chain any AttributeError coming from
>> descr.__get__ into a RuntimeError.  However, I doubt we could get away
>> with that at this point.
>
> Yeah, I think that ship has sailed. It also seems to be hardly worth trying
> to control "double fault" situations like this. (It's not really a double
> fault, but it reeks like it.)
>
> I wonder if maybe you're feeling inspired by PEP 479? But that's really a
> much more special case, and I don't really want to start down a whole
> cascade of trying to "fix" all cases where an AttributeError could be raised
> due to a problem in the user's lookup code.

Nah.  It isn't about fixing all the cases nor directly related to PEP
479.  Instead it is in response to one obscure corner case (the
behavior of callable).

>> Also, while it may be appropriate in general to allow instances to
>> dictate the availability of attributes/methods (e.g. through
>> __getattribute__, __getattr__, or descriptors), I'm not convinced it
>> makes sense for special methods.  We are already explicitly
>> disconnecting special methods from instances in
>> _PyObject_LookupSpecial (except in the case of descriptors...).
>
> I'm still a little bit confused why you consider an error from the
> descriptor as "dictated by the instance".

Sorry, I should have been more clear.  The descriptor part of attr
lookup involves a way for the instance to influence the outcome of the
lookup (at the discretion of the descriptor).  I didn't mean to imply
that the instance has a direct role in special method lookup.

> I think what you're trying to
> describe is that there is a method on the class but trying to bind it to the
> instance fails. Well, all sorts of things may fails. (In fact very few
> things cannot raise an exception in Python.)
>
>>
>> -eric
>>
>> p.s. I also find it a bit strange that instances have any say at all
>> in which methods (i.e. behavior) are *available*.  Certainly instances
>> influence behavior, but I always find their impact on method
>> availability to be surprising.  Conceptually for me instances are all
>> about state and classes about behavior (driven by state).  However, it
>> is very rarely that I run into code that takes advantage of the
>> opportunity. :)
>
>
> If I understand what you're trying to say, what you're describing is due to
> Python's unification of instance variables and methods into attributes. It's
> pretty powerful that if x.foo(args) is a method call, you can also write
> this as (x.foo)(args), and you can separate the attribute access even
> further from the call and pass x.foo to some other function that is
> eventually going to call it. Languages that don't do this have to use a
> lambda at that point. Like every design choice each choice has its pluses
> and minuses, but this is how it's done in Python, and it's not going to
> change.

Again I wasn't very clear.  Rather than the unification of attributes,
I'm referring to how descriptors can raise AttributeError in __get__
and  it gets interpreted as "attribute missing" in
object.__getattribute__ (and similar lookup methods).  However, just
in the context of descriptors it makes a little more sense, even if
still consider it too clever.  I think I was just focusing too much on
the influence of instances on descriptors. :)

-eric