Should instances really be able to dictate the "existence" of special methods?
_PyObject_LookupSpecial is used in place of obj.__getattribute__ for looking up special methods. (As far as I recall it is not exposed in the stdlib, e.g. inspect.getattr_special.) Correct me if I'm wrong (please!), but there are two key reasons: * access to special methods in spite of obj.__getattribute__ * speed While _PyObject_LookupSpecial does not do lookup on obj.__dict__ or call obj.__getattr__, it does resolve descriptors. This is important particularly since special methods will nearly always be some kind of descriptor. However, one consequence of this is that instances can influence whether or not some capability, as relates to the special method, is available. This is accomplished by the descriptor's __get__ raising AttributeError. My question is: was this intentional? Considering the obscure bugs that can result (e.g. where did the AttributeError come from?), it seems more likely that it is an oversight of an obscure corner case. If that is the case then it would be nice if we could fix _PyObject_LookupSpecial to chain any AttributeError coming from descr.__get__ into a RuntimeError. However, I doubt we could get away with that at this point. Also, while it may be appropriate in general to allow instances to dictate the availability of attributes/methods (e.g. through __getattribute__, __getattr__, or descriptors), I'm not convinced it makes sense for special methods. We are already explicitly disconnecting special methods from instances in _PyObject_LookupSpecial (except in the case of descriptors...). -eric p.s. I also find it a bit strange that instances have any say at all in which methods (i.e. behavior) are *available*. Certainly instances influence behavior, but I always find their impact on method availability to be surprising. Conceptually for me instances are all about state and classes about behavior (driven by state). However, it is very rarely that I run into code that takes advantage of the opportunity. :)
(I suppose this new thread is a result of some research you did regarding the thread complaining about callable()?) On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Good question! I don't have an easy pointer to the original discussion, but I do recall that this was introduced in response to some issues with the original behavior, which looked up dunder methods on the instance and relied on the general mechanism for binding it to the instance. I don't think the reason was to circumvent __getattribute__, but your second bullet rings true: for every +, -, * etc. there would be a (usually failing) lookup in the instance dict before searching the class dict and then the base classes etc. There may also have been some confusion where people would e.g. assign a function of two arguments to x.__add__ and would be disappointed to find out it was called with only one argument. I think there were some folks who wanted to fix this by somehow "binding" such calls to the instance (since there's no easy way otherwise to get the first argument) but I thought the use case was sufficiently odd that it was better to avoid it altogether. In any case, it's not just an optimization -- it's an intentional (though obscure) feature.
Well, it's not really the instance that raises AttributeError -- it's the descriptor, which is a separate class (usually but not always a builtin class, such as property or classmethod). And the descriptor is "owned" by the class.
I'm not sure what you would do to avoid this. You can't very well declare that a descriptor's __get__ method must not raise AttributeError. It could be implemented in Python and it could just hit a bug or something. But perhaps I'm misunderstanding the situation you're describing?
Yeah, I think that ship has sailed. It also seems to be hardly worth trying to control "double fault" situations like this. (It's not really a double fault, but it reeks like it.) I wonder if maybe you're feeling inspired by PEP 479? But that's really a much more special case, and I don't really want to start down a whole cascade of trying to "fix" all cases where an AttributeError could be raised due to a problem in the user's lookup code.
I'm still a little bit confused why you consider an error from the descriptor as "dictated by the instance". I think what you're trying to describe is that there is a method on the class but trying to bind it to the instance fails. Well, all sorts of things may fails. (In fact very few things cannot raise an exception in Python.)
If I understand what you're trying to say, what you're describing is due to Python's unification of instance variables and methods into attributes. It's pretty powerful that if x.foo(args) is a method call, you can also write this as (x.foo)(args), and you can separate the attribute access even further from the call and pass x.foo to some other function that is eventually going to call it. Languages that don't do this have to use a lambda at that point. Like every design choice each choice has its pluses and minuses, but this is how it's done in Python, and it's not going to change. -- --Guido van Rossum (python.org/~guido)
On Mon, Apr 20, 2015 at 2:20 AM, Guido van Rossum <guido@python.org> wrote:
(I suppose this new thread is a result of some research you did regarding the thread complaining about callable()?)
Yep. :)
Thanks for explaining.
Sure. That's what I meant. :) The instance can influence what the descriptor returns.
Right. And such a bug will be misinterpreted and obscured and hard to unravel. I ran into this a while back with pickle (which still does lookup for special methods on the instance). Ultimately it's the same old problem of not knowing how to interpret an exception that may have bubbled up from some other layer. Like I said, I don't think there's anything to be done about it either way. I just got the feeling that in the case of special methods, the descriptor part of lookup should not expect AttributeError to come out of the getter. So I wanted to see if my intuition was correct even if the point is essentially irrelevant. :) At this point, though, I think my intuition wasn't quite right, though I still don't think a descriptor's getter is the right place to raise AttributeError.
Nah. It isn't about fixing all the cases nor directly related to PEP 479. Instead it is in response to one obscure corner case (the behavior of callable).
Sorry, I should have been more clear. The descriptor part of attr lookup involves a way for the instance to influence the outcome of the lookup (at the discretion of the descriptor). I didn't mean to imply that the instance has a direct role in special method lookup.
Again I wasn't very clear. Rather than the unification of attributes, I'm referring to how descriptors can raise AttributeError in __get__ and it gets interpreted as "attribute missing" in object.__getattribute__ (and similar lookup methods). However, just in the context of descriptors it makes a little more sense, even if still consider it too clever. I think I was just focusing too much on the influence of instances on descriptors. :) -eric
OK, so I think there isn't anything we can or should do here. Yes, it's possible that type(x).__add__ succeeds but x.__add__ fails. That's how you spell descriptor. :-) You could also use a random number generator in __getattribube__... On Sun, Apr 19, 2015 at 6:36 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
(I suppose this new thread is a result of some research you did regarding the thread complaining about callable()?) On Sun, Apr 19, 2015 at 4:03 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Good question! I don't have an easy pointer to the original discussion, but I do recall that this was introduced in response to some issues with the original behavior, which looked up dunder methods on the instance and relied on the general mechanism for binding it to the instance. I don't think the reason was to circumvent __getattribute__, but your second bullet rings true: for every +, -, * etc. there would be a (usually failing) lookup in the instance dict before searching the class dict and then the base classes etc. There may also have been some confusion where people would e.g. assign a function of two arguments to x.__add__ and would be disappointed to find out it was called with only one argument. I think there were some folks who wanted to fix this by somehow "binding" such calls to the instance (since there's no easy way otherwise to get the first argument) but I thought the use case was sufficiently odd that it was better to avoid it altogether. In any case, it's not just an optimization -- it's an intentional (though obscure) feature.
Well, it's not really the instance that raises AttributeError -- it's the descriptor, which is a separate class (usually but not always a builtin class, such as property or classmethod). And the descriptor is "owned" by the class.
I'm not sure what you would do to avoid this. You can't very well declare that a descriptor's __get__ method must not raise AttributeError. It could be implemented in Python and it could just hit a bug or something. But perhaps I'm misunderstanding the situation you're describing?
Yeah, I think that ship has sailed. It also seems to be hardly worth trying to control "double fault" situations like this. (It's not really a double fault, but it reeks like it.) I wonder if maybe you're feeling inspired by PEP 479? But that's really a much more special case, and I don't really want to start down a whole cascade of trying to "fix" all cases where an AttributeError could be raised due to a problem in the user's lookup code.
I'm still a little bit confused why you consider an error from the descriptor as "dictated by the instance". I think what you're trying to describe is that there is a method on the class but trying to bind it to the instance fails. Well, all sorts of things may fails. (In fact very few things cannot raise an exception in Python.)
If I understand what you're trying to say, what you're describing is due to Python's unification of instance variables and methods into attributes. It's pretty powerful that if x.foo(args) is a method call, you can also write this as (x.foo)(args), and you can separate the attribute access even further from the call and pass x.foo to some other function that is eventually going to call it. Languages that don't do this have to use a lambda at that point. Like every design choice each choice has its pluses and minuses, but this is how it's done in Python, and it's not going to change. -- --Guido van Rossum (python.org/~guido)
On Mon, Apr 20, 2015 at 2:20 AM, Guido van Rossum <guido@python.org> wrote:
(I suppose this new thread is a result of some research you did regarding the thread complaining about callable()?)
Yep. :)
Thanks for explaining.
Sure. That's what I meant. :) The instance can influence what the descriptor returns.
Right. And such a bug will be misinterpreted and obscured and hard to unravel. I ran into this a while back with pickle (which still does lookup for special methods on the instance). Ultimately it's the same old problem of not knowing how to interpret an exception that may have bubbled up from some other layer. Like I said, I don't think there's anything to be done about it either way. I just got the feeling that in the case of special methods, the descriptor part of lookup should not expect AttributeError to come out of the getter. So I wanted to see if my intuition was correct even if the point is essentially irrelevant. :) At this point, though, I think my intuition wasn't quite right, though I still don't think a descriptor's getter is the right place to raise AttributeError.
Nah. It isn't about fixing all the cases nor directly related to PEP 479. Instead it is in response to one obscure corner case (the behavior of callable).
Sorry, I should have been more clear. The descriptor part of attr lookup involves a way for the instance to influence the outcome of the lookup (at the discretion of the descriptor). I didn't mean to imply that the instance has a direct role in special method lookup.
Again I wasn't very clear. Rather than the unification of attributes, I'm referring to how descriptors can raise AttributeError in __get__ and it gets interpreted as "attribute missing" in object.__getattribute__ (and similar lookup methods). However, just in the context of descriptors it makes a little more sense, even if still consider it too clever. I think I was just focusing too much on the influence of instances on descriptors. :) -eric
OK, so I think there isn't anything we can or should do here. Yes, it's possible that type(x).__add__ succeeds but x.__add__ fails. That's how you spell descriptor. :-) You could also use a random number generator in __getattribube__... On Sun, Apr 19, 2015 at 6:36 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
participants (2)
-
Eric Snow -
Guido van Rossum