[Python-ideas] The AttributeError/getattr mechanism

Sun Nov 29 11:07:22 EST 2015

On Nov 29, 2015, at 06:31, 王珺 <wjun77 at gmail.com> wrote:
> 
> >  If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing.
> What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__.

Your problem is that it took you hours to hunt down where the AttributeError came from, because it gave useless information instead of useful information. The only reason that's happening is that you're handling any AttributeError by swallowing it and raising a new one with useless information. Just take that out, and your hours of debugging go away.

> > all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
> If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__. 

The difference, as I've already explained, is that your solution breaks backward compatibility, requiring anyone using such code to change it, while my variation leaves working code alone, only requiring new code to be written differently. All else being equal, the latter is obvious less disruptive a change and therefore better.

So, if you think your more-disruptive solution is better, that must mean all else is not equal. But you have to explain how. What's wrong with my version?

> (By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.)

All the places exceptions are used for flow control in Python and its stdlib and pythonic code in general use general exceptions. Occasionally (as with generators raising StopIteration where they shouldn't be), this can lead to specific trouble, and that occasional specific trouble gets fixed. Maybe this is one of those cases. That would be two times in as many decades. That's not a good reason to change the general principle.

> >  And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).
> Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right?

The point is that @property is a high-level, simplified way to write a specific kind of descriptor. It may be perfectly reasonable that properties should never trigger __getattr__; after all, you can always fall back to writing your own descriptor (or even your own property-like descriptor factory--it's only a few lines of code) if you want to. (Again, I'm not sure one way or the other about this.)

> @property (or descriptor) converting

Nobody is suggestion descriptors do anything different here. The code would be explicit code in the implementation of @property (and in the Descriptor HOWTO section that shows how @property works).

> AttributeError to AttributeDynamicError, or __getattr__ converting AttributeMissError to AttributeError. The former has the advantage of keeping __getattr__ triggered by raising AttributeError in __getattribute__ as documented for decades. But I don't like this idea, because it's conceptually ugly. If you don't agree or understand, my reply can only be 'we have different aesthetic'. I personally prefer no change to this.
> 
> > why are you writing @staticmethods that take a self parameter?
> 'self' is more familiar to type, and I can change the class of the method between the Window class and the ActiveState class without modifying the name between 'self' and 'owner'. In real code there's no need to write @staticmethod explicitly; the metaclass of State will automatically change any normal method to staticmethod. And ActiveState is defined in Window.

"More familiar to type" is a bad reason to use something where it has the wrong meaning. You wouldn't try to write exponentiation as "*" because a single asterisk is more familiar than a double, so why would you use "self" instead of the right parameter name? The fact that in this case it only misleads human readers, instead of also misleading the interpreter, doesn't make it any less of a problem. (And having a metaclass that automatically changes methods to @staticmethod doesn't make a difference. Do you call the first parameter of a __new__ method "self" because you don't have to decorate it?)

> Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all.

First, I'm sure lots of people use __getattr__ all over the place. (Personally, I consider the ability to write simple dynamic proxy classes easier than any language except Smalltalk to be a major selling point for Python, and it's been a deciding factor over ObjC in at least one project.)

More importantly: you've had at least two people saying "I see the problem in principle, but I'd like to see real-life code where you're mixing properties that raise AttributeError and __getattr__", and another person trying to dig out details of your proposal and discuss alternatives. How do you interpret that as a lack of interest? If nobody else saw any point to your proposal, nobody would be responding at all.

> Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the dontraise decorator by Chris or any other means.
> 
> 2015-11-28 14:13 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>>> On Nov 27, 2015, at 20:25, 王珺 <wjun77 at gmail.com> wrote:
>>> 
>>> I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
>> 
>> Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem.
>> 
>> (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.)
>> 
>> As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python?
>> 
>>> > The code below doesn't even have a __getattr__ in it
>>> >> In the window example
>>> I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here.
>>> Sorry for the misleading code. Here's the complete version:
>>> 
>>> class ActiveState(State):
>>>     @staticmethod
>>>     def rightClick(self):
>>>         print('right clicked')
>>> class InactiveState(State):
>>>     @staticmethod
>>>     def rightClick(self):
>>>         pass
>>> 
>>> class Window():
>>>     def __init__(self):
>>>         self.__state = ActiveState
>>>     def __getattr__(self, name):
>>>         try:
>>>             return partial(getattr(self.__state, name), self)
>>>         except AttributeError:
>>>             raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None
>>> 
>>>     @property
>>>     def backgroundImg(self):
>>>         if self._backgroundImg is None: #need update, while the number of items changes
>>>             self.set_backgroundImg()
>>>         return self._backgroundImg
>>> 
>>>     def set_backgroundImg(self):
>>>         self._backgroundImg = loadImg('white.bmp')
>>>         for widget in self.widgets:
>>>             widget.show(self._backgroundImg)
>>> 
>>> Class Widget():
>>>     def show(self, img):
>>>         img.draw(self.item.img, self.pos)
>>> 
>>> 
>>> > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>>> I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. 
>>> 
>>> I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 
>>> 
>>> 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>>>> On Friday, November 27, 2015 3:23 PM, 王珺 <wjun77 at gmail.com> wrote:
>>>> 
>>>> >> Do you have any examples that actually do demonstrate the problem to be solved?
>>>> 
>>>> >So you want more details about AttributeError in property?
>>>> 
>>>> No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example.
>>>> 
>>>> What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.
>>>> 
>>>> > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration:
>>>> >
>>>> >class Window():
>>>> >    @property
>>>> >    def backgroundImg(self):
>>>> >        if self._backgroundImg is None: #need update, while the number of items changes
>>>> >            self.set_backgroundImg()
>>>> >        return self._backgroundImg
>>>> >
>>>> >    def set_backgroundImg(self):
>>>> >        self._backgroundImg = loadImg('white.bmp')
>>>> >        for widget in self.widgets:
>>>> >            widget.show(self._backgroundImg)
>>>> >
>>>> >Class Widget():
>>>> >    def show(self, img):
>>>> >        img.draw(self.item.img, self.pos)
>>>> >
>>>> 
>>>> >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.
>>>> 
>>>> No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method.
>>>> 
>>>> I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything.
>>>> 
>>>> 
>>>> Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing.
>>>> 
>>>> No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.
>>>> 
>>>> 
>>>> > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
>>>> 
>>>> 
>>>> Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
>>>> >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
>>>> >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare.
>>>> >
>>>> >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better
>>>> 
>>>> >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.
>>>> 
>>>> Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>>>> >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
>>>> >This is a big problem, OK.
>>>> >
>>>> >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
>>>> >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior?
>>>> 
>>>> 
>>>> No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__.
>>>> 
>>>> The code currently does something like this pseudocode:
>>>> 
>>>>     try:
>>>>         val = obj.__getattribute__(name)
>>>>     except AttributeError:
>>>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>>>         if __getattr__: return __getattr__(name)
>>>> 
>>>> I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__).
>>>> 
>>>> My suggestion is to change it like this:
>>>> 
>>>> 
>>>>     try:
>>>>         val = obj.__getattribute__(name)
>>>>     except AttributeDynamicError:
>>>>         raise
>>>>     except AttributeError:
>>>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>>>         if __getattr__: return __getattr__(name)
>>>> 
>>>> 
>>>> Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).
>>>> 
>>>> So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily.
>>>> 
>>>> Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151129/260b8ece/attachment-0001.html>

[Python-ideas] The AttributeError/__getattr__ mechanism

[Python-ideas] The AttributeError/getattr mechanism