The AttributeError/__getattr__ mechanism
Hello everyone: I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634: http://bugs.python.org/issue25634 I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me. However, this is a complicated problem including performance issues, and backward compatibility. Thanks for your attention, Jun Wang
On 27 November 2015 at 17:49, 王珺 <wjun77@gmail.com> wrote:
I understand the issue you are reporting (and thanks for stripping it down to a simple example in the issue) but could you give a real world example of how this might actually be an issue in practice? I don't believe I've ever heard anyone report this as an issue before, which implies to me that it's not a particularly common problem in real code. I'm not 100% clear on how bad any backward compatibility or performance issues might be, but it's certainly true that we'd need real-world use cases to justify the cost of the change. Paul
It seems like you can get some of the benefits of this proposal without backward compat issues. Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't). That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code. One more thing:
Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs. Sent from my iPhone
I think the real world use case, as *property* is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice? In fact I'm new tweaking these advanced features of python. Any suggestions are welcome and appreciated. In my case, I delegate failed attribute lookups to one member of the instance, as the given simple example in issue25634. For example, there's a class Point, class Point(): def __init__(self, x, y): self.x = x self.y = y and class Circle class Circle(): def __init__(self, center, radius): self.center = center self.radius = radius I don't think Circle should inherit Point. But I want to write Circle().x instead of Circle().center.x, so I write something like def __getattr__(self, name): try: return getattr(self.center, name) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None Another case is when I try to implement the design pattern of state. That is, when calling window.rightClick(), the behavior differs according to the state of the window, or window.__state. So class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass and class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None (The real situation is more complicated. In fact I've written a state module, and I think I can publish it in a week if anyone is interested.)
2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
At first I think of adding AttributeMissError. In that case, object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError to AttributeError. But after some consideration I think there's no need to add an extra Exception. Just ignore the title of that issue. 2015-11-28 4:20 GMT+08:00 Guido van Rossum <guido@python.org>:
On Sat, Nov 28, 2015 at 7:32 AM, 王珺 <wjun77@gmail.com> wrote:
So... if I'm understanding the issue correctly, it's about the interaction of @property and __getattr__? You're talking about this as a boundary over which AttributeError changes. Borrowing your example from the tracker issue: class property(property): def __get__(self, *a): try: return super().__get__(*a) except AttributeError as e: raise RuntimeError("Property raised AttributeError") from e class A(): def __init__(self, x=None): self.x = x @property def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) Traceback (most recent call last): File "attrerr.py", line 4, in __get__ return super().__get__(*a) File "attrerr.py", line 14, in t return self.x.t AttributeError: 'NoneType' object has no attribute 't' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "attrerr.py", line 19, in <module> print(A().t) File "attrerr.py", line 6, in __get__ raise RuntimeError("Property raised AttributeError") from e RuntimeError: Property raised AttributeError This would create a boundary, same as PEP 479 does for StopIteration, across which AttributeError becomes RuntimeError. This could be incorporated into the built-in property if desired, or kept on a per-module basis with the above. ChrisA
On Nov 27, 2015, at 12:18, 王珺 <wjun77@gmail.com> wrote:
I think the real world use case, as property is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice?
Your own examples seem like reasonable uses of __getattr__ to me. But they don't demonstrate your problem, because you don't have any properties, other custom descriptors, or __getattribute__. Do you have any examples that actually do demonstrate the problem to be solved?
Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError Just like raising StopIteration in generator, as described in PEP479.
But the docs didn't recommend raising StopIteration to exit a generator. The docs implied it by omission, or at least we're ambiguous about what would happen, and you could test it and see that it did, which meant some people took advantage of it to do things like using a StopIteration-throwing function in a comprehension filter clause (with caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having to guess why it works). But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
Although I'm not sure StopIteration has ever been documented as one way to exit generator. And I admit that an unintentional AttributeError in __getattribute__ is a rare case if ever.
That's my point: "fixing" __getattribute__ to eliminate a "rare case if ever" bug, while also introducing a possibly less-rare backward compatibility problem, seems like a terrible idea. Of course that isn't your intention; it's just a side effect of trying to eliminate a more common bug, with probably rarer intentional uses, in descriptors. But to me, that implies that any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better. And, as you imply in the bug report, this could be done by having either the descriptor mechanism itself, or the object.__getattribute__ implementation, handle AttributeError from descriptor lookup by reraising it as something else. But still, this is all a minor side issue about choosing between your different variations. The big problem is that I think all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so. Maybe you don't want to do so very often, but is it really so rare that we can justify making it impossible? (Especially considering the backward-compat issues for any code that's been doing that for years...) You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
class Window(): @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos) However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of *show*, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
2015-11-28 5:37 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Sat, Nov 28, 2015 at 10:23 AM, 王珺 <wjun77@gmail.com> wrote:
Hmm. A possibly more general solution is to declare that a function mustn't ever raise a certain exception. from functools import wraps def dontraise(exc): def wrapper(f): @wraps(f) def inner(*a,**kw): try: f(*a,**kw) except exc as e: raise RuntimeError from e return inner return wrapper class A(): def __init__(self, x=None): self.x = x @property @dontraise(AttributeError) def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) By guarding your function with dontraise(AttributeError), you declare that any AttributeError it raises must not leak out. Same as the property change, but not bound to the property class itself. ChrisA
On Friday, November 27, 2015 3:23 PM, 王珺 <wjun77@gmail.com> wrote:
Do you have any examples that actually do demonstrate the problem to be solved?
So you want more details about AttributeError in property?
No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example. What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.
However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.
No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method. I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything. Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing. No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.
In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.
Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__. The code currently does something like this pseudocode: try: val = obj.__getattribute__(name) except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__). My suggestion is to change it like this: try: val = obj.__getattribute__(name) except AttributeDynamicError: raise except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily. Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)
I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos)
I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Nov 27, 2015, at 20:25, 王珺 <wjun77@gmail.com> wrote:
I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem. (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.) As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python?
Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all. Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the *dontraise* decorator by Chris or any other means. 2015-11-28 14:13 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Nov 29, 2015, at 06:31, 王珺 <wjun77@gmail.com> wrote:
If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__.
Your problem is that it took you hours to hunt down where the AttributeError came from, because it gave useless information instead of useful information. The only reason that's happening is that you're handling any AttributeError by swallowing it and raising a new one with useless information. Just take that out, and your hours of debugging go away.
all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__.
The difference, as I've already explained, is that your solution breaks backward compatibility, requiring anyone using such code to change it, while my variation leaves working code alone, only requiring new code to be written differently. All else being equal, the latter is obvious less disruptive a change and therefore better. So, if you think your more-disruptive solution is better, that must mean all else is not equal. But you have to explain how. What's wrong with my version?
(By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.)
All the places exceptions are used for flow control in Python and its stdlib and pythonic code in general use general exceptions. Occasionally (as with generators raising StopIteration where they shouldn't be), this can lead to specific trouble, and that occasional specific trouble gets fixed. Maybe this is one of those cases. That would be two times in as many decades. That's not a good reason to change the general principle.
And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right?
The point is that @property is a high-level, simplified way to write a specific kind of descriptor. It may be perfectly reasonable that properties should never trigger __getattr__; after all, you can always fall back to writing your own descriptor (or even your own property-like descriptor factory--it's only a few lines of code) if you want to. (Again, I'm not sure one way or the other about this.)
@property (or descriptor) converting
Nobody is suggestion descriptors do anything different here. The code would be explicit code in the implementation of @property (and in the Descriptor HOWTO section that shows how @property works).
"More familiar to type" is a bad reason to use something where it has the wrong meaning. You wouldn't try to write exponentiation as "*" because a single asterisk is more familiar than a double, so why would you use "self" instead of the right parameter name? The fact that in this case it only misleads human readers, instead of also misleading the interpreter, doesn't make it any less of a problem. (And having a metaclass that automatically changes methods to @staticmethod doesn't make a difference. Do you call the first parameter of a __new__ method "self" because you don't have to decorate it?)
Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all.
First, I'm sure lots of people use __getattr__ all over the place. (Personally, I consider the ability to write simple dynamic proxy classes easier than any language except Smalltalk to be a major selling point for Python, and it's been a deciding factor over ObjC in at least one project.) More importantly: you've had at least two people saying "I see the problem in principle, but I'd like to see real-life code where you're mixing properties that raise AttributeError and __getattr__", and another person trying to dig out details of your proposal and discuss alternatives. How do you interpret that as a lack of interest? If nobody else saw any point to your proposal, nobody would be responding at all.
On 27 November 2015 at 17:49, 王珺 <wjun77@gmail.com> wrote:
I understand the issue you are reporting (and thanks for stripping it down to a simple example in the issue) but could you give a real world example of how this might actually be an issue in practice? I don't believe I've ever heard anyone report this as an issue before, which implies to me that it's not a particularly common problem in real code. I'm not 100% clear on how bad any backward compatibility or performance issues might be, but it's certainly true that we'd need real-world use cases to justify the cost of the change. Paul
It seems like you can get some of the benefits of this proposal without backward compat issues. Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't). That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code. One more thing:
Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs. Sent from my iPhone
I think the real world use case, as *property* is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice? In fact I'm new tweaking these advanced features of python. Any suggestions are welcome and appreciated. In my case, I delegate failed attribute lookups to one member of the instance, as the given simple example in issue25634. For example, there's a class Point, class Point(): def __init__(self, x, y): self.x = x self.y = y and class Circle class Circle(): def __init__(self, center, radius): self.center = center self.radius = radius I don't think Circle should inherit Point. But I want to write Circle().x instead of Circle().center.x, so I write something like def __getattr__(self, name): try: return getattr(self.center, name) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None Another case is when I try to implement the design pattern of state. That is, when calling window.rightClick(), the behavior differs according to the state of the window, or window.__state. So class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass and class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None (The real situation is more complicated. In fact I've written a state module, and I think I can publish it in a week if anyone is interested.)
2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
At first I think of adding AttributeMissError. In that case, object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError to AttributeError. But after some consideration I think there's no need to add an extra Exception. Just ignore the title of that issue. 2015-11-28 4:20 GMT+08:00 Guido van Rossum <guido@python.org>:
On Sat, Nov 28, 2015 at 7:32 AM, 王珺 <wjun77@gmail.com> wrote:
So... if I'm understanding the issue correctly, it's about the interaction of @property and __getattr__? You're talking about this as a boundary over which AttributeError changes. Borrowing your example from the tracker issue: class property(property): def __get__(self, *a): try: return super().__get__(*a) except AttributeError as e: raise RuntimeError("Property raised AttributeError") from e class A(): def __init__(self, x=None): self.x = x @property def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) Traceback (most recent call last): File "attrerr.py", line 4, in __get__ return super().__get__(*a) File "attrerr.py", line 14, in t return self.x.t AttributeError: 'NoneType' object has no attribute 't' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "attrerr.py", line 19, in <module> print(A().t) File "attrerr.py", line 6, in __get__ raise RuntimeError("Property raised AttributeError") from e RuntimeError: Property raised AttributeError This would create a boundary, same as PEP 479 does for StopIteration, across which AttributeError becomes RuntimeError. This could be incorporated into the built-in property if desired, or kept on a per-module basis with the above. ChrisA
On Nov 27, 2015, at 12:18, 王珺 <wjun77@gmail.com> wrote:
I think the real world use case, as property is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice?
Your own examples seem like reasonable uses of __getattr__ to me. But they don't demonstrate your problem, because you don't have any properties, other custom descriptors, or __getattribute__. Do you have any examples that actually do demonstrate the problem to be solved?
Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError Just like raising StopIteration in generator, as described in PEP479.
But the docs didn't recommend raising StopIteration to exit a generator. The docs implied it by omission, or at least we're ambiguous about what would happen, and you could test it and see that it did, which meant some people took advantage of it to do things like using a StopIteration-throwing function in a comprehension filter clause (with caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having to guess why it works). But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
Although I'm not sure StopIteration has ever been documented as one way to exit generator. And I admit that an unintentional AttributeError in __getattribute__ is a rare case if ever.
That's my point: "fixing" __getattribute__ to eliminate a "rare case if ever" bug, while also introducing a possibly less-rare backward compatibility problem, seems like a terrible idea. Of course that isn't your intention; it's just a side effect of trying to eliminate a more common bug, with probably rarer intentional uses, in descriptors. But to me, that implies that any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better. And, as you imply in the bug report, this could be done by having either the descriptor mechanism itself, or the object.__getattribute__ implementation, handle AttributeError from descriptor lookup by reraising it as something else. But still, this is all a minor side issue about choosing between your different variations. The big problem is that I think all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so. Maybe you don't want to do so very often, but is it really so rare that we can justify making it impossible? (Especially considering the backward-compat issues for any code that's been doing that for years...) You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
class Window(): @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos) However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of *show*, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'. In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
2015-11-28 5:37 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Sat, Nov 28, 2015 at 10:23 AM, 王珺 <wjun77@gmail.com> wrote:
Hmm. A possibly more general solution is to declare that a function mustn't ever raise a certain exception. from functools import wraps def dontraise(exc): def wrapper(f): @wraps(f) def inner(*a,**kw): try: f(*a,**kw) except exc as e: raise RuntimeError from e return inner return wrapper class A(): def __init__(self, x=None): self.x = x @property @dontraise(AttributeError) def t(self): return self.x.t def __getattr__(self, name): return 'default' print(A().t) By guarding your function with dontraise(AttributeError), you declare that any AttributeError it raises must not leak out. Same as the property change, but not bound to the property class itself. ChrisA
On Friday, November 27, 2015 3:23 PM, 王珺 <wjun77@gmail.com> wrote:
Do you have any examples that actually do demonstrate the problem to be solved?
So you want more details about AttributeError in property?
No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example. What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.
However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.
No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method. I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything. Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing. No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.
In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.
Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__. The code currently does something like this pseudocode: try: val = obj.__getattribute__(name) except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__). My suggestion is to change it like this: try: val = obj.__getattribute__(name) except AttributeDynamicError: raise except AttributeError: __getattr__ = getattr(type(obj), '__getattr__', None) if __getattr__: return __getattr__(name) Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily. Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)
I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
class ActiveState(State): @staticmethod def rightClick(self): print('right clicked') class InactiveState(State): @staticmethod def rightClick(self): pass class Window(): def __init__(self): self.__state = ActiveState def __getattr__(self, name): try: return partial(getattr(self.__state, name), self) except AttributeError: raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None @property def backgroundImg(self): if self._backgroundImg is None: #need update, while the number of items changes self.set_backgroundImg() return self._backgroundImg def set_backgroundImg(self): self._backgroundImg = loadImg('white.bmp') for widget in self.widgets: widget.show(self._backgroundImg) Class Widget(): def show(self, img): img.draw(self.item.img, self.pos)
I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Nov 27, 2015, at 20:25, 王珺 <wjun77@gmail.com> wrote:
I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem. (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.) As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python?
Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all. Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the *dontraise* decorator by Chris or any other means. 2015-11-28 14:13 GMT+08:00 Andrew Barnert <abarnert@yahoo.com>:
On Nov 29, 2015, at 06:31, 王珺 <wjun77@gmail.com> wrote:
If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__.
Your problem is that it took you hours to hunt down where the AttributeError came from, because it gave useless information instead of useful information. The only reason that's happening is that you're handling any AttributeError by swallowing it and raising a new one with useless information. Just take that out, and your hours of debugging go away.
all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__.
The difference, as I've already explained, is that your solution breaks backward compatibility, requiring anyone using such code to change it, while my variation leaves working code alone, only requiring new code to be written differently. All else being equal, the latter is obvious less disruptive a change and therefore better. So, if you think your more-disruptive solution is better, that must mean all else is not equal. But you have to explain how. What's wrong with my version?
(By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.)
All the places exceptions are used for flow control in Python and its stdlib and pythonic code in general use general exceptions. Occasionally (as with generators raising StopIteration where they shouldn't be), this can lead to specific trouble, and that occasional specific trouble gets fixed. Maybe this is one of those cases. That would be two times in as many decades. That's not a good reason to change the general principle.
And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part). Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right?
The point is that @property is a high-level, simplified way to write a specific kind of descriptor. It may be perfectly reasonable that properties should never trigger __getattr__; after all, you can always fall back to writing your own descriptor (or even your own property-like descriptor factory--it's only a few lines of code) if you want to. (Again, I'm not sure one way or the other about this.)
@property (or descriptor) converting
Nobody is suggestion descriptors do anything different here. The code would be explicit code in the implementation of @property (and in the Descriptor HOWTO section that shows how @property works).
"More familiar to type" is a bad reason to use something where it has the wrong meaning. You wouldn't try to write exponentiation as "*" because a single asterisk is more familiar than a double, so why would you use "self" instead of the right parameter name? The fact that in this case it only misleads human readers, instead of also misleading the interpreter, doesn't make it any less of a problem. (And having a metaclass that automatically changes methods to @staticmethod doesn't make a difference. Do you call the first parameter of a __new__ method "self" because you don't have to decorate it?)
Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all.
First, I'm sure lots of people use __getattr__ all over the place. (Personally, I consider the ability to write simple dynamic proxy classes easier than any language except Smalltalk to be a major selling point for Python, and it's been a deciding factor over ObjC in at least one project.) More importantly: you've had at least two people saying "I see the problem in principle, but I'd like to see real-life code where you're mixing properties that raise AttributeError and __getattr__", and another person trying to dig out details of your proposal and discuss alternatives. How do you interpret that as a lack of interest? If nobody else saw any point to your proposal, nobody would be responding at all.
participants (5)
-
Andrew Barnert -
Chris Angelico -
Guido van Rossum -
Paul Moore -
王珺