'hasattr' is broken by design
Hello, I know the issue has been discussed several times already, however I couldn't find any reasonable explanation of its strange behaviour. The main problem with 'hasattr' function is that is swallows all exceptions derived from Exception class. It's a good thing that it doesn't do that with BaseException as it was fixed not a long time ago, but it's definitely not enough. First of all, this behaviour of 'hasattr' contradicts with the very core principle of python: "Errors should never pass silently." And since 'hasattr' function is in builtins module and is a widely used function it impacts the whole language. Secondly, take a look at the following: >>> class Test: ... @property ... def attr(self): ... self['foo'] ... >>> hasattr(Test(), 'attr') False There can be any exception instead of KeyError in the above snippet of code, but this small example shows how 'hasattr': misleadingly breaks the code logic (1) and masks bug (2). And that's the simplest possible example, there are much more in real life. While (1) is maybe acceptable for someone, there is no excuse for the (2). Moreover, current 'hasattr' behaviour tremendously complicates use of '__getattribute__' magic. And forget about importlib magic with LazyImports, one 'hasattr' ruins everything by catching ImportError. To conclude: 1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance. 2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected. - Yury Selivanov, Sprymix Inc. +1-416-509-2807
2010/8/23 Yury Selivanov <yselivanov@gmail.com>:
1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.
I would be in support of that.
2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.
But not this. Compatibility functions don't belong in 2to3. -- Regards, Benjamin
On 2010-08-23, at 10:46 AM, Benjamin Peterson wrote:
There are many possible solutions for the Python 2 porting issue, this one was just one of them. I was trying to make a point, that it's possible to somehow make porting process easier, and meanwhile fix Python 3. - Yury Selivanov Sprymix Inc.
On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson <benjamin@python.org> wrote:
I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now. I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.
2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.
But not this. Compatibility functions don't belong in 2to3.
Indeed. -- --Guido van Rossum (python.org/~guido)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote:
Robust third-party code is written to avoid 'hasattr', for precisely this reason. Third-party code which relies on 'hasattr' to mask non-AttributeErrors is broken alreay (it just doesn't show the borkedness). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxymr0ACgkQ+gerLs4ltQ4rQwCgyHJmqt2TefCgX2di5aJ92pVh 26YAnjKrBrK3gMs7ddo2wHtpT+iq2Mbg =BFxu -----END PGP SIGNATURE-----
Have done a little testing. The patch to builtin 'hasattr' function is trivial, and can be found attached to this letter (with the corresponding unittest): - if (!PyErr_ExceptionMatches(PyExc_Exception)) + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) So, after applying it, hasattr swallows only AttributeError exceptions. All tests from Python 3.2 test suite were good. After that, I've applied the patch on Python 2.6 and tested SqlAlchemy, Django and Twisted. - SqlAlchemy has failed three unittests, but those tests were designed specifically to handle 'hasattr' weird behaviour, so we can consider the change has no impact on SqlAlchemy. - Twisted - failed 3 tests out of ~3200, but it fails them on the same machine on an unpatched Python too, and they seem unrelated. - Django - all tests passed. I tested our internal company framework (~100K LOC) and, again, all tests passed.
Yuri, I think you are making a good case (though I would like for you to be a good citizen and use the bug tracker to submit this for review). Benjamin, what do you think? --Guido On Mon, Aug 23, 2010 at 7:14 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 08/23/2010 04:56 PM, Guido van Rossum wrote:
I'll note that a similar incompatible change has made it to python2.6. This has bitten us in production: class X(object): def __getattr__(self, name): raise KeyError, "error looking for %s" % (name,) def __iter__(self): yield 1 print list(X()) I would expect it to print [1], and in python2.5 it does. In python2.6 it raises a KeyError! The attribute being looked up is an unexpected one: {hrzagude5003}[~]$ python2.6 a.py Traceback (most recent call last): File "a.py", line 8, in <module> print list(X()) File "a.py", line 3, in __getattr__ raise KeyError, "error looking for %s" % (name,) KeyError: 'error looking for __length_hint__' The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list(). Being aware that this can be construed as an argument both in favor and against the change at hand, my point is that, if propagating non-AttributeError exceptions is done in checks intended to be invisible, it should certainly be done in hasattr, where it is at least obvious what is being done. Other generic functions and operators, including boolean ones such as ==, happily propagate exceptions. Also, don't expect that this won't break code out there. It certainly will, it's only a matter of assessment whether such code was broken in a different, harder to detect way, to begin with.
On 08/24/2010 02:31 PM, Benjamin Peterson wrote:
Of course, but that's beside the point. In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail). The point I'm making is that: a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and b) making the proposed change is bound to break real, production code. I still agree with the proposed change, but I wanted to also point out that it will cause breakage and illustrate it with a similar real-world example that occurred during migration to python 2.6.
On 8/24/2010 9:45 AM, Benjamin Peterson wrote:
+1 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:
a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and
FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes. I don't think this is documented anywhere, but experience got this pretty ingrained in my head since Python 2.2 or even earlier.
On Aug 24, 2010, at 8:31 AM, Benjamin Peterson wrote:
As it happens though, list() is _quite_ public. Saying "X is internal and undocumented, so it can do whatever it wants" is never really realistic, especially in response to someone saying "we already saw this problem in production, _without_ calling / referring to / knowing about this private API".
On 8/23/2010 10:22 AM, Yury Selivanov wrote:
I gather that this amounts to changing "an exception" to "AttributeError" in "(This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.)" in both the doc and implementation, so that the implementation actually matches the claimed behavior "The result is True if the string is the name of one of the object’s attributes, False if not. " (and by reasonable implication, an exception if this cannot be determined). -- Terry Jan Reedy
On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:
Thanks for the nice analysis and good example. I disagree with the solution though. If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change. But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior. As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key. IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code. Raymond
On 23/08/2010 22:47, Raymond Hettinger wrote:
It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though. For what it's worth I *agree* with you [1], but for better or worse hasattr / getattr trigger code execution and hasattr can return True for dynamically created members. Something to be revisited for Python 4 perhaps. Michael Foord [1] A while ago I wrote a couple of blog entries on fetching docstrings from members without triggering code execution. It is surprisingly convoluted and even my final code had corner cases it didn't handle: http://www.voidspace.org.uk/python/weblog/arch_d7_2009_05_16.shtml#e1090 http://www.voidspace.org.uk/python/weblog/arch_d7_2009_06_20.shtml#e1103
Michael Foord wrote:
It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though.
Also keep in mind that builtin types mostly don't keep their attributes in dictionaries. To make this work properly, hasattr would need its own special method. -- Greg
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
That would break the assumption that: if hasattr(obj, attr): getattr(obj, attr) # won't raise and hasattr ~= try: getattr(obj, attr) except AttributeError: return False else: return True
That doesn't sound to useful to me. A descriptor could be found with __get__, but that __get__ could just as well raise AttributeError.
Can you provide an example? I've never seen code which explicitly scans MRO and dicts to avoid triggering code. (Besides collections.Callable; that's a special case.) -- Regards, Benjamin
On 23/08/2010 22:59, Benjamin Peterson wrote:
The example I linked to in my previous email did exactly that - the use case was for finding and displaying docstrings on members in an interactive object viewer. We needed to be able to examine objects without triggering code execution in them. To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. Michael
On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:
Well said. The surprise to me in the OP's example was that the property() was executed. Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr(). That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods).
That's the danger of a dynamic language like Python. Even dir() can now trigger things like that.
That's not a honking good thing. I suggest we don't do more of that. Raymond
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
hasattr(x, "y") doesn't look any more passive to me the x.y.
Hard to remember compared to what?
Better to raise an exception into unexpecting code that to have subtly different lookup rules between getattr and hasattr. -- Regards, Benjamin
On 23/08/2010 23:55, Benjamin Peterson wrote:
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not. I'm in both camps though. As we *are* triggering code execution I don't think we should mask exceptions. I just wish we weren't triggering code execution unnecessarily. Michael
2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:
Actually, I'd say given Python's object model, you have to have the attribute in hand to determine that it is present.
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not.
Properties are allowed to raise AttributeError, so you technically have to execute it. -- Regards, Benjamin
On 24/08/2010 00:05, Benjamin Peterson wrote:
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not. If fetching an attribute raises an AttributeError it doesn't mean that attribute doesn't exist (although I admit that at the moment this is exactly what hasattr uses to mean) it just means that fetching that attribute raised an AttributeError. Even if you allow other exceptions to propagate you are still left with the fact that an AttributeError raised as a bug will still be silenced and interpreted as meaning that hasattr should just return False. Michael Foord -- http://www.ironpythoninaction.com/
2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.
I thought you were trying to determine whether the attribute exists not the property.
Raised as a bug? Is this not a valid pattern? @property def myprop(self): if not self.myprop_support: raise AttributeError("myprop") -- Regards, Benjamin
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.
If you don't want to execute properties, do the lookup on the type, not the instance (obviously, you know the dance you need to do, since you've linked the code where you did it). Having apparently simple query operations like hasattr/getattr/len/etc execute arbitrary code is where much of the language's flexibility comes from, so I don't see how it can really be surprising when it happens. To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError". That means it can't figure out whether or not the attribute exists without actually attempting to retrieve it. There are a few places where we instead use a heuristic that says an attribute *probably* exists if it appears in the instance dictionary, or in the dictionary of one of the types in the MRO, and magic methods have the rule that they must be defined on the type rather than the instance in order to count from the interpreter's point of view, but neither of those things change the basic definition. For the record, +1 on narrowing the scope of the exception suppression in hasattr() to only AttributeError, and adding new C API functions that expose the new behaviour. (I've actually long assumed that AttributeError *was* the only thing suppressed by hasattr()). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24/08/2010 00:40, Nick Coghlan wrote:
Certainly that is true for len. getattr obviously involves invoking code if you are fetching a property or descriptor. No idea how you conclude that hasattr executing code adds flexibility to the language though. Yes I know the dance (walking the mro fetching the attribute out of the appropriate type __dict__ or the instance dict - or looking on the metaclass if the object you are introspecting is a type itself), it is just not trivial - which is why I think it is a shame that people are forced to implement it just to ask if a member exists without triggering code execution.
To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError".
Right, and to me Python's object model (the lookup rules hinted at above) define whether or not an object "has an attribute" or not. We just disagree on this. However, it is irrelevant so not really worth continuing the discussion. Changing hasattr in this way would be backwards incompatible and so cannot be done. Doesn't mean I have to like it though. :-) Michael
On Tue, Aug 24, 2010 at 7:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Now, it may be worth considering an addition to the inspect module that was basically: def getattr_static(obj, attr): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass (not needing to deal with classic classes simplifies things a bit) So, allowing for the fact that dir() may report attributes that can only be found via dynamic lookup, your get_docstrings example could become something like: def get_docstrings(obj): try: members = dir(obj) except Exception: members = [] for member in members: try: doc = getattr_static(obj, member).__doc__ except AttributeError: doc = None yield member, doc Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Second attempt with a default value parameter and correctly raising AttributeError if not found: _sentinel = object() def getattr_static(obj, attr, default=_sentinel): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass if default is not _sentinel: return default raise AttributeError(attr) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24/08/2010 01:25, Nick Coghlan wrote:
This doesn't correctly handle the case where obj is a type object or obj uses __slots__. If I have time (currently on vacation with only intermittent internet access) I'll provide an update. Michael
Cheers, Nick.
On 24/08/2010 08:40, Michael Foord wrote:
I managed an updated version (with tests) during the flight home last night. I've attached it to issue 9732, along with a discussion of the caveats and ways of breaking it: http://bugs.python.org/issue9732 Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 2010-08-23, at 5:02 PM, Michael Foord wrote:
As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour. - Yury
On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour.
You nailed it. At the lowest level (either in C or in Python) there is no way to check for an attribute's presence without getting its value. This has so far been fundamental, and everything else is just appearances. I propose that this is good enough and that we should at this point not try to invent another protocol, just decide whether hasattr() can be fixed (and leaving PyObject_HasAttr*() alone). -- --Guido van Rossum (python.org/~guido)
BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general. The more I think about it the more I like the idea. By default, 'hasattr' would check MRO for the attribute, if not found - check for __hasattr__, if not found - fallback to the current schema with 'getattr'. This would not break any current code, but open a lot of potential (especially for things that implement some lazy loading protocols). From the performance point of view, I believe it shouldn't change much, as current 'hasattr' which uses 'getattr' which calls __getattr(ibute)__ methods. I know this is much more then just fixing exception swallowing, but this is somewhat required functionality to complete the current dynamism python offers. - Yury
By default, if you don't overload __hasattr__, it will do exactly that. But let's imagine the situation in some ORM. Sometimes, engine knows that attribute exists even before executing it (which leads to querying DB), and thus it is possible to return True in the overloaded __hasattr__. Example: ... if hasattr(entity, 'title'): # <- __hasattr__ has the info in metadata, # no query to DB ... title = entity.title # <- now, we can query the DB So, the proposed magic method is not intended to change the protocol, but to complement and enhance it. - Yury
2010/8/23 Yury Selivanov <yselivanov@gmail.com>:
In this case, I think it would make more sense to ask the model: type(entity).has_property("title")
So, the proposed magic method is not intended to change the protocol, but to complement and enhance it.
But it still raises the potential to break the relationship between hasattr and getattr. I think this should require a PEP. -- Regards, Benjamin
On Tue, Aug 24, 2010 at 8:25 AM, Benjamin Peterson <benjamin@python.org> wrote:
Definitely needs a PEP, and will require some solid use cases to explain why allowing optimisation of hasattr() is the right way to go. Complexity isn't free, and I doubt the gains here will justify the costs, but that's one of the things the PEP process is intended to figure out. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
At 06:12 PM 8/23/2010 -0400, Yury Selivanov wrote:
BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general.
In order to do this properly, you'd need to also add __has__ or __exists__ (or some such) to the descriptor protocol; otherwise you break descriptors' ability to operate independently of the class they're used in. You would probably also need a __hasattribute__, in order to be able to properly synchronize with __getattr__/__getattribute__. Seems like overkill to me, though, as I'm not sure how such a protocol actually helps ORM or persistence schemes (and I've written a few). Pretty much, if you're trying to check for the existence of an attribute, you're probably about to be getting that attribute anyway. (i.e. why query the existence of an attribute you *don't* intend to use?)
On Mon, Aug 23, 2010 at 3:45 PM, P.J. Eby <pje@telecommunity.com> wrote:
Right. This sounds like way too big a gun to kill this particular mosquito. If Yury wants to write a PEP I won't stop him, but I expect that it will be rejected for want of important use cases compared to the complexity of the solution. There just are too many places that would be affected, for too little value. So just be warned. OTOH I still think that fixing hasattr() to be mroe like getattr(obj, key, None) has a high value and a relative low risk. -- --Guido van Rossum (python.org/~guido)
At 12:02 AM 8/24/2010 +0300, Michael Foord wrote:
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not.
That depends on what you mean by "exists". Note that a property might raise AttributeError to signal that the attribute is not currently set. Likewise, unless you special case __slots__ descriptors, you can have the bizarre condition where hasattr() will return True, but getattr() will still raise an AttributeError. The idea that you could determine the presence of an attribute on an object without executing that object's code is something that hasn't been practical since the birth of descriptors in Python 2.2.
Even if you implement it, you will get wrong answers in some cases. __getattribute__ is allowed to throw out the entire algorithm you just described and replace it utterly with something else. My ProxyTypes library makes use of that fact, for example, so if you actually attempted to inspect a proxy instance with your re-implemented "dance", your code will fail to notice what attributes the proxy actually has.
On 23/08/2010 23:13, Benjamin Peterson wrote:
Well yes. One of the reasons a full workaround is so arcane is that if you *really* don't want to trigger code execution you can't call dir... (Otherwise name in dir(obj) is a reasonable approximation for hasattr(obj, name).) Michael -- http://www.ironpythoninaction.com/
On 2010-08-23, at 3:47 PM, Raymond Hettinger wrote:
This is impossible to implement because of '__getattribute__' and '__getattr__' methods. There is no way of detecting the presence of an attribute but trying to get it through 'getattr'. Partial solution like getting information about property presence through __dict__ and call __getattribute__/__getattr__ if they are defined wouldn't work either. So, your solution would make 'hasattr' even more incompatible. - Yury Selivanov
On Mon, Aug 23, 2010 at 12:47 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Hm... That sounds like scope creep to me. Properties are supposed to be cheap and idempotent. Trying to figure out whether an attribute exists without using __getattr__ is fraught with problems -- as soon as a class overrides __getattr__ or __getattribute__ you're hosed anyway. I can vouch that the reason hasattr() catches too many exceptions is that when I first added it (around 1990, I think :-) I wasn't very attuned yet to the problems it could cause. The main problem I can see with letting exceptions other than AttributeError bubble through (besides perverted dependencies on the current semantics) is that there are some situations where it is pretty arbitrary whether TypeError or AttributeError is raised. I can't recall the details, and possibly this was more of a problem with classic classes, but I do think it warrants some research. -- --Guido van Rossum (python.org/~guido)
2010/8/23 Guido van Rossum <guido@python.org>:
I believe this was with regards to actual operations, though, not fetching the attribute. For example, float(a) would raise an AttributeError for a classic instance and a TypeError for a new-style instance. However both would raise AttributeError on a.__float__. -- Regards, Benjamin
On Aug 23, 2010, at 1:03 PM, Guido van Rossum wrote:
I don't have a specific proposal in mind. My main questions are * Is there anything that hasattr(obj, key) can or should do that can't already be done with getattr(obj, key, None)? If not, do we really need to change anything? * Why do people typically use hasattr() instead getattr()? Aren't they are really trying to just determine whether a key exists somewhere in the MRO? If so, then doing anything more than that is probably a surprise. I know my own uses of hasattr() do not expect any exceptions at all. It comes up in duck typing support different handling for different types of inputs. If others are using it the same way, I think it is unlikely that they have unittests to cover the possibility that hasattr() would ever start raising exceptions.
Fire-up the time machine? Raymond P.S. The current behavior seems to be deeply embedded: int PyObject_HasAttr(PyObject *o, PyObject *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds. int PyObject_HasAttrString(PyObject *o, const char *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds.
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
For one, it's annoying when "key" can be None.
It's generally more convenient that getattr(obj, "blah", None) is not None.
P.S. The current behavior seems to be deeply embedded:
Well, that's what happens when a behavior is added in 1990. :) -- Regards, Benjamin
On Mon, Aug 23, 2010 at 1:50 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Changing C APIs is even harder than changing Python API because you can't add exceptions to something that wasn't returning exceptions before. We did that for comparisons in the past and it was a pain (but worth it). For these two little APIs I think it is not worth it, though it may be worth it to create new APIs that *do* return exceptions. -- --Guido van Rossum (python.org/~guido)
On Mon, Aug 23, 2010 at 1:33 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
I don't have a specific proposal in mind.
That's why I called it scope creep. :-) Trust me, your proposal will not lead to a quick and better replacement for hasattr(). (See several other people's replies.)
getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly. * Why do people typically use hasattr() instead getattr()?
Most users who call hasattr() probably don't even know what MRO means. They call hasattr() because they want to avoid a try/except clause. The reasons they are not calling getattr(obj, key, None) could be many: never heard of it, too obscure (it surely doesn't spell "has the attribute" like hasattr() does to the beginning user), or (probably common) the actual attribute access is in some other piece of code they are about to call but don't control.
It already raises *some* exceptions (those derived from BaseException but not Exception). I think that change was a definite non-event.
But note that hasattr() doesn't call those.
-- --Guido van Rossum (python.org/~guido)
On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:
Why do you say it's ugly? It's a short, sweet, simple two-liner: mark = object() getattr(obj, 'key', mark) is not mark Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function. It still doesn't cope with dynamically-generated attributes that are either expensive or have side-effects (both of which are probably poor design, but nevertheless I'm sure they're out there), but neither does the existing hasattr.
Well, yes, but most users never write __getattr__ or __getattribute__ methods either. I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__. Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist. -- Steven D'Aprano
On Mon, Aug 23, 2010 at 17:04, Benjamin Peterson <benjamin@python.org> wrote:
It is also non-obvious to any beginner. Are we really going to want to propagate the knowledge of this trick as a fundamental idiom? I would rather leave hasattr in that instance. But I'm +1 on only swallowing AttributeError. -Brett
On Mon, Aug 23, 2010 at 4:56 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Because if you didn't have the foresight to write that utility function, you have to break your train of thought (aka yak shaving) to either write it (big yak) or write those two lines (little yak, times the number of times it happens). Whereas with hasattr() you can just type the correct hasattr() expression in-line in the if that you already started typing.
It tests for the existence of an attribute -- how the attribute is defined should not have to occur to you (and there are lots of other ways for attributes to be defined besides the ways you came up with just now).
The try/except block *also* requires you to break your train of thought. And most of the time the error case just isn't important. You sound like you are over-engineering it and focusing too much on performance instead of on getting things done. Like those people who learn that it saves an usec to copy a built-in function into a defalt arg (def foo(arg1, len=len): ...) and then overuse the trick even when the time it took them to write the exra line is more than the time they'll save in a lifetime in execution time. -- --Guido van Rossum (python.org/~guido)
On Tue, 24 Aug 2010 11:09:10 am Guido van Rossum wrote:
But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist. If it existed, __getattr__ would never be called. A minor semantic difference, to be sure, but it's real to me. Whether I should care about the difference is a separate issue. This conversation has been valuable to me for one thing though... it reminded me of a piece of code I had written a long time ago. A simplified version: class K(object): def __getattribute__(self, name): if hasattr(self, name): # no computation needed print "Attr exists" else: # compute something... print "Attr doesn't exist" I couldn't work out why it was behaving so strangely, and rather than spend time investigating, I abandoned the whole dynamic attribute approach and did something completely different. So at least now I know why it wasn't working as I expected.
I never suggested that it would be easy.
Performance could be an issue, of course, if somebody writes an expensive __getattr__ or property. Computed attributes should be cheap. But I'm actually more concerned about side-effects than performance. If I'm not worried about potential side-effects, I use a try...except block. If I am worried, I "Look Before You Leap". Only now I've learned that what I thought was LBYL is nothing of the sort, and hasattr gives me no protection against side-effects. That being the case, I might as well just stick to try...except and be done with it. I'm not suggesting this is the One True Way. If others prefer hasattr, then I have no problem with that. I'm just saying that now that I know what it actually does, its value *for me* is minimal.
Aha! The penny drops! Is that why so many methods in random.Random have an "int=int" argument? E.g. def randrange(self, start, stop=None, step=1, int=int, default=None, maxwidth=1L<<BPF): I'd wondered about that. So there you go, now I've learned two things. -- Steven D'Aprano
On Tue, Aug 24, 2010 at 4:51 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Eh? If "x.y" succeeds, in what sense does y not exist?
Whether I should care about the difference is a separate issue.
Right, you are breaking through too much abstraction.
Performance could be an issue, of course, if somebody writes an expensive __getattr__ or property. Computed attributes should be cheap.
Yes, and it should be considered the problem of the author of that property, not of the user.
But I'm actually more concerned about side-effects than performance.
Properties should not have side effects. That would be a problem when using them just as much as with hasattr(). -- --Guido van Rossum (python.org/~guido)
Steven D'Aprano wrote:
But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist.
Maybe for your particular use case, but the concept of whether an attribute is dynamically defined or not is not well-defined in general. Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want. -- Greg
At 12:10 PM 8/25/2010 +1200, Greg Ewing wrote:
Yep. That's why the proposed addition to inspect is a bad idea. If we encourage that sort of static thinking, it will lead to people creating all sorts of breakage with respect to more dynamic code. AFAICT, the whole "avoid running code" thing only makes sense for a debugging tool -- at which point, you can always use the trace facility and throw an error when any Python code runs that's not part of your debugging tool. Something like: def exists(ob, attr): __running__ = True # ... set trace function here try: try: getattr(ob, attr) return True except AttributeError: return False except CodeRanError: return True # or False if you prefer finally: __running__ = False # restore old tracing here Where the trace function is just something that throws CodeRanError if it detects a "call" event and the __running__ flag is True. This would stop any Python code from actually executing. (It'd need to keep the same trace function for c_call events, since that might lead to nested non-C calls .) Of course, a debugger's object inspection tool would probably actually want to return either the attribute value, or a special value to mean "dyanmic calculation needed".
On 25/08/2010 19:27, P.J. Eby wrote:
I mentioned another use case - pulling out docstrings. IDEs or other tools that work with live objects may have many such use cases. For proxying objects you can't use the __getattr__ approach for the "magic methods" which aren't looked up through the dynamic attribute 'faking' process. Several proxy libraries I've seen get round this by providing *every* magic method on the proxy class and delegating. This still breaks certain types of duck typing. For example with Python 2.6:
If your proxy class defines __call__ then callable returns True, even if the delegation to the proxied object would cause an AttributeError to be raised. A *better* approach (IMO), for both the magic methods and the normal attributes / methods, is to dynamically generate a class that mimics the proxied object (caching generated classes for proxying objects of the same type if you are worried about overhead). This would also work with the suggested "passive introspection" function. All the best, Michael Foord
At 08:58 PM 8/25/2010 +0300, Michael Foord wrote:
Nope. You just have to use delegate via __getattribute__ (since 2.2) instead of __getattr__:
As you can see, the __call__ attribute in each case is whatever the proxied object's __call__ attribute is, even though the proxy itself has a __call__ method, that is invoked when the proxy is called. This is actually pretty straightforward stuff since the introduction of __getattribute__. (The code is at http://pypi.python.org/pypi/ProxyTypes, btw.)
On 25 August 2010 20:57, P.J. Eby <pje@telecommunity.com> wrote:
For what it's worth this code is useful enough (and simple enough) that I would support its inclusion in the standard library. I've written proxy objects several times (for various different purposes) and this would have saved me the effort. :-) All the best, Michael Foord -- http://www.voidspace.org.uk
On 2010-08-23, at 4:33 PM, Raymond Hettinger wrote:
OK, if my key has None value, how will you detect it with 'getattr'? ... marker = object() ... if (getattr(obj, key, marker) is not marker: Like in the above? Too much code. Ugly code. Just to hide the issue under the carpet.
My example in the initial email clearly showed that sometimes, key is in MRO, but 'hasattr' returns False. For everything in any complicated system (which Python obviously is) should be a strict protocol. In case of properties this protocol is to raise AttributeError if property is missing. Everything else is a potential bug. Current 'hasattr' just masks them. - Yury
2010/8/23 Yury Selivanov <yselivanov@gmail.com>:
1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.
I would be in support of that.
2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.
But not this. Compatibility functions don't belong in 2to3. -- Regards, Benjamin
On 2010-08-23, at 10:46 AM, Benjamin Peterson wrote:
There are many possible solutions for the Python 2 porting issue, this one was just one of them. I was trying to make a point, that it's possible to somehow make porting process easier, and meanwhile fix Python 3. - Yury Selivanov Sprymix Inc.
On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson <benjamin@python.org> wrote:
I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now. I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.
2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.
But not this. Compatibility functions don't belong in 2to3.
Indeed. -- --Guido van Rossum (python.org/~guido)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guido van Rossum wrote:
Robust third-party code is written to avoid 'hasattr', for precisely this reason. Third-party code which relies on 'hasattr' to mask non-AttributeErrors is broken alreay (it just doesn't show the borkedness). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxymr0ACgkQ+gerLs4ltQ4rQwCgyHJmqt2TefCgX2di5aJ92pVh 26YAnjKrBrK3gMs7ddo2wHtpT+iq2Mbg =BFxu -----END PGP SIGNATURE-----
Have done a little testing. The patch to builtin 'hasattr' function is trivial, and can be found attached to this letter (with the corresponding unittest): - if (!PyErr_ExceptionMatches(PyExc_Exception)) + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) So, after applying it, hasattr swallows only AttributeError exceptions. All tests from Python 3.2 test suite were good. After that, I've applied the patch on Python 2.6 and tested SqlAlchemy, Django and Twisted. - SqlAlchemy has failed three unittests, but those tests were designed specifically to handle 'hasattr' weird behaviour, so we can consider the change has no impact on SqlAlchemy. - Twisted - failed 3 tests out of ~3200, but it fails them on the same machine on an unpatched Python too, and they seem unrelated. - Django - all tests passed. I tested our internal company framework (~100K LOC) and, again, all tests passed.
Yuri, I think you are making a good case (though I would like for you to be a good citizen and use the bug tracker to submit this for review). Benjamin, what do you think? --Guido On Mon, Aug 23, 2010 at 7:14 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 08/23/2010 04:56 PM, Guido van Rossum wrote:
I'll note that a similar incompatible change has made it to python2.6. This has bitten us in production: class X(object): def __getattr__(self, name): raise KeyError, "error looking for %s" % (name,) def __iter__(self): yield 1 print list(X()) I would expect it to print [1], and in python2.5 it does. In python2.6 it raises a KeyError! The attribute being looked up is an unexpected one: {hrzagude5003}[~]$ python2.6 a.py Traceback (most recent call last): File "a.py", line 8, in <module> print list(X()) File "a.py", line 3, in __getattr__ raise KeyError, "error looking for %s" % (name,) KeyError: 'error looking for __length_hint__' The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list(). Being aware that this can be construed as an argument both in favor and against the change at hand, my point is that, if propagating non-AttributeError exceptions is done in checks intended to be invisible, it should certainly be done in hasattr, where it is at least obvious what is being done. Other generic functions and operators, including boolean ones such as ==, happily propagate exceptions. Also, don't expect that this won't break code out there. It certainly will, it's only a matter of assessment whether such code was broken in a different, harder to detect way, to begin with.
On 08/24/2010 02:31 PM, Benjamin Peterson wrote:
Of course, but that's beside the point. In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail). The point I'm making is that: a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and b) making the proposed change is bound to break real, production code. I still agree with the proposed change, but I wanted to also point out that it will cause breakage and illustrate it with a similar real-world example that occurred during migration to python 2.6.
On 8/24/2010 9:45 AM, Benjamin Peterson wrote:
+1 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/
At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:
a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and
FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes. I don't think this is documented anywhere, but experience got this pretty ingrained in my head since Python 2.2 or even earlier.
On Aug 24, 2010, at 8:31 AM, Benjamin Peterson wrote:
As it happens though, list() is _quite_ public. Saying "X is internal and undocumented, so it can do whatever it wants" is never really realistic, especially in response to someone saying "we already saw this problem in production, _without_ calling / referring to / knowing about this private API".
On 8/23/2010 10:22 AM, Yury Selivanov wrote:
I gather that this amounts to changing "an exception" to "AttributeError" in "(This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.)" in both the doc and implementation, so that the implementation actually matches the claimed behavior "The result is True if the string is the name of one of the object’s attributes, False if not. " (and by reasonable implication, an exception if this cannot be determined). -- Terry Jan Reedy
On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:
Thanks for the nice analysis and good example. I disagree with the solution though. If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change. But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior. As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key. IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code. Raymond
On 23/08/2010 22:47, Raymond Hettinger wrote:
It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though. For what it's worth I *agree* with you [1], but for better or worse hasattr / getattr trigger code execution and hasattr can return True for dynamically created members. Something to be revisited for Python 4 perhaps. Michael Foord [1] A while ago I wrote a couple of blog entries on fetching docstrings from members without triggering code execution. It is surprisingly convoluted and even my final code had corner cases it didn't handle: http://www.voidspace.org.uk/python/weblog/arch_d7_2009_05_16.shtml#e1090 http://www.voidspace.org.uk/python/weblog/arch_d7_2009_06_20.shtml#e1103
Michael Foord wrote:
It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though.
Also keep in mind that builtin types mostly don't keep their attributes in dictionaries. To make this work properly, hasattr would need its own special method. -- Greg
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
That would break the assumption that: if hasattr(obj, attr): getattr(obj, attr) # won't raise and hasattr ~= try: getattr(obj, attr) except AttributeError: return False else: return True
That doesn't sound to useful to me. A descriptor could be found with __get__, but that __get__ could just as well raise AttributeError.
Can you provide an example? I've never seen code which explicitly scans MRO and dicts to avoid triggering code. (Besides collections.Callable; that's a special case.) -- Regards, Benjamin
On 23/08/2010 22:59, Benjamin Peterson wrote:
The example I linked to in my previous email did exactly that - the use case was for finding and displaying docstrings on members in an interactive object viewer. We needed to be able to examine objects without triggering code execution in them. To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. Michael
On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:
Well said. The surprise to me in the OP's example was that the property() was executed. Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr(). That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods).
That's the danger of a dynamic language like Python. Even dir() can now trigger things like that.
That's not a honking good thing. I suggest we don't do more of that. Raymond
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
hasattr(x, "y") doesn't look any more passive to me the x.y.
Hard to remember compared to what?
Better to raise an exception into unexpecting code that to have subtly different lookup rules between getattr and hasattr. -- Regards, Benjamin
On 23/08/2010 23:55, Benjamin Peterson wrote:
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not. I'm in both camps though. As we *are* triggering code execution I don't think we should mask exceptions. I just wish we weren't triggering code execution unnecessarily. Michael
2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:
Actually, I'd say given Python's object model, you have to have the attribute in hand to determine that it is present.
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not.
Properties are allowed to raise AttributeError, so you technically have to execute it. -- Regards, Benjamin
On 24/08/2010 00:05, Benjamin Peterson wrote:
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not. If fetching an attribute raises an AttributeError it doesn't mean that attribute doesn't exist (although I admit that at the moment this is exactly what hasattr uses to mean) it just means that fetching that attribute raised an AttributeError. Even if you allow other exceptions to propagate you are still left with the fact that an AttributeError raised as a bug will still be silenced and interpreted as meaning that hasattr should just return False. Michael Foord -- http://www.ironpythoninaction.com/
2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.
I thought you were trying to determine whether the attribute exists not the property.
Raised as a bug? Is this not a valid pattern? @property def myprop(self): if not self.myprop_support: raise AttributeError("myprop") -- Regards, Benjamin
Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.
If you don't want to execute properties, do the lookup on the type, not the instance (obviously, you know the dance you need to do, since you've linked the code where you did it). Having apparently simple query operations like hasattr/getattr/len/etc execute arbitrary code is where much of the language's flexibility comes from, so I don't see how it can really be surprising when it happens. To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError". That means it can't figure out whether or not the attribute exists without actually attempting to retrieve it. There are a few places where we instead use a heuristic that says an attribute *probably* exists if it appears in the instance dictionary, or in the dictionary of one of the types in the MRO, and magic methods have the rule that they must be defined on the type rather than the instance in order to count from the interpreter's point of view, but neither of those things change the basic definition. For the record, +1 on narrowing the scope of the exception suppression in hasattr() to only AttributeError, and adding new C API functions that expose the new behaviour. (I've actually long assumed that AttributeError *was* the only thing suppressed by hasattr()). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24/08/2010 00:40, Nick Coghlan wrote:
Certainly that is true for len. getattr obviously involves invoking code if you are fetching a property or descriptor. No idea how you conclude that hasattr executing code adds flexibility to the language though. Yes I know the dance (walking the mro fetching the attribute out of the appropriate type __dict__ or the instance dict - or looking on the metaclass if the object you are introspecting is a type itself), it is just not trivial - which is why I think it is a shame that people are forced to implement it just to ask if a member exists without triggering code execution.
To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError".
Right, and to me Python's object model (the lookup rules hinted at above) define whether or not an object "has an attribute" or not. We just disagree on this. However, it is irrelevant so not really worth continuing the discussion. Changing hasattr in this way would be backwards incompatible and so cannot be done. Doesn't mean I have to like it though. :-) Michael
On Tue, Aug 24, 2010 at 7:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Now, it may be worth considering an addition to the inspect module that was basically: def getattr_static(obj, attr): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass (not needing to deal with classic classes simplifies things a bit) So, allowing for the fact that dir() may report attributes that can only be found via dynamic lookup, your get_docstrings example could become something like: def get_docstrings(obj): try: members = dir(obj) except Exception: members = [] for member in members: try: doc = getattr_static(obj, member).__doc__ except AttributeError: doc = None yield member, doc Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Second attempt with a default value parameter and correctly raising AttributeError if not found: _sentinel = object() def getattr_static(obj, attr, default=_sentinel): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass if default is not _sentinel: return default raise AttributeError(attr) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24/08/2010 01:25, Nick Coghlan wrote:
This doesn't correctly handle the case where obj is a type object or obj uses __slots__. If I have time (currently on vacation with only intermittent internet access) I'll provide an update. Michael
Cheers, Nick.
On 24/08/2010 08:40, Michael Foord wrote:
I managed an updated version (with tests) during the flight home last night. I've attached it to issue 9732, along with a discussion of the caveats and ways of breaking it: http://bugs.python.org/issue9732 Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 2010-08-23, at 5:02 PM, Michael Foord wrote:
As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour. - Yury
On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour.
You nailed it. At the lowest level (either in C or in Python) there is no way to check for an attribute's presence without getting its value. This has so far been fundamental, and everything else is just appearances. I propose that this is good enough and that we should at this point not try to invent another protocol, just decide whether hasattr() can be fixed (and leaving PyObject_HasAttr*() alone). -- --Guido van Rossum (python.org/~guido)
BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general. The more I think about it the more I like the idea. By default, 'hasattr' would check MRO for the attribute, if not found - check for __hasattr__, if not found - fallback to the current schema with 'getattr'. This would not break any current code, but open a lot of potential (especially for things that implement some lazy loading protocols). From the performance point of view, I believe it shouldn't change much, as current 'hasattr' which uses 'getattr' which calls __getattr(ibute)__ methods. I know this is much more then just fixing exception swallowing, but this is somewhat required functionality to complete the current dynamism python offers. - Yury
By default, if you don't overload __hasattr__, it will do exactly that. But let's imagine the situation in some ORM. Sometimes, engine knows that attribute exists even before executing it (which leads to querying DB), and thus it is possible to return True in the overloaded __hasattr__. Example: ... if hasattr(entity, 'title'): # <- __hasattr__ has the info in metadata, # no query to DB ... title = entity.title # <- now, we can query the DB So, the proposed magic method is not intended to change the protocol, but to complement and enhance it. - Yury
2010/8/23 Yury Selivanov <yselivanov@gmail.com>:
In this case, I think it would make more sense to ask the model: type(entity).has_property("title")
So, the proposed magic method is not intended to change the protocol, but to complement and enhance it.
But it still raises the potential to break the relationship between hasattr and getattr. I think this should require a PEP. -- Regards, Benjamin
On Tue, Aug 24, 2010 at 8:25 AM, Benjamin Peterson <benjamin@python.org> wrote:
Definitely needs a PEP, and will require some solid use cases to explain why allowing optimisation of hasattr() is the right way to go. Complexity isn't free, and I doubt the gains here will justify the costs, but that's one of the things the PEP process is intended to figure out. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
At 06:12 PM 8/23/2010 -0400, Yury Selivanov wrote:
BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general.
In order to do this properly, you'd need to also add __has__ or __exists__ (or some such) to the descriptor protocol; otherwise you break descriptors' ability to operate independently of the class they're used in. You would probably also need a __hasattribute__, in order to be able to properly synchronize with __getattr__/__getattribute__. Seems like overkill to me, though, as I'm not sure how such a protocol actually helps ORM or persistence schemes (and I've written a few). Pretty much, if you're trying to check for the existence of an attribute, you're probably about to be getting that attribute anyway. (i.e. why query the existence of an attribute you *don't* intend to use?)
On Mon, Aug 23, 2010 at 3:45 PM, P.J. Eby <pje@telecommunity.com> wrote:
Right. This sounds like way too big a gun to kill this particular mosquito. If Yury wants to write a PEP I won't stop him, but I expect that it will be rejected for want of important use cases compared to the complexity of the solution. There just are too many places that would be affected, for too little value. So just be warned. OTOH I still think that fixing hasattr() to be mroe like getattr(obj, key, None) has a high value and a relative low risk. -- --Guido van Rossum (python.org/~guido)
At 12:02 AM 8/24/2010 +0300, Michael Foord wrote:
For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not.
That depends on what you mean by "exists". Note that a property might raise AttributeError to signal that the attribute is not currently set. Likewise, unless you special case __slots__ descriptors, you can have the bizarre condition where hasattr() will return True, but getattr() will still raise an AttributeError. The idea that you could determine the presence of an attribute on an object without executing that object's code is something that hasn't been practical since the birth of descriptors in Python 2.2.
Even if you implement it, you will get wrong answers in some cases. __getattribute__ is allowed to throw out the entire algorithm you just described and replace it utterly with something else. My ProxyTypes library makes use of that fact, for example, so if you actually attempted to inspect a proxy instance with your re-implemented "dance", your code will fail to notice what attributes the proxy actually has.
On 23/08/2010 23:13, Benjamin Peterson wrote:
Well yes. One of the reasons a full workaround is so arcane is that if you *really* don't want to trigger code execution you can't call dir... (Otherwise name in dir(obj) is a reasonable approximation for hasattr(obj, name).) Michael -- http://www.ironpythoninaction.com/
On 2010-08-23, at 3:47 PM, Raymond Hettinger wrote:
This is impossible to implement because of '__getattribute__' and '__getattr__' methods. There is no way of detecting the presence of an attribute but trying to get it through 'getattr'. Partial solution like getting information about property presence through __dict__ and call __getattribute__/__getattr__ if they are defined wouldn't work either. So, your solution would make 'hasattr' even more incompatible. - Yury Selivanov
On Mon, Aug 23, 2010 at 12:47 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Hm... That sounds like scope creep to me. Properties are supposed to be cheap and idempotent. Trying to figure out whether an attribute exists without using __getattr__ is fraught with problems -- as soon as a class overrides __getattr__ or __getattribute__ you're hosed anyway. I can vouch that the reason hasattr() catches too many exceptions is that when I first added it (around 1990, I think :-) I wasn't very attuned yet to the problems it could cause. The main problem I can see with letting exceptions other than AttributeError bubble through (besides perverted dependencies on the current semantics) is that there are some situations where it is pretty arbitrary whether TypeError or AttributeError is raised. I can't recall the details, and possibly this was more of a problem with classic classes, but I do think it warrants some research. -- --Guido van Rossum (python.org/~guido)
2010/8/23 Guido van Rossum <guido@python.org>:
I believe this was with regards to actual operations, though, not fetching the attribute. For example, float(a) would raise an AttributeError for a classic instance and a TypeError for a new-style instance. However both would raise AttributeError on a.__float__. -- Regards, Benjamin
On Aug 23, 2010, at 1:03 PM, Guido van Rossum wrote:
I don't have a specific proposal in mind. My main questions are * Is there anything that hasattr(obj, key) can or should do that can't already be done with getattr(obj, key, None)? If not, do we really need to change anything? * Why do people typically use hasattr() instead getattr()? Aren't they are really trying to just determine whether a key exists somewhere in the MRO? If so, then doing anything more than that is probably a surprise. I know my own uses of hasattr() do not expect any exceptions at all. It comes up in duck typing support different handling for different types of inputs. If others are using it the same way, I think it is unlikely that they have unittests to cover the possibility that hasattr() would ever start raising exceptions.
Fire-up the time machine? Raymond P.S. The current behavior seems to be deeply embedded: int PyObject_HasAttr(PyObject *o, PyObject *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds. int PyObject_HasAttrString(PyObject *o, const char *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds.
2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:
For one, it's annoying when "key" can be None.
It's generally more convenient that getattr(obj, "blah", None) is not None.
P.S. The current behavior seems to be deeply embedded:
Well, that's what happens when a behavior is added in 1990. :) -- Regards, Benjamin
On Mon, Aug 23, 2010 at 1:50 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Changing C APIs is even harder than changing Python API because you can't add exceptions to something that wasn't returning exceptions before. We did that for comparisons in the past and it was a pain (but worth it). For these two little APIs I think it is not worth it, though it may be worth it to create new APIs that *do* return exceptions. -- --Guido van Rossum (python.org/~guido)
On Mon, Aug 23, 2010 at 1:33 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
I don't have a specific proposal in mind.
That's why I called it scope creep. :-) Trust me, your proposal will not lead to a quick and better replacement for hasattr(). (See several other people's replies.)
getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly. * Why do people typically use hasattr() instead getattr()?
Most users who call hasattr() probably don't even know what MRO means. They call hasattr() because they want to avoid a try/except clause. The reasons they are not calling getattr(obj, key, None) could be many: never heard of it, too obscure (it surely doesn't spell "has the attribute" like hasattr() does to the beginning user), or (probably common) the actual attribute access is in some other piece of code they are about to call but don't control.
It already raises *some* exceptions (those derived from BaseException but not Exception). I think that change was a definite non-event.
But note that hasattr() doesn't call those.
-- --Guido van Rossum (python.org/~guido)
On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:
Why do you say it's ugly? It's a short, sweet, simple two-liner: mark = object() getattr(obj, 'key', mark) is not mark Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function. It still doesn't cope with dynamically-generated attributes that are either expensive or have side-effects (both of which are probably poor design, but nevertheless I'm sure they're out there), but neither does the existing hasattr.
Well, yes, but most users never write __getattr__ or __getattribute__ methods either. I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__. Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist. -- Steven D'Aprano
On Mon, Aug 23, 2010 at 17:04, Benjamin Peterson <benjamin@python.org> wrote:
It is also non-obvious to any beginner. Are we really going to want to propagate the knowledge of this trick as a fundamental idiom? I would rather leave hasattr in that instance. But I'm +1 on only swallowing AttributeError. -Brett
On Mon, Aug 23, 2010 at 4:56 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Because if you didn't have the foresight to write that utility function, you have to break your train of thought (aka yak shaving) to either write it (big yak) or write those two lines (little yak, times the number of times it happens). Whereas with hasattr() you can just type the correct hasattr() expression in-line in the if that you already started typing.
It tests for the existence of an attribute -- how the attribute is defined should not have to occur to you (and there are lots of other ways for attributes to be defined besides the ways you came up with just now).
The try/except block *also* requires you to break your train of thought. And most of the time the error case just isn't important. You sound like you are over-engineering it and focusing too much on performance instead of on getting things done. Like those people who learn that it saves an usec to copy a built-in function into a defalt arg (def foo(arg1, len=len): ...) and then overuse the trick even when the time it took them to write the exra line is more than the time they'll save in a lifetime in execution time. -- --Guido van Rossum (python.org/~guido)
On Tue, 24 Aug 2010 11:09:10 am Guido van Rossum wrote:
But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist. If it existed, __getattr__ would never be called. A minor semantic difference, to be sure, but it's real to me. Whether I should care about the difference is a separate issue. This conversation has been valuable to me for one thing though... it reminded me of a piece of code I had written a long time ago. A simplified version: class K(object): def __getattribute__(self, name): if hasattr(self, name): # no computation needed print "Attr exists" else: # compute something... print "Attr doesn't exist" I couldn't work out why it was behaving so strangely, and rather than spend time investigating, I abandoned the whole dynamic attribute approach and did something completely different. So at least now I know why it wasn't working as I expected.
I never suggested that it would be easy.
Performance could be an issue, of course, if somebody writes an expensive __getattr__ or property. Computed attributes should be cheap. But I'm actually more concerned about side-effects than performance. If I'm not worried about potential side-effects, I use a try...except block. If I am worried, I "Look Before You Leap". Only now I've learned that what I thought was LBYL is nothing of the sort, and hasattr gives me no protection against side-effects. That being the case, I might as well just stick to try...except and be done with it. I'm not suggesting this is the One True Way. If others prefer hasattr, then I have no problem with that. I'm just saying that now that I know what it actually does, its value *for me* is minimal.
Aha! The penny drops! Is that why so many methods in random.Random have an "int=int" argument? E.g. def randrange(self, start, stop=None, step=1, int=int, default=None, maxwidth=1L<<BPF): I'd wondered about that. So there you go, now I've learned two things. -- Steven D'Aprano
On Tue, Aug 24, 2010 at 4:51 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Eh? If "x.y" succeeds, in what sense does y not exist?
Whether I should care about the difference is a separate issue.
Right, you are breaking through too much abstraction.
Performance could be an issue, of course, if somebody writes an expensive __getattr__ or property. Computed attributes should be cheap.
Yes, and it should be considered the problem of the author of that property, not of the user.
But I'm actually more concerned about side-effects than performance.
Properties should not have side effects. That would be a problem when using them just as much as with hasattr(). -- --Guido van Rossum (python.org/~guido)
Steven D'Aprano wrote:
But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist.
Maybe for your particular use case, but the concept of whether an attribute is dynamically defined or not is not well-defined in general. Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want. -- Greg
At 12:10 PM 8/25/2010 +1200, Greg Ewing wrote:
Yep. That's why the proposed addition to inspect is a bad idea. If we encourage that sort of static thinking, it will lead to people creating all sorts of breakage with respect to more dynamic code. AFAICT, the whole "avoid running code" thing only makes sense for a debugging tool -- at which point, you can always use the trace facility and throw an error when any Python code runs that's not part of your debugging tool. Something like: def exists(ob, attr): __running__ = True # ... set trace function here try: try: getattr(ob, attr) return True except AttributeError: return False except CodeRanError: return True # or False if you prefer finally: __running__ = False # restore old tracing here Where the trace function is just something that throws CodeRanError if it detects a "call" event and the __running__ flag is True. This would stop any Python code from actually executing. (It'd need to keep the same trace function for c_call events, since that might lead to nested non-C calls .) Of course, a debugger's object inspection tool would probably actually want to return either the attribute value, or a special value to mean "dyanmic calculation needed".
On 25/08/2010 19:27, P.J. Eby wrote:
I mentioned another use case - pulling out docstrings. IDEs or other tools that work with live objects may have many such use cases. For proxying objects you can't use the __getattr__ approach for the "magic methods" which aren't looked up through the dynamic attribute 'faking' process. Several proxy libraries I've seen get round this by providing *every* magic method on the proxy class and delegating. This still breaks certain types of duck typing. For example with Python 2.6:
If your proxy class defines __call__ then callable returns True, even if the delegation to the proxied object would cause an AttributeError to be raised. A *better* approach (IMO), for both the magic methods and the normal attributes / methods, is to dynamically generate a class that mimics the proxied object (caching generated classes for proxying objects of the same type if you are worried about overhead). This would also work with the suggested "passive introspection" function. All the best, Michael Foord
At 08:58 PM 8/25/2010 +0300, Michael Foord wrote:
Nope. You just have to use delegate via __getattribute__ (since 2.2) instead of __getattr__:
As you can see, the __call__ attribute in each case is whatever the proxied object's __call__ attribute is, even though the proxy itself has a __call__ method, that is invoked when the proxy is called. This is actually pretty straightforward stuff since the introduction of __getattribute__. (The code is at http://pypi.python.org/pypi/ProxyTypes, btw.)
On 25 August 2010 20:57, P.J. Eby <pje@telecommunity.com> wrote:
For what it's worth this code is useful enough (and simple enough) that I would support its inclusion in the standard library. I've written proxy objects several times (for various different purposes) and this would have saved me the effort. :-) All the best, Michael Foord -- http://www.voidspace.org.uk
On 2010-08-23, at 4:33 PM, Raymond Hettinger wrote:
OK, if my key has None value, how will you detect it with 'getattr'? ... marker = object() ... if (getattr(obj, key, marker) is not marker: Like in the above? Too much code. Ugly code. Just to hide the issue under the carpet.
My example in the initial email clearly showed that sometimes, key is in MRO, but 'hasattr' returns False. For everything in any complicated system (which Python obviously is) should be a strict protocol. In case of properties this protocol is to raise AttributeError if property is missing. Everything else is a potential bug. Current 'hasattr' just masks them. - Yury
participants (18)
-
Barry Warsaw -
Benjamin Peterson -
Brett Cannon -
Glyph Lefkowitz -
Greg Ewing -
Guido van Rossum -
Hrvoje Niksic -
James Y Knight -
Michael Foord -
Nick Coghlan -
P.J. Eby -
R. David Murray -
Raymond Hettinger -
Steve Holden -
Steven D'Aprano -
Terry Reedy -
Tres Seaver -
Yury Selivanov