On 2010-08-23, at 10:46 AM, Benjamin Peterson wrote:

2010/8/23 Yury Selivanov <yselivanov@gmail.com>:

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I would be in support of that.

...

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

But not this. Compatibility functions don't belong in 2to3.

There are many possible solutions for the Python 2 porting issue, this one was just one of them. I was trying to make a point, that it's possible to somehow make porting process easier, and meanwhile fix Python 3. - Yury Selivanov Sprymix Inc.

2010/8/23 Guido van Rossum <guido@python.org>:

On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson <benjamin@python.org> wrote:

...

2010/8/23 Yury Selivanov <yselivanov@gmail.com>:

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr').  Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.

The test suite passes complete without modification for me. -- Regards, Benjamin

On 2010-08-23, at 10:56 AM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson <benjamin@python.org> wrote:

...

2010/8/23 Yury Selivanov <yselivanov@gmail.com>:

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.

Have done a little testing. The patch to builtin 'hasattr' function is trivial, and can be found attached to this letter (with the corresponding unittest): - if (!PyErr_ExceptionMatches(PyExc_Exception)) + if (!PyErr_ExceptionMatches(PyExc_AttributeError)) So, after applying it, hasattr swallows only AttributeError exceptions. All tests from Python 3.2 test suite were good. After that, I've applied the patch on Python 2.6 and tested SqlAlchemy, Django and Twisted. - SqlAlchemy has failed three unittests, but those tests were designed specifically to handle 'hasattr' weird behaviour, so we can consider the change has no impact on SqlAlchemy. - Twisted - failed 3 tests out of ~3200, but it fails them on the same machine on an unpatched Python too, and they seem unrelated. - Django - all tests passed. I tested our internal company framework (~100K LOC) and, again, all tests passed.

On 2010-08-23, at 10:37 PM, Guido van Rossum wrote: Yuri, I think you are making a good case (though I would like for you to be a good citizen and use the bug tracker to submit this for review). Benjamin, what do you think?

NP, issue #9666 ;-) - Yury

On 08/23/2010 04:56 PM, Guido van Rossum wrote:

On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson<benjamin@python.org> wrote:

...

2010/8/23 Yury Selivanov<yselivanov@gmail.com>:

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I'll note that a similar incompatible change has made it to python2.6. This has bitten us in production: class X(object): def __getattr__(self, name): raise KeyError, "error looking for %s" % (name,) def __iter__(self): yield 1 print list(X()) I would expect it to print [1], and in python2.5 it does. In python2.6 it raises a KeyError! The attribute being looked up is an unexpected one: {hrzagude5003}[~]$ python2.6 a.py Traceback (most recent call last): File "a.py", line 8, in <module> print list(X()) File "a.py", line 3, in __getattr__ raise KeyError, "error looking for %s" % (name,) KeyError: 'error looking for __length_hint__' The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list(). Being aware that this can be construed as an argument both in favor and against the change at hand, my point is that, if propagating non-AttributeError exceptions is done in checks intended to be invisible, it should certainly be done in hasattr, where it is at least obvious what is being done. Other generic functions and operators, including boolean ones such as ==, happily propagate exceptions. Also, don't expect that this won't break code out there. It certainly will, it's only a matter of assessment whether such code was broken in a different, harder to detect way, to begin with.

On 08/24/2010 02:31 PM, Benjamin Peterson wrote:

2010/8/24 Hrvoje Niksic<hrvoje.niksic@avl.com>:

...

The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list().

__length_hint__ is internal and undocumented, so it can do whatever it wants.

Of course, but that's beside the point. In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail). The point I'm making is that: a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and b) making the proposed change is bound to break real, production code. I still agree with the proposed change, but I wanted to also point out that it will cause breakage and illustrate it with a similar real-world example that occurred during migration to python 2.6.

On 8/24/2010 9:45 AM, Benjamin Peterson wrote:

2010/8/24 Hrvoje Niksic <hrvoje.niksic@avl.com>:

...

On 08/24/2010 02:31 PM, Benjamin Peterson wrote:

...

2010/8/24 Hrvoje Niksic<hrvoje.niksic@avl.com>:

...

The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list().

__length_hint__ is internal and undocumented, so it can do whatever it wants.

Of course, but that's beside the point. In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail). The point I'm making is that:

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

b) making the proposed change is bound to break real, production code.

I agree with. This is why the change is not making its way into any maintenance release.

+1 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/

2010/8/24 P.J. Eby <pje@telecommunity.com>:

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug. -- Regards, Benjamin

2010/8/24 James Y Knight <foom@fuhm.net>:

On Aug 24, 2010, at 10:26 AM, Benjamin Peterson wrote:

...

2010/8/24 P.J. Eby <pje@telecommunity.com>:

...

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug.

Uh, did you see the message that was in response to?

Maybe it should be a bug report?

Old version of Python I think. -- Regards, Benjamin

2010/8/24 R. David Murray <rdmurray@bitdance.com>:

On Tue, 24 Aug 2010 09:26:09 -0500, Benjamin Peterson <benjamin@python.org> wrote:

...

2010/8/24 P.J. Eby <pje@telecommunity.com>:

...

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug.

Benjamin, I remember you fixing various special method lookups, so just for clarity's sake, which versions of Python does your statement apply to?

2.7, 3.1, and 3.2 should all be good. -- Regards, Benjamin

On 23/08/2010 22:47, Raymond Hettinger wrote:

On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:

...

I know the issue has been discussed several times already, however I couldn't find any reasonable explanation of its strange behaviour. The main problem with 'hasattr' function is that is swallows all exceptions derived from Exception class. It's a good thing that it doesn't do that with BaseException as it was fixed not a long time ago, but it's definitely not enough.

First of all, this behaviour of 'hasattr' contradicts with the very core principle of python: "Errors should never pass silently." And since 'hasattr' function is in builtins module and is a widely used function it impacts the whole language.

Secondly, take a look at the following:

...

...

...

class Test: ... @property ... def attr(self): ... self['foo'] ... hasattr(Test(), 'attr') False

There can be any exception instead of KeyError in the above snippet of code, but this small example shows how 'hasattr': misleadingly breaks the code logic (1) and masks bug (2). And that's the simplest possible example, there are much more in real life.

While (1) is maybe acceptable for someone, there is no excuse for the (2). Moreover, current 'hasattr' behaviour tremendously complicates use of '__getattribute__' magic. And forget about importlib magic with LazyImports, one 'hasattr' ruins everything by catching ImportError.

To conclude:

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

Thanks for the nice analysis and good example.

I disagree with the solution though. If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change.

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though. For what it's worth I *agree* with you [1], but for better or worse hasattr / getattr trigger code execution and hasattr can return True for dynamically created members. Something to be revisited for Python 4 perhaps. Michael Foord [1] A while ago I wrote a couple of blog entries on fetching docstrings from members without triggering code execution. It is surprisingly convoluted and even my final code had corner cases it didn't handle: http://www.voidspace.org.uk/python/weblog/arch_d7_2009_05_16.shtml#e1090 http://www.voidspace.org.uk/python/weblog/arch_d7_2009_06_20.shtml#e1103

Raymond

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/

2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:

Thanks for the nice analysis and good example.

I disagree with the solution though.  If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change.

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem.  The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

That would break the assumption that: if hasattr(obj, attr): getattr(obj, attr) # won't raise and hasattr ~= try: getattr(obj, attr) except AttributeError: return False else: return True

As your example shows, property() defeats this intent by actually executing the code.   A better behavior would not run the code at all.  It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

That doesn't sound to useful to me. A descriptor could be found with __get__, but that __get__ could just as well raise AttributeError.

IMO, this is a much better solution, more in line with known use cases for hasattr().   If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

Can you provide an example? I've never seen code which explicitly scans MRO and dicts to avoid triggering code. (Besides collections.Callable; that's a special case.) -- Regards, Benjamin

2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

That's the danger of a dynamic language like Python. Even dir() can now trigger things like that. -- Regards, Benjamin

2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

hasattr(x, "y") doesn't look any more passive to me the x.y.

Well said.  The surprise to me in the OP's example was that the property() was executed.  Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr().

Hard to remember compared to what?

That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods).

Better to raise an exception into unexpecting code that to have subtly different lookup rules between getattr and hasattr. -- Regards, Benjamin

2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:

 On 23/08/2010 23:55, Benjamin Peterson wrote:

...

2010/8/23 Raymond Hettinger<raymond.hettinger@gmail.com>:

...

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord<fuzzyman@voidspace.org.uk>:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

hasattr(x, "y") doesn't look any more passive to me the x.y.

One says "does this object have attribute y" the other fetches attribute y. I'm amazed they don't look different to you. Given Python's object model there is no reason that the first *should* fetch the attribute to determine that it is present, *except* for the dynamic attribute creation of __getattr__ and __getattribute__.

Actually, I'd say given Python's object model, you have to have the attribute in hand to determine that it is present.

For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not.

Properties are allowed to raise AttributeError, so you technically have to execute it. -- Regards, Benjamin

2010/8/23 Michael Foord <fuzzyman@voidspace.org.uk>:

Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.

I thought you were trying to determine whether the attribute exists not the property.

If fetching an attribute raises an AttributeError it doesn't mean that attribute doesn't exist (although I admit that at the moment this is exactly what hasattr uses to mean) it just means that fetching that attribute raised an AttributeError. Even if you allow other exceptions to propagate you are still left with the fact that an AttributeError raised as a bug will still be silenced and interpreted as meaning that hasattr should just return False.

Raised as a bug? Is this not a valid pattern? @property def myprop(self): if not self.myprop_support: raise AttributeError("myprop") -- Regards, Benjamin

On 24/08/2010 00:40, Nick Coghlan wrote:

...

Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not. If you don't want to execute properties, do the lookup on the type, not the instance (obviously, you know the dance you need to do, since you've linked the code where you did it). Having apparently simple query operations like hasattr/getattr/len/etc execute arbitrary code is where much of the language's flexibility comes from, so I don't see how it can really be surprising when it happens.

Certainly that is true for len. getattr obviously involves invoking code if you are fetching a property or descriptor. No idea how you conclude that hasattr executing code adds flexibility to the language though. Yes I know the dance (walking the mro fetching the attribute out of the appropriate type __dict__ or the instance dict - or looking on the metaclass if the object you are introspecting is a type itself), it is just not trivial - which is why I think it is a shame that people are forced to implement it just to ask if a member exists without triggering code execution.

To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError".

Right, and to me Python's object model (the lookup rules hinted at above) define whether or not an object "has an attribute" or not. We just disagree on this. However, it is irrelevant so not really worth continuing the discussion. Changing hasattr in this way would be backwards incompatible and so cannot be done. Doesn't mean I have to like it though. :-) Michael

That means it can't figure out whether or not the attribute exists without actually attempting to retrieve it. There are a few places where we instead use a heuristic that says an attribute *probably* exists if it appears in the instance dictionary, or in the dictionary of one of the types in the MRO, and magic methods have the rule that they must be defined on the type rather than the instance in order to count from the interpreter's point of view, but neither of those things change the basic definition.

For the record, +1 on narrowing the scope of the exception suppression in hasattr() to only AttributeError, and adding new C API functions that expose the new behaviour. (I've actually long assumed that AttributeError *was* the only thing suppressed by hasattr()).

Cheers, Nick.

-- http://www.ironpythoninaction.com/

On Tue, Aug 24, 2010 at 7:49 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

Certainly that is true for len. getattr obviously involves invoking code if you are fetching a property or descriptor. No idea how you conclude that hasattr executing code adds flexibility to the language though.

Proxy objects like the one in the weakref module don't work if hasattr() doesn't implicitly invoke __getattr__. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

Now, it may be worth considering an addition to the inspect module that was basically:

def getattr_static(obj, attr):    """Retrieve attributes without triggering dynamic lookup via the descriptor protocol,        __getattr__ or __getattribute__.

       Note: this function may not be able to retrieve all attributes reported by dir(obj)    """    try:        instance_dict = object.__getattribute__(obj, "__dict__")    except AttributeError:        pass    else:        if attr in instance_dict:            return instance_dict[attr]    for entry in getmro(obj.__class__):        try:            return entry.__dict__[attr]        except AttributeError:            pass

Second attempt with a default value parameter and correctly raising AttributeError if not found: _sentinel = object() def getattr_static(obj, attr, default=_sentinel): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__. Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass if default is not _sentinel: return default raise AttributeError(attr) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 24/08/2010 08:40, Michael Foord wrote:

On 24/08/2010 01:25, Nick Coghlan wrote:

...

On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan<ncoghlan@gmail.com> wrote:

...

Now, it may be worth considering an addition to the inspect module that was basically:

def getattr_static(obj, attr): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__.

Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass Second attempt with a default value parameter and correctly raising AttributeError if not found:

_sentinel = object() def getattr_static(obj, attr, default=_sentinel): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__.

Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass if default is not _sentinel: return default raise AttributeError(attr)

This doesn't correctly handle the case where obj is a type object or obj uses __slots__.

If I have time (currently on vacation with only intermittent internet access) I'll provide an update.

I managed an updated version (with tests) during the flight home last night. I've attached it to issue 9732, along with a discussion of the caveats and ways of breaking it: http://bugs.python.org/issue9732 Michael

Michael

...

Cheers, Nick.

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 2010-08-23, at 5:22 PM, Yury Selivanov wrote:

On 2010-08-23, at 5:02 PM, Michael Foord wrote:

...

On 23/08/2010 23:55, Benjamin Peterson wrote:

...

2010/8/23 Raymond Hettinger<raymond.hettinger@gmail.com>:

...

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord<fuzzyman@voidspace.org.uk>:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. hasattr(x, "y") doesn't look any more passive to me the x.y. One says "does this object have attribute y" the other fetches attribute y. I'm amazed they don't look different to you. Given Python's object model there is no reason that the first *should* fetch the attribute to determine that it is present, *except* for the dynamic attribute creation of __getattr__ and __getattribute__.

As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour.

I'm not sure we should do this, but if such method exists, let's call it __hasattribute__, ORMs, for instance, probably would benefit. - Yury

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov <yselivanov@gmail.com> wrote:

...

As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour.

You nailed it. At the lowest level (either in C or in Python) there is no way to check for an attribute's presence without getting its value. This has so far been fundamental, and everything else is just appearances.

I propose that this is good enough and that we should at this point not try to invent another protocol, just decide whether hasattr() can be fixed (and leaving PyObject_HasAttr*() alone).

BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general. The more I think about it the more I like the idea. By default, 'hasattr' would check MRO for the attribute, if not found - check for __hasattr__, if not found - fallback to the current schema with 'getattr'. This would not break any current code, but open a lot of potential (especially for things that implement some lazy loading protocols). From the performance point of view, I believe it shouldn't change much, as current 'hasattr' which uses 'getattr' which calls __getattr(ibute)__ methods. I know this is much more then just fixing exception swallowing, but this is somewhat required functionality to complete the current dynamism python offers. - Yury

On 2010-08-23, at 6:17 PM, Benjamin Peterson wrote: 2010/8/23 Yury Selivanov <yselivanov@gmail.com>:

...

...

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov <yselivanov@gmail.com> wrote: BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general.

-1 The whole point of hasattr() is that getattr() on that attribute would return something. Let's not destroy that.

By default, if you don't overload __hasattr__, it will do exactly that. But let's imagine the situation in some ORM. Sometimes, engine knows that attribute exists even before executing it (which leads to querying DB), and thus it is possible to return True in the overloaded __hasattr__. Example: ... if hasattr(entity, 'title'): # <- __hasattr__ has the info in metadata, # no query to DB ... title = entity.title # <- now, we can query the DB So, the proposed magic method is not intended to change the protocol, but to complement and enhance it. - Yury

On Tue, Aug 24, 2010 at 8:25 AM, Benjamin Peterson <benjamin@python.org> wrote:

2010/8/23 Yury Selivanov <yselivanov@gmail.com>:

...

So, the proposed magic method is not intended to change the protocol, but to complement and enhance it.

But it still raises the potential to break the relationship between hasattr and getattr.

I think this should require a PEP.

Definitely needs a PEP, and will require some solid use cases to explain why allowing optimisation of hasattr() is the right way to go. Complexity isn't free, and I doubt the gains here will justify the costs, but that's one of the things the PEP process is intended to figure out. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Mon, Aug 23, 2010 at 3:45 PM, P.J. Eby <pje@telecommunity.com> wrote:

At 06:12 PM 8/23/2010 -0400, Yury Selivanov wrote:

...

BTW, is it possible to add new magic method __hasattr__?  Maybe not in Python 3.2, but in general.

In order to do this properly, you'd need to also add __has__ or __exists__ (or some such) to the descriptor protocol; otherwise you break descriptors' ability to operate independently of the class they're used in.  You would probably also need a __hasattribute__, in order to be able to properly synchronize with __getattr__/__getattribute__.

Seems like overkill to me, though, as I'm not sure how such a protocol actually helps ORM or persistence schemes (and I've written a few).  Pretty much, if you're trying to check for the existence of an attribute, you're probably about to be getting that attribute anyway.  (i.e. why query the existence of an attribute you *don't* intend to use?)

Right. This sounds like way too big a gun to kill this particular mosquito. If Yury wants to write a PEP I won't stop him, but I expect that it will be rejected for want of important use cases compared to the complexity of the solution. There just are too many places that would be affected, for too little value. So just be warned. OTOH I still think that fixing hasattr() to be mroe like getattr(obj, key, None) has a high value and a relative low risk. -- --Guido van Rossum (python.org/~guido)

On 23/08/2010 23:13, Benjamin Peterson wrote:

2010/8/23 Michael Foord<fuzzyman@voidspace.org.uk>:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. That's the danger of a dynamic language like Python. Even dir() can now trigger things like that.

Well yes. One of the reasons a full workaround is so arcane is that if you *really* don't want to trigger code execution you can't call dir... (Otherwise name in dir(obj) is a reasonable approximation for hasattr(obj, name).) Michael -- http://www.ironpythoninaction.com/

On 2010-08-23, at 3:47 PM, Raymond Hettinger wrote:

On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:

...

I know the issue has been discussed several times already, however I couldn't find any reasonable explanation of its strange behaviour. The main problem with 'hasattr' function is that is swallows all exceptions derived from Exception class. It's a good thing that it doesn't do that with BaseException as it was fixed not a long time ago, but it's definitely not enough.

First of all, this behaviour of 'hasattr' contradicts with the very core principle of python: "Errors should never pass silently." And since 'hasattr' function is in builtins module and is a widely used function it impacts the whole language.

Secondly, take a look at the following:

...

...

...

class Test: ... @property ... def attr(self): ... self['foo'] ... hasattr(Test(), 'attr') False

There can be any exception instead of KeyError in the above snippet of code, but this small example shows how 'hasattr': misleadingly breaks the code logic (1) and masks bug (2). And that's the simplest possible example, there are much more in real life.

While (1) is maybe acceptable for someone, there is no excuse for the (2). Moreover, current 'hasattr' behaviour tremendously complicates use of '__getattribute__' magic. And forget about importlib magic with LazyImports, one 'hasattr' ruins everything by catching ImportError.

To conclude:

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

Thanks for the nice analysis and good example.

I disagree with the solution though. If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change.

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

This is impossible to implement because of '__getattribute__' and '__getattr__' methods. There is no way of detecting the presence of an attribute but trying to get it through 'getattr'. Partial solution like getting information about property presence through __dict__ and call __getattribute__/__getattr__ if they are defined wouldn't work either. So, your solution would make 'hasattr' even more incompatible. - Yury Selivanov

2010/8/23 Guido van Rossum <guido@python.org>:

The main problem I can see with letting exceptions other than AttributeError bubble through (besides perverted dependencies on the current semantics) is that there are some situations where it is pretty arbitrary whether TypeError or AttributeError is raised. I can't recall the details, and possibly this was more of a problem with classic classes, but I do think it warrants some research.

I believe this was with regards to actual operations, though, not fetching the attribute. For example, float(a) would raise an AttributeError for a classic instance and a TypeError for a new-style instance. However both would raise AttributeError on a.__float__. -- Regards, Benjamin

2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:

I don't have a specific proposal in mind.  My main questions are * Is there anything that hasattr(obj, key) can or should do that   can't already be done with getattr(obj, key, None)?   If not, do we really need to change anything?

For one, it's annoying when "key" can be None.

* Why do people typically use hasattr() instead getattr()?    Aren't they are really trying to just determine    whether a key exists somewhere in the MRO?    If so, then doing anything more than that is probably a surprise.

It's generally more convenient that getattr(obj, "blah", None) is not None.

P.S.  The current behavior seems to be deeply embedded:

Well, that's what happens when a behavior is added in 1990. :) -- Regards, Benjamin

On Mon, Aug 23, 2010 at 1:50 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:

On Aug 23, 2010, at 1:45 PM, Benjamin Peterson wrote:

...

2010/8/23 Raymond Hettinger <raymond.hettinger@gmail.com>:

...

P.S.  The current behavior seems to be deeply embedded:

Well, that's what happens when a behavior is added in 1990. :)

More generally:  it is an API code smell whenever documentation promises something like "this always succeeds"     ;-)

Changing C APIs is even harder than changing Python API because you can't add exceptions to something that wasn't returning exceptions before. We did that for comparisons in the past and it was a pain (but worth it). For these two little APIs I think it is not worth it, though it may be worth it to create new APIs that *do* return exceptions. -- --Guido van Rossum (python.org/~guido)

On Aug 23, 2010, at 03:45 PM, Benjamin Peterson wrote:

It's generally more convenient that getattr(obj, "blah", None) is not None.

Clearly, the solution is a new builtin called 'Missing'. <wink> -Barry

On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:

...

* Is there anything that hasattr(obj, key) can or should do that can't already be done with getattr(obj, key, None)? If not, do we really need to change anything?

getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly.

Why do you say it's ugly? It's a short, sweet, simple two-liner: mark = object() getattr(obj, 'key', mark) is not mark Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function. It still doesn't cope with dynamically-generated attributes that are either expensive or have side-effects (both of which are probably poor design, but nevertheless I'm sure they're out there), but neither does the existing hasattr.

* Why do people typically use hasattr() instead getattr()?

...

Aren't they are really trying to just determine whether a key exists somewhere in the MRO? If so, then doing anything more than that is probably a surprise.

Most users who call hasattr() probably don't even know what MRO means.

Well, yes, but most users never write __getattr__ or __getattribute__ methods either. I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__. Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist. -- Steven D'Aprano

On Mon, Aug 23, 2010 at 17:04, Benjamin Peterson <benjamin@python.org> wrote:

2010/8/23 Steven D'Aprano <steve@pearwood.info>:

...

On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:

...

...

* Is there anything that hasattr(obj, key) can or should do that   can't already be done with getattr(obj, key, None)?   If not, do we really need to change anything?

getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly.

Why do you say it's ugly? It's a short, sweet, simple two-liner:

mark = object() getattr(obj, 'key', mark) is not mark

Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function.

That's not all, though. You still have to test it with an "if" and that gets cumbersome after a while.

It is also non-obvious to any beginner. Are we really going to want to propagate the knowledge of this trick as a fundamental idiom? I would rather leave hasattr in that instance. But I'm +1 on only swallowing AttributeError. -Brett

...

I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__.

Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist.

Well, that's exactly what hasattr does...

-- Regards, Benjamin _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On Tue, 24 Aug 2010 11:09:10 am Guido van Rossum wrote:

On Mon, Aug 23, 2010 at 4:56 PM, Steven D'Aprano <steve@pearwood.info> wrote: [...]

...

I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__.

It tests for the existence of an attribute -- how the attribute is defined should not have to occur to you

But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist. If it existed, __getattr__ would never be called. A minor semantic difference, to be sure, but it's real to me. Whether I should care about the difference is a separate issue. This conversation has been valuable to me for one thing though... it reminded me of a piece of code I had written a long time ago. A simplified version: class K(object): def __getattribute__(self, name): if hasattr(self, name): # no computation needed print "Attr exists" else: # compute something... print "Attr doesn't exist" I couldn't work out why it was behaving so strangely, and rather than spend time investigating, I abandoned the whole dynamic attribute approach and did something completely different. So at least now I know why it wasn't working as I expected.

(and there are lots of other ways for attributes to be defined besides the ways you came up with just now).

I never suggested that it would be easy.

...

Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist.

The try/except block *also* requires you to break your train of thought. And most of the time the error case just isn't important. You sound like you are over-engineering it and focusing too much on performance instead of on getting things done.

Performance could be an issue, of course, if somebody writes an expensive __getattr__ or property. Computed attributes should be cheap. But I'm actually more concerned about side-effects than performance. If I'm not worried about potential side-effects, I use a try...except block. If I am worried, I "Look Before You Leap". Only now I've learned that what I thought was LBYL is nothing of the sort, and hasattr gives me no protection against side-effects. That being the case, I might as well just stick to try...except and be done with it. I'm not suggesting this is the One True Way. If others prefer hasattr, then I have no problem with that. I'm just saying that now that I know what it actually does, its value *for me* is minimal.

Like those people who learn that it saves an usec to copy a built-in function into a defalt arg (def foo(arg1, len=len): ...) and then overuse the trick even when the time it took them to write the exra line is more than the time they'll save in a lifetime in execution time.

Aha! The penny drops! Is that why so many methods in random.Random have an "int=int" argument? E.g. def randrange(self, start, stop=None, step=1, int=int, default=None, maxwidth=1L<<BPF): I'd wondered about that. So there you go, now I've learned two things. -- Steven D'Aprano

Steven D'Aprano wrote:

But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist.

Maybe for your particular use case, but the concept of whether an attribute is dynamically defined or not is not well-defined in general. Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want. -- Greg

On 25/08/2010 19:27, P.J. Eby wrote:

At 12:10 PM 8/25/2010 +1200, Greg Ewing wrote:

...

Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want.

Yep. That's why the proposed addition to inspect is a bad idea. If we encourage that sort of static thinking, it will lead to people creating all sorts of breakage with respect to more dynamic code.

AFAICT, the whole "avoid running code" thing only makes sense for a debugging tool

I mentioned another use case - pulling out docstrings. IDEs or other tools that work with live objects may have many such use cases. For proxying objects you can't use the __getattr__ approach for the "magic methods" which aren't looked up through the dynamic attribute 'faking' process. Several proxy libraries I've seen get round this by providing *every* magic method on the proxy class and delegating. This still breaks certain types of duck typing. For example with Python 2.6:

...

...

class x(object): ... @property ... def __call__(self): ... raise AttributeError ... a = x() callable(a) True

If your proxy class defines __call__ then callable returns True, even if the delegation to the proxied object would cause an AttributeError to be raised. A *better* approach (IMO), for both the magic methods and the normal attributes / methods, is to dynamically generate a class that mimics the proxied object (caching generated classes for proxying objects of the same type if you are worried about overhead). This would also work with the suggested "passive introspection" function. All the best, Michael Foord

-- at which point, you can always use the trace facility and throw an error when any Python code runs that's not part of your debugging tool. Something like:

def exists(ob, attr): __running__ = True # ... set trace function here try: try: getattr(ob, attr) return True except AttributeError: return False except CodeRanError: return True # or False if you prefer finally: __running__ = False # restore old tracing here

Where the trace function is just something that throws CodeRanError if it detects a "call" event and the __running__ flag is True. This would stop any Python code from actually executing. (It'd need to keep the same trace function for c_call events, since that might lead to nested non-C calls .)

Of course, a debugger's object inspection tool would probably actually want to return either the attribute value, or a special value to mean "dyanmic calculation needed".

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/

On 25 August 2010 20:57, P.J. Eby <pje@telecommunity.com> wrote:

[snip...] As you can see, the __call__ attribute in each case is whatever the proxied object's __call__ attribute is, even though the proxy itself has a __call__ method, that is invoked when the proxy is called.

This is actually pretty straightforward stuff since the introduction of __getattribute__.

(The code is at http://pypi.python.org/pypi/ProxyTypes, btw.)

For what it's worth this code is useful enough (and simple enough) that I would support its inclusion in the standard library. I've written proxy objects several times (for various different purposes) and this would have saved me the effort. :-) All the best, Michael Foord -- http://www.voidspace.org.uk