On 2010-08-23, at 10:46 AM, Benjamin Peterson wrote:

2010/8/23 Yury Selivanov :

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I would be in support of that.

...

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

But not this. Compatibility functions don't belong in 2to3.

There are many possible solutions for the Python 2 porting issue, this one was just one of them. I was trying to make a point, that it's possible to somehow make porting process easier, and meanwhile fix Python 3. - Yury Selivanov Sprymix Inc.

2010/8/23 Guido van Rossum :

On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson wrote:

...

2010/8/23 Yury Selivanov :

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr').  Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.

The test suite passes complete without modification for me. -- Regards, Benjamin

On 8/23/2010 10:22 AM, Yury Selivanov wrote:

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I gather that this amounts to changing "an exception" to "AttributeError" in "(This is implemented by calling getattr(object, name) and seeing whether it raises an exception or not.)" in both the doc and implementation, so that the implementation actually matches the claimed behavior "The result is True if the string is the name of one of the object’s attributes, False if not. " (and by reasonable implication, an exception if this cannot be determined). -- Terry Jan Reedy

On 23/08/2010 22:47, Raymond Hettinger wrote:

On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:

...

I know the issue has been discussed several times already, however I couldn't find any reasonable explanation of its strange behaviour. The main problem with 'hasattr' function is that is swallows all exceptions derived from Exception class. It's a good thing that it doesn't do that with BaseException as it was fixed not a long time ago, but it's definitely not enough.

First of all, this behaviour of 'hasattr' contradicts with the very core principle of python: "Errors should never pass silently." And since 'hasattr' function is in builtins module and is a widely used function it impacts the whole language.

Secondly, take a look at the following:

...

...

...

class Test: ... @property ... def attr(self): ... self['foo'] ... hasattr(Test(), 'attr') False

There can be any exception instead of KeyError in the above snippet of code, but this small example shows how 'hasattr': misleadingly breaks the code logic (1) and masks bug (2). And that's the simplest possible example, there are much more in real life.

While (1) is maybe acceptable for someone, there is no excuse for the (2). Moreover, current 'hasattr' behaviour tremendously complicates use of '__getattribute__' magic. And forget about importlib magic with LazyImports, one 'hasattr' ruins everything by catching ImportError.

To conclude:

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline. This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

Thanks for the nice analysis and good example.

I disagree with the solution though. If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change.

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though. For what it's worth I *agree* with you [1], but for better or worse hasattr / getattr trigger code execution and hasattr can return True for dynamically created members. Something to be revisited for Python 4 perhaps. Michael Foord [1] A while ago I wrote a couple of blog entries on fetching docstrings from members without triggering code execution. It is surprisingly convoluted and even my final code had corner cases it didn't handle: http://www.voidspace.org.uk/python/weblog/arch_d7_2009_05_16.shtml#e1090 http://www.voidspace.org.uk/python/weblog/arch_d7_2009_06_20.shtml#e1103

Raymond

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/

At 12:47 PM 8/23/2010 -0700, Raymond Hettinger wrote:

As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

That just introduces a new class of error when the descriptor can raise AttributeError (e.g. __slots__ descriptors). And of course, it ignoress __getattr__ and __getattribute__.

On Mon, Aug 23, 2010 at 12:47 PM, Raymond Hettinger wrote:

On Aug 23, 2010, at 7:22 AM, Yury Selivanov wrote:

...

I know the issue has been discussed several times already, however I couldn't find any reasonable explanation of its strange behaviour.  The main problem with 'hasattr' function is that is swallows all exceptions derived from Exception class.  It's a good thing that it doesn't do that with BaseException as it was fixed not a long time ago, but it's definitely not enough.

First of all, this behaviour of 'hasattr' contradicts with the very core principle of python: "Errors should never pass silently."  And since 'hasattr' function is in builtins module and is a widely used function it impacts the whole language.

Secondly, take a look at the following:

...

...

...

class Test:    ...     @property    ...     def attr(self):    ...         self['foo']    ... hasattr(Test(), 'attr')    False

There can be any exception instead of KeyError in the above snippet of code, but this small example shows how 'hasattr': misleadingly breaks the code logic (1) and masks bug (2).  And that's the simplest possible example, there are much more in real life.

While (1) is maybe acceptable for someone, there is no excuse for the (2).  Moreover, current 'hasattr' behaviour tremendously complicates use of '__getattribute__' magic.  And forget about importlib magic with LazyImports, one 'hasattr' ruins everything by catching ImportError.

To conclude:

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr').  Probably, Python 3.2 release is our last chance.

2) If you afraid that this new behaviour will break too much python 2 code converted with 2to3, we can introduce another 'hasattr' function defined in 2to3 module itself, and make it imported automatically in all files passed through 2to3 transformation pipeline.  This new function will mimic 'hasattr' behaviour from python 2 and converted code should work as expected.

Thanks for the nice analysis and good example.

I disagree with the solution though.  If we want to see the exceptions associated with actually getting an attribute, then using getattr() instead is a perfectly reasonable solution that people can already use without a language change.

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem.  The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

As your example shows, property() defeats this intent by actually executing the code.   A better behavior would not run the code at all.  It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

IMO, this is a much better solution, more in line with known use cases for hasattr().   If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

Hm... That sounds like scope creep to me. Properties are supposed to be cheap and idempotent. Trying to figure out whether an attribute exists without using __getattr__ is fraught with problems -- as soon as a class overrides __getattr__ or __getattribute__ you're hosed anyway. I can vouch that the reason hasattr() catches too many exceptions is that when I first added it (around 1990, I think :-) I wasn't very attuned yet to the problems it could cause. The main problem I can see with letting exceptions other than AttributeError bubble through (besides perverted dependencies on the current semantics) is that there are some situations where it is pretty arbitrary whether TypeError or AttributeError is raised. I can't recall the details, and possibly this was more of a problem with classic classes, but I do think it warrants some research. -- --Guido van Rossum (python.org/~guido)

2010/8/23 Michael Foord :

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

That's the danger of a dynamic language like Python. Even dir() can now trigger things like that. -- Regards, Benjamin

On Aug 23, 2010, at 1:03 PM, Guido van Rossum wrote:

...

But hasattr() has a far different set of use cases, so we should explore an alternate solution to the problem. The usual reason that people use hasattr() instead of getattr() is that they want to check for the presence of of a method/attribute without actually running it, binding it, or triggering any other behavior.

As your example shows, property() defeats this intent by actually executing the code. A better behavior would not run the code at all. It would check the dictionaries along the MRO but not execute any descriptors associated with a given key.

IMO, this is a much better solution, more in line with known use cases for hasattr(). If the proposed change when through, it would fail to address the common use case and cause people to start writing their own versions of hasattr() that just scan but do not run code.

Hm... That sounds like scope creep to me. Properties are supposed to be cheap and idempotent. Trying to figure out whether an attribute exists without using __getattr__ is fraught with problems -- as soon as a class overrides __getattr__ or __getattribute__ you're hosed anyway.

I don't have a specific proposal in mind. My main questions are * Is there anything that hasattr(obj, key) can or should do that can't already be done with getattr(obj, key, None)? If not, do we really need to change anything? * Why do people typically use hasattr() instead getattr()? Aren't they are really trying to just determine whether a key exists somewhere in the MRO? If so, then doing anything more than that is probably a surprise. I know my own uses of hasattr() do not expect any exceptions at all. It comes up in duck typing support different handling for different types of inputs. If others are using it the same way, I think it is unlikely that they have unittests to cover the possibility that hasattr() would ever start raising exceptions.

I can vouch that the reason hasattr() catches too many exceptions is that when I first added it (around 1990, I think :-) I wasn't very attuned yet to the problems it could cause.

Fire-up the time machine? Raymond P.S. The current behavior seems to be deeply embedded: int PyObject_HasAttr(PyObject *o, PyObject *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds. int PyObject_HasAttrString(PyObject *o, const char *attr_name) Returns 1 if o has the attribute attr_name, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds.

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

2010/8/23 Michael Foord :

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

Well said. The surprise to me in the OP's example was that the property() was executed. Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr(). That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods).

That's the danger of a dynamic language like Python. Even dir() can now trigger things like that.

That's not a honking good thing. I suggest we don't do more of that. Raymond

On Mon, Aug 23, 2010 at 1:33 PM, Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

I don't have a specific proposal in mind.

That's why I called it scope creep. :-) Trust me, your proposal will not lead to a quick and better replacement for hasattr(). (See several other people's replies.)

My main questions are

* Is there anything that hasattr(obj, key) can or should do that can't already be done with getattr(obj, key, None)? If not, do we really need to change anything?

getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly. * Why do people typically use hasattr() instead getattr()?

Aren't they are really trying to just determine whether a key exists somewhere in the MRO? If so, then doing anything more than that is probably a surprise.

Most users who call hasattr() probably don't even know what MRO means. They call hasattr() because they want to avoid a try/except clause. The reasons they are not calling getattr(obj, key, None) could be many: never heard of it, too obscure (it surely doesn't spell "has the attribute" like hasattr() does to the beginning user), or (probably common) the actual attribute access is in some other piece of code they are about to call but don't control.

I know my own uses of hasattr() do not expect any exceptions at all. It comes up in duck typing support different handling for different types of inputs. If others are using it the same way, I think it is unlikely that they have unittests to cover the possibility that hasattr() would ever start raising exceptions.

It already raises *some* exceptions (those derived from BaseException but not Exception). I think that change was a definite non-event.

I can vouch that the reason hasattr() catches too many exceptions is that when I first added it (around 1990, I think :-) I wasn't very attuned yet to the problems it could cause.

Fire-up the time machine?

Raymond

P.S. The current behavior seems to be deeply embedded:

But note that hasattr() doesn't call those.

int PyObject_HasAttr(PyObject http://structures.html#PyObject* *o*, PyObject http://structures.html#PyObject* *attr_name*)<#12aa0a7c7d265b64_PyObject_HasAttr> Returns 1 if *o* has the attribute *attr_name*, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds. int PyObject_HasAttrString(PyObjecthttp://structures.html#PyObject * *o*, const char* *attr_name*) <#12aa0a7c7d265b64_PyObject_HasAttrString> Returns 1 if *o* has the attribute *attr_name*, and 0 otherwise. This is equivalent to the Python expression hasattr(o, attr_name). This function always succeeds.

-- --Guido van Rossum (python.org/~guido)

On 23/08/2010 23:13, Benjamin Peterson wrote:

2010/8/23 Michael Foord:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. That's the danger of a dynamic language like Python. Even dir() can now trigger things like that.

Well yes. One of the reasons a full workaround is so arcane is that if you *really* don't want to trigger code execution you can't call dir... (Otherwise name in dir(obj) is a reasonable approximation for hasattr(obj, name).) Michael -- http://www.ironpythoninaction.com/

2010/8/23 Raymond Hettinger :

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord :

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane.

hasattr(x, "y") doesn't look any more passive to me the x.y.

Well said.  The surprise to me in the OP's example was that the property() was executed.  Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr().

Hard to remember compared to what?

That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods).

Better to raise an exception into unexpecting code that to have subtly different lookup rules between getattr and hasattr. -- Regards, Benjamin

On 23/08/2010 23:55, Benjamin Peterson wrote:

2010/8/23 Raymond Hettinger:

...

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. hasattr(x, "y") doesn't look any more passive to me the x.y. One says "does this object have attribute y" the other fetches attribute y. I'm amazed they don't look different to you. Given Python's object model there is no reason that the first *should* fetch the attribute to determine that it is present, *except* for the dynamic attribute creation of __getattr__ and __getattribute__.

For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not. I'm in both camps though. As we *are* triggering code execution I don't think we should mask exceptions. I just wish we weren't triggering code execution unnecessarily. Michael

...

Well said. The surprise to me in the OP's example was that the property() was executed. Regular methods aren't run by hasattr() so it's hard to remember that when writing code using hasattr(). Hard to remember compared to what?

...

That is especially unfortunate because someone turning a regular attribute into a property may be doing so long after client code has been written (IIRC, that was a key use case for properties). IOW, the user of the hasattr() may have had no way of knowing that an exception could ever be raised (because it is perfectly safe with regular attributes and methods). Better to raise an exception into unexpecting code that to have subtly different lookup rules between getattr and hasattr.

-- http://www.ironpythoninaction.com/

On 24/08/2010 00:05, Benjamin Peterson wrote:

2010/8/23 Michael Foord:

...

...

2010/8/23 Raymond Hettinger:

...

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. hasattr(x, "y") doesn't look any more passive to me the x.y. One says "does this object have attribute y" the other fetches attribute y. I'm amazed they don't look different to you. Given Python's object model

On 23/08/2010 23:55, Benjamin Peterson wrote: there is no reason that the first *should* fetch the attribute to determine that it is present, *except* for the dynamic attribute creation of __getattr__ and __getattribute__. Actually, I'd say given Python's object model, you have to have the attribute in hand to determine that it is present.

...

For properties there is *no reason* why code should be executed merely in order to discover if the attribute exists or not. Properties are allowed to raise AttributeError, so you technically have to execute it.

Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not. If fetching an attribute raises an AttributeError it doesn't mean that attribute doesn't exist (although I admit that at the moment this is exactly what hasattr uses to mean) it just means that fetching that attribute raised an AttributeError. Even if you allow other exceptions to propagate you are still left with the fact that an AttributeError raised as a bug will still be silenced and interpreted as meaning that hasattr should just return False. Michael Foord -- http://www.ironpythoninaction.com/

On 2010-08-23, at 5:02 PM, Michael Foord wrote:

On 23/08/2010 23:55, Benjamin Peterson wrote:

...

2010/8/23 Raymond Hettinger:

...

On Aug 23, 2010, at 1:13 PM, Benjamin Peterson wrote:

...

2010/8/23 Michael Foord:

...

To me hasattr *looks* like a passive introspection function, and the fact that it can trigger arbitrary code execution is unfortunate - especially because a full workaround is pretty arcane. hasattr(x, "y") doesn't look any more passive to me the x.y. One says "does this object have attribute y" the other fetches attribute y. I'm amazed they don't look different to you. Given Python's object model there is no reason that the first *should* fetch the attribute to determine that it is present, *except* for the dynamic attribute creation of __getattr__ and __getattribute__.

As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour. - Yury

Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.

If you don't want to execute properties, do the lookup on the type, not the instance (obviously, you know the dance you need to do, since you've linked the code where you did it). Having apparently simple query operations like hasattr/getattr/len/etc execute arbitrary code is where much of the language's flexibility comes from, so I don't see how it can really be surprising when it happens. To me, Python's definition of an object having an attribute is "Object x has an attribute y if x.y does not raise AttributeError". That means it can't figure out whether or not the attribute exists without actually attempting to retrieve it. There are a few places where we instead use a heuristic that says an attribute *probably* exists if it appears in the instance dictionary, or in the dictionary of one of the types in the MRO, and magic methods have the rule that they must be defined on the type rather than the instance in order to count from the interpreter's point of view, but neither of those things change the basic definition. For the record, +1 on narrowing the scope of the exception suppression in hasattr() to only AttributeError, and adding new C API functions that expose the new behaviour. (I've actually long assumed that AttributeError *was* the only thing suppressed by hasattr()). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

2010/8/23 Michael Foord :

 On 24/08/2010 00:40, Nick Coghlan wrote:

...

...

Properties are allowed to do whatever the heck they want. That doesn't mean that you have to execute code to determine whether they exist or not.

If you don't want to execute properties, do the lookup on the type, not the instance (obviously, you know the dance you need to do, since you've linked the code where you did it). Having apparently simple query operations like hasattr/getattr/len/etc execute arbitrary code is where much of the language's flexibility comes from, so I don't see how it can really be surprising when it happens.

Certainly that is true for len. getattr obviously involves invoking code if you are fetching a property or descriptor. No idea how you conclude that hasattr executing code adds flexibility to the language though.

Yes I know the dance (walking the mro fetching the attribute out of the appropriate type __dict__ or the instance dict - or looking on the metaclass if the object you are introspecting is a type itself), it is just not trivial - which is why I think it is a shame that people are forced to implement it just to ask if a member exists without triggering code execution.

Sounds like something for a new function. -- Regards, Benjamin

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov wrote:

...

As I understand the only possible way to make 'hasattr' work as it name indicates (i.e. just check if attribute exists, not run it), is to add another magic method(s?) to the existing __getattr__ and __getattribute__ which will tell whether attribute exists or not, and by default this method would mimic current 'hasattr' behaviour.

You nailed it. At the lowest level (either in C or in Python) there is no way to check for an attribute's presence without getting its value. This has so far been fundamental, and everything else is just appearances.

I propose that this is good enough and that we should at this point not try to invent another protocol, just decide whether hasattr() can be fixed (and leaving PyObject_HasAttr*() alone).

BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general. The more I think about it the more I like the idea. By default, 'hasattr' would check MRO for the attribute, if not found - check for __hasattr__, if not found - fallback to the current schema with 'getattr'. This would not break any current code, but open a lot of potential (especially for things that implement some lazy loading protocols). From the performance point of view, I believe it shouldn't change much, as current 'hasattr' which uses 'getattr' which calls __getattr(ibute)__ methods. I know this is much more then just fixing exception swallowing, but this is somewhat required functionality to complete the current dynamism python offers. - Yury

2010/8/23 Yury Selivanov :

...

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov wrote: BTW, is it possible to add new magic method __hasattr__?  Maybe not in Python 3.2, but in general.

-1 The whole point of hasattr() is that getattr() on that attribute would return something. Let's not destroy that. -- Regards, Benjamin

On 2010-08-23, at 6:17 PM, Benjamin Peterson wrote: 2010/8/23 Yury Selivanov :

...

...

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov wrote: BTW, is it possible to add new magic method __hasattr__? Maybe not in Python 3.2, but in general.

-1 The whole point of hasattr() is that getattr() on that attribute would return something. Let's not destroy that.

By default, if you don't overload __hasattr__, it will do exactly that. But let's imagine the situation in some ORM. Sometimes, engine knows that attribute exists even before executing it (which leads to querying DB), and thus it is possible to return True in the overloaded __hasattr__. Example: ... if hasattr(entity, 'title'): # <- __hasattr__ has the info in metadata, # no query to DB ... title = entity.title # <- now, we can query the DB So, the proposed magic method is not intended to change the protocol, but to complement and enhance it. - Yury

2010/8/23 Yury Selivanov :

...

On 2010-08-23, at 6:17 PM, Benjamin Peterson wrote: 2010/8/23 Yury Selivanov :

...

...

On 2010-08-23, at 6:00 PM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 2:22 PM, Yury Selivanov wrote: BTW, is it possible to add new magic method __hasattr__?  Maybe not in Python 3.2, but in general.

-1 The whole point of hasattr() is that getattr() on that attribute would return something. Let's not destroy that.

By default, if you don't overload __hasattr__, it will do exactly that.

But let's imagine the situation in some ORM.  Sometimes, engine knows that attribute exists even before executing it (which leads to querying DB), and thus it is possible to return True in the overloaded __hasattr__.

Example: ... if hasattr(entity, 'title'): # <- __hasattr__ has the info in metadata,                                 #    no query to DB ...     title = entity.title     # <- now, we can query the DB

In this case, I think it would make more sense to ask the model: type(entity).has_property("title")

So, the proposed magic method is not intended to change the protocol, but to complement and enhance it.

But it still raises the potential to break the relationship between hasattr and getattr. I think this should require a PEP. -- Regards, Benjamin

On Tue, Aug 24, 2010 at 8:25 AM, Benjamin Peterson wrote:

2010/8/23 Yury Selivanov :

...

So, the proposed magic method is not intended to change the protocol, but to complement and enhance it.

But it still raises the potential to break the relationship between hasattr and getattr.

I think this should require a PEP.

Definitely needs a PEP, and will require some solid use cases to explain why allowing optimisation of hasattr() is the right way to go. Complexity isn't free, and I doubt the gains here will justify the costs, but that's one of the things the PEP process is intended to figure out. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Aug 23, 2010, at 03:45 PM, Benjamin Peterson wrote:

It's generally more convenient that getattr(obj, "blah", None) is not None.

Clearly, the solution is a new builtin called 'Missing'. <wink> -Barry

Michael Foord wrote:

It would be backwards incompatible with usage of hasattr for dynamically created 'members' using __getattr__ though.

Also keep in mind that builtin types mostly don't keep their attributes in dictionaries. To make this work properly, hasattr would need its own special method. -- Greg

2010/8/23 Steven D'Aprano :

On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:

...

...

* Is there anything that hasattr(obj, key) can or should do that   can't already be done with getattr(obj, key, None)?   If not, do we really need to change anything?

getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly.

Why do you say it's ugly? It's a short, sweet, simple two-liner:

mark = object() getattr(obj, 'key', mark) is not mark

Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function.

That's not all, though. You still have to test it with an "if" and that gets cumbersome after a while.

I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__.

Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist.

Well, that's exactly what hasattr does... -- Regards, Benjamin

On Mon, Aug 23, 2010 at 4:56 PM, Steven D'Aprano wrote:

On Tue, 24 Aug 2010 06:50:19 am Guido van Rossum wrote:

...

...

* Is there anything that hasattr(obj, key) can or should do that   can't already be done with getattr(obj, key, None)?   If not, do we really need to change anything?

getattr(obj, 'key', None) returns None when obj.key exists and has the value None. The workaround is ugly.

Why do you say it's ugly? It's a short, sweet, simple two-liner:

mark = object() getattr(obj, 'key', mark) is not mark

Nothing ugly about it at all. But if somebody really objected to using a two lines, they could put it in a utility function.

Because if you didn't have the foresight to write that utility function, you have to break your train of thought (aka yak shaving) to either write it (big yak) or write those two lines (little yak, times the number of times it happens). Whereas with hasattr() you can just type the correct hasattr() expression in-line in the if that you already started typing.

It still doesn't cope with dynamically-generated attributes that are either expensive or have side-effects (both of which are probably poor design, but nevertheless I'm sure they're out there), but neither does the existing hasattr.

...

* Why do people typically use hasattr() instead getattr()?

...

   Aren't they are really trying to just determine    whether a key exists somewhere in the MRO?    If so, then doing anything more than that is probably a surprise.

Most users who call hasattr() probably don't even know what MRO means.

Well, yes, but most users never write __getattr__ or __getattribute__ methods either.

I have always thought that hasattr() does what it says on the box: it tests for the *existence* of an attribute, that is, one that statically exists rather than being dynamically generated. In other words, it is a key in the instance __dict__ or is inherited from the class __dict__ or that of a superclass, or a __slot__.

It tests for the existence of an attribute -- how the attribute is defined should not have to occur to you (and there are lots of other ways for attributes to be defined besides the ways you came up with just now).

Now that I know that hasattr doesn't do what I thought it does or what the name implies it does, it has little or no utility for me. In the future, I'll just write a try...except block and catch errors if the attribute doesn't exist.

The try/except block *also* requires you to break your train of thought. And most of the time the error case just isn't important. You sound like you are over-engineering it and focusing too much on performance instead of on getting things done. Like those people who learn that it saves an usec to copy a built-in function into a defalt arg (def foo(arg1, len=len): ...) and then overuse the trick even when the time it took them to write the exra line is more than the time they'll save in a lifetime in execution time. -- --Guido van Rossum (python.org/~guido)

Yuri, I think you are making a good case (though I would like for you to be a good citizen and use the bug tracker to submit this for review). Benjamin, what do you think? --Guido On Mon, Aug 23, 2010 at 7:14 PM, Yury Selivanov wrote:

...

On 2010-08-23, at 10:56 AM, Guido van Rossum wrote: On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson wrote:

...

2010/8/23 Yury Selivanov :

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr').  Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I recommend that you create a patch, apply it, run the *entire* stdlib test suite and see how much breaks. That will give you an idea of the damage to expect for 3rd party code.

Have done a little testing.

The patch to builtin 'hasattr' function is trivial, and can be found attached to this letter (with the corresponding unittest):

-        if (!PyErr_ExceptionMatches(PyExc_Exception)) +        if (!PyErr_ExceptionMatches(PyExc_AttributeError))

So, after applying it, hasattr swallows only AttributeError exceptions.

All tests from Python 3.2 test suite were good.

After that, I've applied the patch on Python 2.6 and tested SqlAlchemy, Django and Twisted.

- SqlAlchemy has failed three unittests, but those tests were designed specifically to handle 'hasattr' weird behaviour, so we can consider the change has no impact on SqlAlchemy.

- Twisted - failed 3 tests out of ~3200, but it fails them on the same machine on an unpatched Python too, and they seem unrelated.

- Django - all tests passed.

I tested our internal company framework (~100K LOC) and, again, all tests passed.

-- --Guido van Rossum (python.org/~guido)

2010/8/23 Guido van Rossum :

Yuri, I think you are making a good case (though I would like for you to be a good citizen and use the bug tracker to submit this for review). Benjamin, what do you think?

I will have a look right now. -- Regards, Benjamin

On 08/23/2010 04:56 PM, Guido van Rossum wrote:

On Mon, Aug 23, 2010 at 7:46 AM, Benjamin Peterson wrote:

...

2010/8/23 Yury Selivanov:

...

1) I propose to change 'hasattr' behaviour in Python 3, making it to swallow only AttributeError exceptions (exactly like 'getattr'). Probably, Python 3.2 release is our last chance.

I would be in support of that.

I am cautiously in favor. The existing behavior is definitely a mistake and a trap. But it has been depended on for almost 20 years now.

I'll note that a similar incompatible change has made it to python2.6. This has bitten us in production: class X(object): def __getattr__(self, name): raise KeyError, "error looking for %s" % (name,) def __iter__(self): yield 1 print list(X()) I would expect it to print [1], and in python2.5 it does. In python2.6 it raises a KeyError! The attribute being looked up is an unexpected one: {hrzagude5003}[~]$ python2.6 a.py Traceback (most recent call last): File "a.py", line 8, in <module> print list(X()) File "a.py", line 3, in __getattr__ raise KeyError, "error looking for %s" % (name,) KeyError: 'error looking for __length_hint__' The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list(). Being aware that this can be construed as an argument both in favor and against the change at hand, my point is that, if propagating non-AttributeError exceptions is done in checks intended to be invisible, it should certainly be done in hasattr, where it is at least obvious what is being done. Other generic functions and operators, including boolean ones such as ==, happily propagate exceptions. Also, don't expect that this won't break code out there. It certainly will, it's only a matter of assessment whether such code was broken in a different, harder to detect way, to begin with.

2010/8/24 Hrvoje Niksic :

The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others.  I'm not sure if this is a bug.  On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list().

__length_hint__ is internal and undocumented, so it can do whatever it wants. -- Regards, Benjamin

2010/8/24 Hrvoje Niksic :

On 08/24/2010 02:31 PM, Benjamin Peterson wrote:

...

2010/8/24 Hrvoje Niksic:

...

 The __length_hint__ lookup expects either no exception or AttributeError,  and will propagate others.  I'm not sure if this is a bug.  On the one hand,  throwing anything except AttributeError from __getattr__ is bad style (which  is why we fixed the bug by deriving our business exception from  AttributeError), but the __length_hint__ check is supposed to be an internal  optimization completely invisible to the caller of list().

__length_hint__ is internal and undocumented, so it can do whatever it wants.

Of course, but that's beside the point.  In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail).  The point I'm making is that:

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

b) making the proposed change is bound to break real, production code.

I agree with. This is why the change is not making its way into any maintenance release. -- Regards, Benjamin

2010/8/24 P.J. Eby :

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug. -- Regards, Benjamin

On 8/24/2010 9:45 AM, Benjamin Peterson wrote:

2010/8/24 Hrvoje Niksic :

...

On 08/24/2010 02:31 PM, Benjamin Peterson wrote:

...

2010/8/24 Hrvoje Niksic:

...

The __length_hint__ lookup expects either no exception or AttributeError, and will propagate others. I'm not sure if this is a bug. On the one hand, throwing anything except AttributeError from __getattr__ is bad style (which is why we fixed the bug by deriving our business exception from AttributeError), but the __length_hint__ check is supposed to be an internal optimization completely invisible to the caller of list().

__length_hint__ is internal and undocumented, so it can do whatever it wants.

Of course, but that's beside the point. In this case __length_hint__ was neither implemented in the class, nor were we aware of its existence, and the code still broke (as in the example in previous mail). The point I'm making is that:

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

b) making the proposed change is bound to break real, production code.

I agree with. This is why the change is not making its way into any maintenance release.

+1 regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/

On Tue, 24 Aug 2010 09:26:09 -0500, Benjamin Peterson wrote:

2010/8/24 P.J. Eby :

...

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug.

Benjamin, I remember you fixing various special method lookups, so just for clarity's sake, which versions of Python does your statement apply to? -- R. David Murray www.bitdance.com

2010/8/24 James Y Knight :

On Aug 24, 2010, at 10:26 AM, Benjamin Peterson wrote:

...

2010/8/24 P.J. Eby :

...

At 03:37 PM 8/24/2010 +0200, Hrvoje Niksic wrote:

...

a) a "business" case of throwing anything other than AttributeError from __getattr__ and friends is almost certainly a bug waiting to happen, and

FYI, best practice for __getattr__ is generally to bail with an AttributeError as soon as you see double underscores in the name, unless you intend to support special attributes.

Unless you're in an old-style class, you shouldn't get an double underscore methods in __getattr__ (or __getattribute__). If you do, it's a bug.

Uh, did you see the message that was in response to?

Maybe it should be a bug report?

Old version of Python I think. -- Regards, Benjamin

Steven D'Aprano wrote:

But that's the thing... as far as I am concerned, a dynamically defined attribute *doesn't* exist.

Maybe for your particular use case, but the concept of whether an attribute is dynamically defined or not is not well-defined in general. Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want. -- Greg

On 25/08/2010 19:27, P.J. Eby wrote:

At 12:10 PM 8/25/2010 +1200, Greg Ewing wrote:

...

Consider an object that is trying to be a transparent proxy for another object, and behave as much as possible as though it really were the other object. Should an attribute statically defined on the proxied object be considered dynamically defined on the proxy? If so, then the proxy isn't as transparent as some people may want.

Yep. That's why the proposed addition to inspect is a bad idea. If we encourage that sort of static thinking, it will lead to people creating all sorts of breakage with respect to more dynamic code.

AFAICT, the whole "avoid running code" thing only makes sense for a debugging tool

I mentioned another use case - pulling out docstrings. IDEs or other tools that work with live objects may have many such use cases. For proxying objects you can't use the __getattr__ approach for the "magic methods" which aren't looked up through the dynamic attribute 'faking' process. Several proxy libraries I've seen get round this by providing *every* magic method on the proxy class and delegating. This still breaks certain types of duck typing. For example with Python 2.6:

...

...

class x(object): ... @property ... def __call__(self): ... raise AttributeError ... a = x() callable(a) True

If your proxy class defines __call__ then callable returns True, even if the delegation to the proxied object would cause an AttributeError to be raised. A *better* approach (IMO), for both the magic methods and the normal attributes / methods, is to dynamically generate a class that mimics the proxied object (caching generated classes for proxying objects of the same type if you are worried about overhead). This would also work with the suggested "passive introspection" function. All the best, Michael Foord

-- at which point, you can always use the trace facility and throw an error when any Python code runs that's not part of your debugging tool. Something like:

def exists(ob, attr): __running__ = True # ... set trace function here try: try: getattr(ob, attr) return True except AttributeError: return False except CodeRanError: return True # or False if you prefer finally: __running__ = False # restore old tracing here

Where the trace function is just something that throws CodeRanError if it detects a "call" event and the __running__ flag is True. This would stop any Python code from actually executing. (It'd need to keep the same trace function for c_call events, since that might lead to nested non-C calls .)

Of course, a debugger's object inspection tool would probably actually want to return either the attribute value, or a special value to mean "dyanmic calculation needed".

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/

On 24/08/2010 08:40, Michael Foord wrote:

On 24/08/2010 01:25, Nick Coghlan wrote:

...

On Tue, Aug 24, 2010 at 8:15 AM, Nick Coghlan wrote:

...

Now, it may be worth considering an addition to the inspect module that was basically:

def getattr_static(obj, attr): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__.

Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass Second attempt with a default value parameter and correctly raising AttributeError if not found:

_sentinel = object() def getattr_static(obj, attr, default=_sentinel): """Retrieve attributes without triggering dynamic lookup via the descriptor protocol, __getattr__ or __getattribute__.

Note: this function may not be able to retrieve all attributes reported by dir(obj) """ try: instance_dict = object.__getattribute__(obj, "__dict__") except AttributeError: pass else: if attr in instance_dict: return instance_dict[attr] for entry in getmro(obj.__class__): try: return entry.__dict__[attr] except AttributeError: pass if default is not _sentinel: return default raise AttributeError(attr)

This doesn't correctly handle the case where obj is a type object or obj uses __slots__.

If I have time (currently on vacation with only intermittent internet access) I'll provide an update.

I managed an updated version (with tests) during the flight home last night. I've attached it to issue 9732, along with a discussion of the caveats and ways of breaking it: http://bugs.python.org/issue9732 Michael

Michael

...

Cheers, Nick.

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.