Fix that broken callable builtin
Hello, I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute. Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`. For example: Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64
bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
callable <built-in function callable> class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... a = A() a <__main__.A object at 0x000000000365B5C0> a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away callable(a) True # it should be False :(
So it boils down to this:
hasattr(a, "__call__") False hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__"). To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
class B: ... @property ... def __call__(self): ... return lambda: 1 ...
b = B() b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry. So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g. $ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
class C: pass ... c = C() c.__call__ = lambda *a: a c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable callable(c) False hasattr(c, '__call__') True
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64
bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
callable <built-in function callable> class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... a = A() a <__main__.A object at 0x000000000365B5C0> a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away callable(a) True # it should be False :(
So it boils down to this:
hasattr(a, "__call__") False hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
class B: ... @property ... def __call__(self): ... return lambda: 1 ...
b = B() b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor.
From that perspective it's inconsistent.
Also there's the issue about not being able to implement a true proxy (as outlined before). What actually prevents this being fixed? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido@python.org> wrote:
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g.
$ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
class C: pass ... c = C() c.__call__ = lambda *a: a c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable callable(c) False hasattr(c, '__call__') True
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș <contact@ionelmc.ro
wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64
bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
callable <built-in function callable> class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... a = A() a <__main__.A object at 0x000000000365B5C0> a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away callable(a) True # it should be False :(
So it boils down to this:
hasattr(a, "__call__") False hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
class B: ... @property ... def __call__(self): ... return lambda: 1 ...
b = B() b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function. Something like (untested): def callable(obj): try: func = obj.__call__ return True except AttributeError: return False -- ~Ethan~
You won't have any more luck defining __add__ as a property -- just don't do that. On how to implement a proxy, I'll let other explain. But this is not it. On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor. From that perspective it's inconsistent.
Also there's the issue about not being able to implement a true proxy (as outlined before).
What actually prevents this being fixed?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido@python.org> wrote:
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g.
$ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
class C: pass ... c = C() c.__call__ = lambda *a: a c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable callable(c) False hasattr(c, '__call__') True
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64
bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
> callable <built-in function callable> > class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... > a = A() > a <__main__.A object at 0x000000000365B5C0> > a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away > callable(a) True > # it should be False :(
So it boils down to this:
> hasattr(a, "__call__") False > hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
class B: ... @property ... def __call__(self): ... return lambda: 1 ... > b = B() > b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
__add__ as a property/descriptor seems to work fine, eg:
class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ...
C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum <guido@python.org> wrote:
You won't have any more luck defining __add__ as a property -- just don't do that.
On how to implement a proxy, I'll let other explain. But this is not it.
On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian Mărieș <contact@ionelmc.ro
wrote:
Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor. From that perspective it's inconsistent.
Also there's the issue about not being able to implement a true proxy (as outlined before).
What actually prevents this being fixed?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido@python.org> wrote:
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g.
$ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
class C: pass ... c = C() c.__call__ = lambda *a: a c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable callable(c) False hasattr(c, '__call__') True
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600
64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information.
>> callable <built-in function callable> >> class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... >> a = A() >> a <__main__.A object at 0x000000000365B5C0> >> a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away >> callable(a) True >> # it should be False :(
So it boils down to this:
>> hasattr(a, "__call__") False >> hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
> class B: ... @property ... def __call__(self): ... return lambda: 1 ... >> b = B() >> b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Yes indeed, that's one way but I wouldn't want to monkeypatch the `callable` builtin. People wouldn't expect that library would even dare monkeypatch builtin. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function.
Something like (untested):
def callable(obj): try: func = obj.__call__ return True except AttributeError: return False
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class). What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-( In the end callable() doesn't always produce a correct answer; but maybe we can make it work in this case by first testing the class and then the instance? Something like (untested): def callable(x): return hasattr(x.__class__, '__call__') and hasattr(x, '__call__') On Fri, Apr 17, 2015 at 2:19 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
__add__ as a property/descriptor seems to work fine, eg:
class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ...
C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum <guido@python.org> wrote:
You won't have any more luck defining __add__ as a property -- just don't do that.
On how to implement a proxy, I'll let other explain. But this is not it.
On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor. From that perspective it's inconsistent.
Also there's the issue about not being able to implement a true proxy (as outlined before).
What actually prevents this being fixed?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido@python.org> wrote:
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g.
$ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
> class C: pass ... > c = C() > c.__call__ = lambda *a: a > c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable > callable(c) False > hasattr(c, '__call__') True >
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600
64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> callable <built-in function callable> >>> class A: ... @property ... def __call__(self): ... raise AttributeError('go away') ... >>> a = A() >>> a <__main__.A object at 0x000000000365B5C0> >>> a.__call__ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __call__ AttributeError: go away >>> callable(a) True >>> # it should be False :(
So it boils down to this:
>>> hasattr(a, "__call__") False >>> hasattr(type(a), "__call__") True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
>> class B: ... @property ... def __call__(self): ... return lambda: 1 ... >>> b = B() >>> b() 1
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
On 04/18, Ionel Cristian Mărieș wrote:
__add__ as a property/descriptor seems to work fine, eg:
class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ...
C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something?
What happens when your __add__ raises an AttributeError? -- ~Ethan~
Same as what would happen when you use the call operator on an object that has an AttributeError raising property:
class D: ... @property ... def __add__(self): ... raise AttributeError('Not so fast, pardner!') ...
D() + D() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in __add__ AttributeError: Not so fast, pardner!
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:44 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 04/18, Ionel Cristian Mărieș wrote:
__add__ as a property/descriptor seems to work fine, eg:
class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ...
C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something?
What happens when your __add__ raises an AttributeError?
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
So propose a patch for callable() along these lines. It'll have to be C code. On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
Yes indeed, that's one way but I wouldn't want to monkeypatch the `callable` builtin. People wouldn't expect that library would even dare monkeypatch builtin.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function.
Something like (untested):
def callable(obj): try: func = obj.__call__ return True except AttributeError: return False
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Yes, it could test first for obj->ob_type->tp_call - no point in testing the instance if the type doesn't have it at all. Regarding the property and how it's used for magic methods, as I understood the cpython code, the slots on the type hold some special wrappers (PyWrapperDescrObject?) that respect the descriptor protocol when called from C (this what makes staticmethod, classmethod and incidentally, property and descriptors in general work for special methods). Eg: https://hg.python.org/cpython/file/v3.4.2/Objects/typeobject.c#l6402 To me all that looks very intentional. It's just that it's not consistent. But what do I know, I've just read the code :-) So all I'm asking is to add an extra check for hasattr(x, "__call__") in the callable builtin. It shouldn't break any code, and it would make `callable` a little bit more reliable. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:39 AM, Guido van Rossum <guido@python.org> wrote:
I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class).
What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-(
In the end callable() doesn't always produce a correct answer; but maybe we can make it work in this case by first testing the class and then the instance? Something like (untested):
def callable(x): return hasattr(x.__class__, '__call__') and hasattr(x, '__call__')
On Fri, Apr 17, 2015 at 2:19 PM, Ionel Cristian Mărieș <contact@ionelmc.ro
wrote:
__add__ as a property/descriptor seems to work fine, eg:
class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ...
C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum <guido@python.org> wrote:
You won't have any more luck defining __add__ as a property -- just don't do that.
On how to implement a proxy, I'll let other explain. But this is not it.
On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor. From that perspective it's inconsistent.
Also there's the issue about not being able to implement a true proxy (as outlined before).
What actually prevents this being fixed?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido@python.org> wrote:
I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g.
$ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
>> class C: pass ... >> c = C() >> c.__call__ = lambda *a: a >> c() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'C' object is not callable >> callable(c) False >> hasattr(c, '__call__') True >>
On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Hello,
I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute.
Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`.
For example:
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 > 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more > information. > >>> callable > <built-in function callable> > >>> class A: > ... @property > ... def __call__(self): > ... raise AttributeError('go away') > ... > >>> a = A() > >>> a > <__main__.A object at 0x000000000365B5C0> > >>> a.__call__ > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "<stdin>", line 4, in __call__ > AttributeError: go away > >>> callable(a) > True > >>> # it should be False :( >
So it boils down to this:
> >>> hasattr(a, "__call__") > False > >>> hasattr(type(a), "__call__") > True
My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__").
To put in contrast, this is legal and clearly indicates the descriptors are being used as expected:
>>> class B: > ... @property > ... def __call__(self): > ... return lambda: 1 > ... > >>> b = B() > >>> b() > 1 >
There 's some more discussing in issue 23990 <http://bugs.python.org/issue23990> where I get slightly angry, sorry.
So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct. This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Ah, yes, someone else has already made a patch here: http://bugs.python.org/file39090/callable.diff This is the issue: http://bugs.python.org/issue23990 Would you add your thoughts there? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 1:19 AM, Guido van Rossum <guido@python.org> wrote:
So propose a patch for callable() along these lines. It'll have to be C code.
On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian Mărieș <contact@ionelmc.ro
wrote:
Yes indeed, that's one way but I wouldn't want to monkeypatch the `callable` builtin. People wouldn't expect that library would even dare monkeypatch builtin.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function.
Something like (untested):
def callable(obj): try: func = obj.__call__ return True except AttributeError: return False
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
No, but you can refer to this thread. On Fri, Apr 17, 2015 at 3:34 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
Ah, yes, someone else has already made a patch here: http://bugs.python.org/file39090/callable.diff
This is the issue: http://bugs.python.org/issue23990
Would you add your thoughts there?
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 1:19 AM, Guido van Rossum <guido@python.org> wrote:
So propose a patch for callable() along these lines. It'll have to be C code.
On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian Mărieș < contact@ionelmc.ro> wrote:
Yes indeed, that's one way but I wouldn't want to monkeypatch the `callable` builtin. People wouldn't expect that library would even dare monkeypatch builtin.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function.
Something like (untested):
def callable(obj): try: func = obj.__call__ return True except AttributeError: return False
-- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
On 4/17/2015 5:19 PM, Ionel Cristian Mărieș wrote:
__add__ as a property/descriptor seems to work fine, eg:
>>> class C: ... @property ... def __add__(self): ... return lambda other: [self, other] ... >>> C() + C() [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at 0x0000000003652CC0>]
Am I missing something?
An instance method is neither a class method nor a static method. It is also not a property. In your example above, __add__ is a property and not an instance method. This violate the Reference Manual definition of the reserved name '__add__' as an instance method. If f is an one-parameter instance method of class C and c is an instance of C, c.f() == C.f(c). This is not true for a property f, as C.f is not callable, and c.f is only callable if the value of the property is callable. -- What you have done above is to curry what is intended to be a binary instance method into a pair of functions, with the outer, function-returning function getting the original name. Having this curried pair implement the binary operator is quite clever. It works because CPython apparently implements c1 + c2, at least for user class instances, as c1.__add__(c2), instead of the presumed equivalent C.__add__(c1, c2). c1.__add__ is normally a bound method with c1 and C.__add__ as attributes. c1.__add__(c2) then calls C.__add__(c1, c2). In your example, c1.__add__ is a unary function with a captured nonlocal object. Calling it with a second object combines (adds) the two objects. The possibility of @property making this work is new to me. -- Terry Jan Reedy
On Fri, Apr 17, 2015 at 02:10:30PM -0700, Ethan Furman wrote:
On 04/18, Ionel Cristian Mărieș wrote:
Also there's the issue about not being able to implement a true proxy (as outlined before).
Proxies are a bit of a pain. But you can create your own callable function.
Something like (untested):
def callable(obj): try: func = obj.__call__ return True except AttributeError: return False
You can define it like that, but it doesn't work. Using the above definition of callable: py> class Spam(object): ... pass ... py> x = Spam() py> x.__call__ = lambda self: 23 py> callable(x) True py> x() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'Spam' object is not callable "foo()" syntax does not look for a __call__ method on the instance, only on the type. The current behaviour of callable is correct: py> builtins.callable(x) False With new-style classes (but not classic classes), all dunder methods are only accessed through the class, not the instance. Hence type(obj).__call__ is correct and obj.__call__ is incorrect. Could this be changed? That deserves a new thread, at least, and possibly a PEP, but briefly: (1) The current behaviour is documented, so it would require some form of transition; (2) The current behaviour is intended as an optimization. Using Python 2.7 on my system, I find that a minimal __add__ method is more than three times faster using a new-style class compared to a classic class. -- Steve
On Sat, Apr 18, 2015 at 1:26 PM, Steven D'Aprano <steve@pearwood.info> wrote:
With new-style classes (but not classic classes), all dunder methods are only accessed through the class, not the instance. Hence type(obj).__call__ is correct and obj.__call__ is incorrect.
Yes, this is correct. But unfortunately is also very commonly misunderstood, because even special methods will work through the descriptor protocol. The __new__ method is the only special case here (the infamous "special cased static method"). py> x.__call__ = lambda self: 23
The issue never was about patching __call__ on an instance, it's about making `callable` respect how the method is actually looked up fully (lookup on type + descriptor protocol). What `callable` is missing now is an additional check that will run the descriptor protocol. Could this be changed? That deserves a new thread, at least, and
possibly a PEP, but briefly: So what exactly are you proposing, making a PEP that documents the fact that functions are descriptors and the descriptor protocol is used even for special methods? To give more context here, this is valid and it works right now: >>> class NonIter:
... pass ...
iter(NonIter()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'NonIter' object is not iterable
class DynamicNonIter: ... has_iter = False ... ... @property ... def __iter__(self): ... if self.has_iter: ... from functools import partial ... return partial(iter, [1, 2, 3]) ... else: ... raise AttributeError("Not really ...") ... dni = DynamicNonIter() iter(dni) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'DynamicNonIter' object is not iterable dni.has_iter = True iter(dni) <list_iterator object at 0x000000000362FF60>
I don't see why `callable` shouldn't work the same. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Fri, Apr 17, 2015 at 09:04:56PM -0400, Terry Reedy wrote:
It works because CPython apparently implements c1 + c2, at least for user class instances, as c1.__add__(c2), instead of the presumed equivalent C.__add__(c1, c2). c1.__add__ is normally a bound method with c1 and C.__add__ as attributes. c1.__add__(c2) then calls C.__add__(c1, c2).
I believe that analysis is wrong. In Python 3, and with new-style classes in Python 2, c1+c2 is implemented as type(c1).__add__(c1, c2) and not c1.__add__(c2). (Actually, it is more complex than that, since the + operator also has to consider __radd__, and various other complications. But let's ignore those complications.) We can test this by giving instances an __add__ attribute in the instance dict, and see what happens. First I confirm that classic classes do implement a+b as a.__add__(b): py> class Classic: ... def __add__(self, other): ... return 23 ... py> classic = Classic() py> classic + None 23 py> from types import MethodType py> classic.__add__ = MethodType(lambda self, other: 42, classic) py> classic + None 42 But the same is not the case with a new-style class: py> class Newstyle(object): ... def __add__(self, other): ... return 23 ... py> newstyle = Newstyle() py> newstyle + None 23 py> newstyle.__add__ = MethodType(lambda self, other: 42, newstyle) py> newstyle + None # ignores the __add__ method on the instance 23 py> newstyle.__add__(None) 42 This demonstrates that when it comes to newstyle classes, dunder methods on the instance are not used. I don't think that using __add__ as a property is intended, but it works, and I don't think it is an accident that it works. It is simply a consequence of how descriptors work in Python. Here is an example: class Demo(object): @property def __add__(self): # Note that there is only a self argument. # Return a function of one argument. If we use a closure, # we can access self. return lambda other: [self, other] py> x = Demo() py> x + None [<__main__.Demo object at 0xb7c2f5ec>, None] Let's break that process down. As I show above, + is implemented as type(x).__add__ which returns a property object, a descriptor. The descriptor protocol says that the __get__ method will be called. So we have: x + None -> looks up type(x).__add__ -> which calls __get__ -> and finally calls the result of that py> type(x).__add__ <property object at 0x935c0a4> py> type(x).__add__.__get__(x, type(x)) <function <lambda> at 0xb7c29a04> py> type(x).__add__.__get__(x, type(x))(None) [<__main__.Demo object at 0xb7c2f5ec>, None] -- Steve
On Sat, Apr 18, 2015 at 01:41:04PM +0300, Ionel Cristian Mărieș wrote:
On Sat, Apr 18, 2015 at 1:26 PM, Steven D'Aprano <steve@pearwood.info> wrote:
With new-style classes (but not classic classes), all dunder methods are only accessed through the class, not the instance. Hence type(obj).__call__ is correct and obj.__call__ is incorrect.
Yes, this is correct. But unfortunately is also very commonly misunderstood, because even special methods will work through the descriptor protocol. The __new__ method is the only special case here (the infamous "special cased static method").
I must admit that until now I had never even thought about making a property with a dunder name. But now that I have considered it, why should it not work? The property (or other descriptor) is on the class, not the instance. py> class Test(object): ... @property ... def __len__(self): ... return lambda: 42 ... py> x = Test() py> len(x) 42 py> type(x).__len__.__get__(x, type(x))() 42 But this won't work on the instance.
The issue never was about patching __call__ on an instance, it's about making `callable` respect how the method is actually looked up fully (lookup on type + descriptor protocol). What `callable` is missing now is an additional check that will run the descriptor protocol.
Okay, I misunderstood you, my apologies. I was mislead by your original suggestion to use hasattr(x, '__call__'), sorry about that. Are you sure this doesn't already work? It works for me in Python 3.3: py> class CallableTest(object): ... @property ... def __call__(self): ... def inner(arg1, arg2): ... return [self, "called with", arg1, arg2] ... return inner ... py> x = CallableTest() py> x(1, 2) [<__main__.CallableTest object at 0xb7b9072c>, 'called with', 1, 2] py> callable(x) True but perhaps I have missed something.
Could this be changed? That deserves a new thread, at least, and possibly a PEP, but briefly:
So what exactly are you proposing, making a PEP that documents the fact that functions are descriptors and the descriptor protocol is used even for special methods?
No, I thought you wanted dunder methods to be used from the instance __dict__, like with classic classes. If that is not your intent, then my comments about a PEP are irrelevant. -- Steve
On Sat, Apr 18, 2015 at 2:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Are you sure this doesn't already work? It works for me in Python 3.3:
It doesn't really work because of the incomplete checks done in `callable_builtin`. This is what I've tried:
class DynamicCallable: ... is_callable = True ... ... def __init__(self, target): ... self.target = target ... ... @property ... def __call__(self): ... if self.is_callable: ... return self.target ... else: ... raise AttributeError("Not really ...") ...
dc = DynamicCallable(print) dc(1, 2, 3) 1 2 3 callable(dc) True dc.is_callable = False callable(dc) True ###### This should be False :(
If the "bug" is fixed, then the last thing in the above example would return False. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 9:41 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
class DynamicCallable: ... is_callable = True ... ... def __init__(self, target): ... self.target = target ... ... @property ... def __call__(self): ... if self.is_callable: ... return self.target ... else: ... raise AttributeError("Not really ...")
Before charging ahead and making the callable() function check for this, is it actually a supported concept? Is it intentional that property-getter functions can raise AttributeError to signal that the attribute does not (currently) exist? Because if that's a hack, then there's no reason to support it, and callable() is absolutely correct to say "there is a __call__ attribute on the class, ergo it's callable". ChrisA
On 17 April 2015 at 18:39, Guido van Rossum <guido@python.org> wrote:
I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class).
What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-(
And for much that I've searched I've never found out the reasoning on that exorcism. Since you are at it, could you describe it? I am glad it is back - I think it is definitely needed - even if just works in a sort of naive way.
On Sat, Apr 18, 2015 at 3:25 PM, Chris Angelico <rosuav@gmail.com> wrote:
Is it intentional that property-getter functions can raise AttributeError to signal that the attribute does not (currently) exist? Because if that's a hack, then there's no reason to support it, and callable() is absolutely correct to say "there is a __call__ attribute on the class, ergo it's callable".
It's hard to say what's intended or not. At best is speculation, as python-the-language does not have a specification. What we have now is made up from many contributions from many people. What we should talk about is "do we want to have this intent or not". Now that's a worthwhile discussion, that can avoid speculation and subjectivity about what's in the docs, what's a hack in contrast to what is in the docs, what's and if there's prior use. You can't talk about how tall your house is going to be until you know what you build it on - is it sand or is it rock?
From my perspective consistency is very important, and it's lacking here w.r.t. how `callable` behaves versus `hasattr` or the call operator. The `iter` builtin already handles `__iter__` differently compared to how `callable` handles `__call__` so there, what do we do with that then? Do we make `iter` raise AttributeError or TypeError (as it's now)? I strongly believe fixing `callable` is the shortest way to achieve some consistency here.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sat, Apr 18, 2015 at 02:41:29PM +0300, Ionel Cristian Mărieș wrote:
On Sat, Apr 18, 2015 at 2:23 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Are you sure this doesn't already work? It works for me in Python 3.3:
It doesn't really work because of the incomplete checks done in `callable_builtin`. This is what I've tried:
class DynamicCallable: ... is_callable = True ... ... def __init__(self, target): ... self.target = target ... ... @property ... def __call__(self): ... if self.is_callable: ... return self.target ... else: ... raise AttributeError("Not really ...") ...
dc = DynamicCallable(print) dc(1, 2, 3) 1 2 3 callable(dc) True dc.is_callable = False callable(dc) True ###### This should be False :(
Why do you think that it should be false? If you actually call dc, the __call__ method/function/property (whatever you want to name it) gets called, and it raises an exception just like you programmed it to. The traceback clearly shows __call__ in the stack. hasattr(type(dc), '__call__') returns true, therefore dc is callable. I think that the fact that the __call__ method ends up raising an AttributeError is irrelevant -- lots of callables raise AttributeError: def func(): return None.spam However, I think *this* is a bug in callable: py> class Callable: ... def __getattribute__(self, name): ... if name == '__call__': ... raise AttributeError # Oh the lies we tell. ... return super().__getattribute__(name) ... def __call__(self): ... return 23 ... py> x = Callable() py> callable(x) False py> x() 23 Clearly x is callable, since I just called it and it returned a value, but callable() thinks it is not. Here is another example: py> class X: pass ... py> x = X() py> x.__call__ = 42 py> callable(x) True py> x() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'X' object is not callable I think callable() is badly broken. (Or at least *was* broken in 3.3 -- I don't have 3.4 handy to try it.) -- Steve
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 2015-04-17 23:39, Guido van Rossum wrote:
I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class).
What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-(
In the end callable() doesn't always produce a correct answer; but maybe we can make it work in this case by first testing the class and then the instance? Something like (untested):
def callable(x): return hasattr(x.__class__, '__call__') and hasattr(x, '__call__')
The code behind callable() is very simple and very fast: int PyCallable_Check(PyObject *x) { if (x == NULL) return 0; return x->ob_type->tp_call != NULL; } IMHO the behavior is well in range of the documentation. It also conforms to my gut feeling and the behavior of PyPy and Jython (tested with Jython 2.7b3+ and PyPy 2.4.0). The three major Python implementation agree on callable(o) == hasattr(type(o), '__call__') for new style classes. Because PyCallable_Check() is so fast with just two pointer derefs, it may be used in several hot paths. Any modification may cause a slow down. This aspect must be thoroughly investigates before the code is changed. For all this reasons I'm -1 on the proposed change. Christian -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJVMpecAAoJEIZoUkkhLbaJyjAH/AiueFdO0wECxZkc53f10Txk Kjb1RB2SRyNIwcvOR5sXJVCP4OrazlTyDSOeCxQ50I8IBXk2vAbdKVEfjuNW4SqQ Dr6xijhA2JjAq/TfBHdMJkcGUySBPkBNTn7Dd50TvJm+PE+D4zlGXpgI7rfZXGM5 MwWrphk0/sB6bZ6WSDjdoCQ40V6CZ1uWTU2N5yd/+vtpA91Yl/FB5Xu7x3sRwt0Y A24GbJHqwgwgnQ7kFozBIbilN3dpcI+Pn5LC6KbqldlNvdp9IMCZh0dm+psnKHVq 2kClbv8f03EahScnKzVh3PblJZ2DB8AEq+PRalmi/v4m0BvWT8a073708BQLocg= =VJMz -----END PGP SIGNATURE-----
On Sat, Apr 18, 2015 at 5:42 AM, Joao S. O. Bueno <jsbueno@python.org.br> wrote:
On 17 April 2015 at 18:39, Guido van Rossum <guido@python.org> wrote:
I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class).
What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-(
And for much that I've searched I've never found out the reasoning on that exorcism. Since you are at it, could you describe it?
I am glad it is back - I think it is definitely needed - even if just works in a sort of naive way.
Glad you asked. The reason for the exorcism was actually the kind of issues brought up in this thread -- there are a variety of edge cases where an object may in fact be called but callable() returns False, and other edge cases where callable() returns True but calling the object fails. (Not to mention that callable() says nothing about the acceptable arguments.) Not having callable() would have avoided this entire thread. -- --Guido van Rossum (python.org/~guido)
On Sun, Apr 19, 2015 at 2:33 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Sat, Apr 18, 2015 at 3:25 PM, Chris Angelico <rosuav@gmail.com> wrote:
Is it intentional that property-getter functions can raise AttributeError to signal that the attribute does not (currently) exist? Because if that's a hack, then there's no reason to support it, and callable() is absolutely correct to say "there is a __call__ attribute on the class, ergo it's callable".
It's hard to say what's intended or not. At best is speculation, as python-the-language does not have a specification.
That's not true; there is a spec for the language, which is independent of CPython, PyPy, etc, which are the implementations of it. There are times when the spec is less than clear, which are often flagged by someone coming to python-dev saying "I'm trying to add Feature X to SomePython, and I'm not sure whether this is how it should be done or not - that's how CPython does it", and then the answer to that question becomes a language spec improvement. But in this case, the question isn't one of Python vs CPython, but one of the use of application-level code. If this is considered a hack, then it's not part of the language spec at all, but if it's deemed a feature, then (a) every Python implementation will be required to match it, and (b) other parts of the language (in this case, callable()) will probably be required to acknowledge it. ChrisA
On Sun, Apr 19, 2015 at 12:10 AM, Chris Angelico <rosuav@gmail.com> wrote:
That's not true; there is a spec for the language, which is independent of CPython, PyPy, etc, which are the implementations of it.
I'm not aware of such a document - can you point me to it? AFAIK `callable` is too old to be in a PEP. I hope you didn't have in mind the "docs as specification", that's not really a specification, it's a "state of things". But in this case, the question isn't one of Python vs CPython, but one
of the use of application-level code. If this is considered a hack, then it's not part of the language spec at all, but if it's deemed a feature, then (a) every Python implementation will be required to match it, and (b) other parts of the language (in this case, callable()) will probably be required to acknowledge it.
I wouldn't care about other python implementations, they are years behind CPython, and this is a very small change anyway. Why would this matter for them if they still implement python-2.7-language? It doesn't really make sense, unless you are considering to fix this in 2.7 too. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Sun, Apr 19, 2015 at 7:27 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Sun, Apr 19, 2015 at 12:10 AM, Chris Angelico <rosuav@gmail.com> wrote:
That's not true; there is a spec for the language, which is independent of CPython, PyPy, etc, which are the implementations of it.
I'm not aware of such a document - can you point me to it? AFAIK `callable` is too old to be in a PEP. I hope you didn't have in mind the "docs as specification", that's not really a specification, it's a "state of things".
Not sure why the docs don't count as a spec. You're probably right that callable() doesn't have any definition outside of the main docs, but you generalized that to the entire language not having a spec, which is definitely not the case. But this thread could easily get coalesced into a simple line or two in the docs, which then _would_ be the spec.
But in this case, the question isn't one of Python vs CPython, but one of the use of application-level code. If this is considered a hack, then it's not part of the language spec at all, but if it's deemed a feature, then (a) every Python implementation will be required to match it, and (b) other parts of the language (in this case, callable()) will probably be required to acknowledge it.
I wouldn't care about other python implementations, they are years behind CPython, and this is a very small change anyway. Why would this matter for them if they still implement python-2.7-language? It doesn't really make sense, unless you are considering to fix this in 2.7 too.
Uhh, that's not exactly true. Maybe it's true of the other three of the Big Four (Jython, IronPython, PyPy), but there are several other Pythons which are compliant with a version 3.x spec; MicroPython and Brython come to mind. ChrisA
On Sun, Apr 19, 2015 at 12:34 AM, Chris Angelico <rosuav@gmail.com> wrote:
Uhh, that's not exactly true. Maybe it's true of the other three of the Big Four (Jython, IronPython, PyPy), but there are several other Pythons which are compliant with a version 3.x spec; MicroPython and Brython come to mind.
Ok, but what's reasonable here? I think you're implying here that this small change should go through a PEP process. So a question for the list, does this seemingly small change warrant a PEP? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 18 April 2015 at 22:40, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Sun, Apr 19, 2015 at 12:34 AM, Chris Angelico <rosuav@gmail.com> wrote:
Uhh, that's not exactly true. Maybe it's true of the other three of the Big Four (Jython, IronPython, PyPy), but there are several other Pythons which are compliant with a version 3.x spec; MicroPython and Brython come to mind.
Ok, but what's reasonable here? I think you're implying here that this small change should go through a PEP process.
So a question for the list, does this seemingly small change warrant a PEP?
Note that the docs for callable() explicitly allow for false positives - callable(x) is true, but x() fails. See https://docs.python.org/3/library/functions.html#callable ("If this returns true, it is still possible that a call fails"). The "bug" you mentioned earlier in the thread is precisely that - a case where callable() returns True but calling the object fails:
It doesn't really work because of the incomplete checks done in `callable_builtin`. This is what I've tried:
class DynamicCallable: ... is_callable = True ... ... def __init__(self, target): ... self.target = target ... ... @property ... def __call__(self): ... if self.is_callable: ... return self.target ... else: ... raise AttributeError("Not really ...") ... dc = DynamicCallable(print) dc(1, 2, 3) 1 2 3 callable(dc) True dc.is_callable = False callable(dc) True ###### This should be False :(
If the "bug" is fixed, then the last thing in the above example would return False..
So I'm not clear what "small change" you're referring to here: * A change to CPython to reduce the number of false positives by making this case return False? If that, then no, I don't think a PEP is needed. But note that user code still can't assume that the above behaviour couldn't still happen in *other* cases, so the change would be of limited value (and whether it gets accepted depends on whether the complexity is justified by the benefit). * A change to the definition of callable() to remove the possibility of false positives at all? In that case, yes, a PEP probably *is* needed, as that's going to affect an awful lot of corner cases, and will impact all implementations. It's probably not correct to call this a "small change". * Something else? Paul
On Sun, Apr 19, 2015 at 2:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
So I'm not clear what "small change" you're referring to here:
* A change to CPython to reduce the number of false positives by making this case return False? If that, then no, I don't think a PEP is needed. But note that user code still can't assume that the above behaviour couldn't still happen in *other* cases, so the change would be of limited value (and whether it gets accepted depends on whether the complexity is justified by the benefit). * A change to the definition of callable() to remove the possibility of false positives at all? In that case, yes, a PEP probably *is* needed, as that's going to affect an awful lot of corner cases, and will impact all implementations. It's probably not correct to call this a "small change". * Something else?
The discussion was about this small change: int PyCallable_Check(PyObject *x) { - if (x == NULL) + if (x == NULL) { return 0; - return x->ob_type->tp_call != NULL; + } + + return Py_TYPE(x)->tp_call && _PyObject_HasAttrId(x, &PyId___call__); } There are more explanations in the thread in case you want to know more. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
Sorry, I pasted a patch with changes unrelated to this discussion. This is the gist of the proposed change: int PyCallable_Check(PyObject *x) { if (x == NULL) return 0; - return x->ob_type->tp_call; + return x->ob_type->tp_call && _PyObject_HasAttrId(x, &PyId___call__); } Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro On Sun, Apr 19, 2015 at 3:30 AM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Sun, Apr 19, 2015 at 2:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
So I'm not clear what "small change" you're referring to here:
* A change to CPython to reduce the number of false positives by making this case return False? If that, then no, I don't think a PEP is needed. But note that user code still can't assume that the above behaviour couldn't still happen in *other* cases, so the change would be of limited value (and whether it gets accepted depends on whether the complexity is justified by the benefit). * A change to the definition of callable() to remove the possibility of false positives at all? In that case, yes, a PEP probably *is* needed, as that's going to affect an awful lot of corner cases, and will impact all implementations. It's probably not correct to call this a "small change". * Something else?
The discussion was about this small change:
int PyCallable_Check(PyObject *x) { - if (x == NULL) + if (x == NULL) { return 0; - return x->ob_type->tp_call != NULL; + } + + return Py_TYPE(x)->tp_call && _PyObject_HasAttrId(x, &PyId___call__); }
There are more explanations in the thread in case you want to know more.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 19 April 2015 at 01:37, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
Sorry, I pasted a patch with changes unrelated to this discussion. This is the gist of the proposed change:
int PyCallable_Check(PyObject *x) { if (x == NULL) return 0; - return x->ob_type->tp_call; + return x->ob_type->tp_call && _PyObject_HasAttrId(x, &PyId___call__); }
OK, thanks. I'd say that doesn't need a PEP. But because it makes callable() a bit slower, and means that it can execute arbitrary Python code where it doesn't at the moment, for a pretty small gain (a few rare cases will no longer give a "false positive" result from callable(), but other false positives could well still exist), I can imagine the change being rejected based on the benefits not justifying the cost. Personally, I don't often use callable(). So I don't have a strong opinion. But when I *do* use it, it's because I want to check if something is callable *without* calling it - so the fact that callable() doesn't run arbitrary code is relevant to me. And the motivating use cases (__call__ being a property on the class object) seem pretty obscure - I've never encountered anything like that in real life. So overall, I'd be -0 on the change. Paul
On Sun, Apr 19, 2015 at 1:14 PM, Paul Moore <p.f.moore@gmail.com> wrote:
so the fact that callable() doesn't run arbitrary code is relevant to me.
_PyObject_HasAttrId(x, &PyId___call__) wouldn't actually do the call. What did you meant to say? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 19 April 2015 at 12:57, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
so the fact that callable() doesn't run arbitrary code is relevant to me.
_PyObject_HasAttrId(x, &PyId___call__) wouldn't actually do the call. What did you meant to say?
Wouldn't it run the get method of the descriptor? I thought that was the point of the change? That was the "arbitrary code" I was referring to (obviously it's well-known code for properties, but for a user-defined descriptor it could be anything). Paul
On 04/19, Ionel Cristian Mărieș wrote:
On Sun, Apr 19, 2015 at 1:14 PM, Paul Moore <p.f.moore@gmail.com> wrote:
so the fact that callable() doesn't run arbitrary code is relevant to me.
_PyObject_HasAttrId(x, &PyId___call__) wouldn't actually do the call. What did you meant to say?
The __get__ method of the descriptor, whether of 'property' or something home-grown, is the "arbitrary code", and could conceivably be quite complex and/or slow -- especially if it has to hit the network to determine whether a proxy object is callable. -- ~Ethan~
On Sun, Apr 19, 2015 at 6:51 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
The __get__ method of the descriptor, whether of 'property' or something home-grown, is the "arbitrary code", and could conceivably be quite complex and/or slow -- especially if it has to hit the network to determine whether a proxy object is callable.
Well indeed. But any property could do evil stuff on attribute access (as opposed to a call), should the programmer choose so. I don't see how this is relevant. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
It's hard to say what's intended or not. At best is speculation, as
On Apr 18, 2015 10:34 AM, "Ionel Cristian Mărieș" <contact@ionelmc.ro> wrote: python-the-language does not have a specification. What we have now is made up from many contributions from many people. The language reference in the docs *is* the spec. Perhaps you mean some something else by "specification"? Are you looking for something from a technical commitee or similar? The spec for Python is defined by Guido and the Python committers. It is not exhaustive but it is authoritative. The python-dev list is where language discussions are driven, including clarifications where the language reference has insufficient information. The PEP process is used to make changes to the language. So keep all this in mind if you are coming from another language community. In the case of __call__, the language reference is sufficiently clear. I have other concerns and suggestions regarding changing callable. See them on the issue on the tracker. -eric
On Sun, Apr 19, 2015 at 03:38:57AM +1000, Steven D'Aprano wrote:
However, I think *this* is a bug in callable: [snip examples] I think callable() is badly broken. (Or at least *was* broken in 3.3 -- I don't have 3.4 handy to try it.)
Ah, the embarassment! It turned out that I had monkey-patched callable() and was calling the broken monkey-patched version, not the original. Sorry for the noise. -- Steve
participants (10)
-
Chris Angelico -
Christian Heimes -
Eric Snow -
Ethan Furman -
Guido van Rossum -
Ionel Cristian Mărieș -
Joao S. O. Bueno -
Paul Moore -
Steven D'Aprano -
Terry Reedy