[Python-ideas] Fix that broken callable builtin

[Ionel Cristian Mărieș]

Yes, it could test first for obj->ob_type->tp_call - no point in testing
the instance if the type doesn't have it at all.

Regarding the property and how it's used for magic methods, as I understood
the cpython code, the slots on the type hold some special wrappers
(PyWrapperDescrObject?) that respect the descriptor protocol when called
from C (this what makes staticmethod, classmethod and incidentally,
property and descriptors in general work for special methods). Eg:
https://hg.python.org/cpython/file/v3.4.2/Objects/typeobject.c#l6402

To me all that looks very intentional. It's just that it's not consistent.
But what do I know, I've just read the code :-)

​ So all I​'m asking is to add an extra check for hasattr(x, "__call__") in
the callable builtin. It shouldn't break any code, and it would make
`callable` a little bit more reliable.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Sat, Apr 18, 2015 at 12:39 AM, Guido van Rossum <guido at python.org> wrote:

> I think you've found an unintended and undocumented backdoor. I admit I
> don't understand how this works in CPython. Overloaded operators like
> __add__ or __call__ should be methods in the class, and we don't look for
> them in the instance. But somehow defining them with @property works (I
> guess because @property is in the class).
>
> What's different for __call__ is that callable() exists. And this is
> probably why I exorcised it Python 3.0 -- but apparently it's back. :-(
>
> In the end callable() doesn't always produce a correct answer; but maybe
> we can make it work in this case by first testing the class and then the
> instance? Something like (untested):
>
> def callable(x):
>     return hasattr(x.__class__, '__call__') and hasattr(x, '__call__')
>
>
> On Fri, Apr 17, 2015 at 2:19 PM, Ionel Cristian Mărieș <contact at ionelmc.ro
> > wrote:
>
>> __add__ as a property/descriptor seems to work fine, eg:
>>
>> >>> class C:
>>> ...  @property
>>> ...  def __add__(self):
>>> ...   return lambda other: [self, other]
>>> ...
>>> >>> C() + C()
>>> [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at
>>> 0x0000000003652CC0>]
>>>
>>
>> Am I missing something?
>>
>>
>> Thanks,
>> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>>
>> On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> You won't have any more luck defining __add__ as a property -- just
>>> don't do that.
>>>
>>> On how to implement a proxy, I'll let other explain. But this is not it.
>>>
>>> On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian Mărieș <
>>> contact at ionelmc.ro> wrote:
>>>
>>>> Well yes, from that example it look right, because the call operator
>>>> uses the __call__ attribute from the type of the object. However, when the
>>>> call operator gets the __call__ method it will actually use it as a
>>>> descriptor. From that perspective it's inconsistent.
>>>>
>>>> Also there's the issue about not being able to implement a true proxy
>>>> (as outlined before).
>>>>
>>>> What actually prevents this being fixed?
>>>>
>>>>
>>>> Thanks,
>>>> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>>>>
>>>> On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum <guido at python.org>
>>>> wrote:
>>>>
>>>>> I think you're fighting windmills. Like most special operations (e.g.
>>>>> __add__), a __call__ attribute on the object does not work, i.e. it does
>>>>> not make the object callable. E.g.
>>>>>
>>>>> $ python3
>>>>> Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar  8 2015, 01:12:06)
>>>>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> >>> class C: pass
>>>>> ...
>>>>> >>> c = C()
>>>>> >>> c.__call__ = lambda *a: a
>>>>> >>> c()
>>>>> Traceback (most recent call last):
>>>>>   File "<stdin>", line 1, in <module>
>>>>> TypeError: 'C' object is not callable
>>>>> >>> callable(c)
>>>>> False
>>>>> >>> hasattr(c, '__call__')
>>>>> True
>>>>> >>>
>>>>>
>>>>> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian Mărieș <
>>>>> contact at ionelmc.ro> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I had an issue today with the `callable` builtin because it doesn't
>>>>>> correctly check that the object has the __call__ attribute.
>>>>>>
>>>>>> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')`
>>>>>> but that's not very straightforward. A more straightforward implementation
>>>>>> would do something like `hasattr(a, '__call__')`.
>>>>>>
>>>>>> For example:
>>>>>>
>>>>>> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600
>>>>>>> 64 bit (AMD64)] on win32
>>>>>>> Type "help", "copyright", "credits" or "license" for more
>>>>>>> information.
>>>>>>> >>> callable
>>>>>>> <built-in function callable>
>>>>>>> >>> class A:
>>>>>>> ...  @property
>>>>>>> ...  def __call__(self):
>>>>>>> ...   raise AttributeError('go away')
>>>>>>> ...
>>>>>>> >>> a = A()
>>>>>>> >>> a
>>>>>>> <__main__.A object at 0x000000000365B5C0>
>>>>>>> >>> a.__call__
>>>>>>> Traceback (most recent call last):
>>>>>>>   File "<stdin>", line 1, in <module>
>>>>>>>   File "<stdin>", line 4, in __call__
>>>>>>> AttributeError: go away
>>>>>>> >>> callable(a)
>>>>>>> True
>>>>>>> >>> # it should be False :(
>>>>>>>
>>>>>>
>>>>>> ​So it boils down to this:
>>>>>>
>>>>>>> >>> hasattr(a, "__call__")
>>>>>>> False
>>>>>>> >>> hasattr(type(a), "__call__")
>>>>>>> True
>>>>>>
>>>>>>
>>>>>> My issue is that I didn't call `callable(type(a))` but just
>>>>>> `callable(a)`. Clearly mismatching what happens when you do hasattr(a,
>>>>>> "__call__").
>>>>>>
>>>>>> To put in contrast, this is legal and clearly indicates the
>>>>>> descriptors are being used as expected:
>>>>>>
>>>>>> >>> class B:
>>>>>>> ...  @property
>>>>>>> ...  def __call__(self):
>>>>>>> ...   return lambda: 1
>>>>>>> ...
>>>>>>> >>> b = B()
>>>>>>> >>> b()
>>>>>>> 1
>>>>>>>
>>>>>>
>>>>>> ​There​
>>>>>> ​'s some more discussing in issue 23990
>>>>>> <http://bugs.python.org/issue23990>​
>>>>>> ​ where I get slightly angry, sorry.​
>>>>>>
>>>>>>
>>>>>> ​So were is this change actually useful? Proxies! Since new-style
>>>>>> objects in Python you cannot really proxy the callable aspect of objects,
>>>>>> because `callable` just checks that a field is set in a C struct.​
>>>>>> ​ This is fairly inconvenient because you have to know upfront if
>>>>>> your target is going to be callable or not.​
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>>>>>>
>>>>>> _______________________________________________
>>>>>> Python-ideas mailing list
>>>>>> Python-ideas at python.org
>>>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Guido van Rossum (python.org/~guido)
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150418/94e84a09/attachment.html>