[Python-Dev] PyObject_CallFunction(func, "O", arg) special case

[Victor Stinner]

Oh, I found another recent bug fixed because of the "O" weird behaviour:
http://bugs.python.org/issue26478
"dict views don't implement subtraction correctly"

Victor

2016-12-09 18:46 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> The PyObject_CallFunction() function has a special case when the
> format string is "O", to pass exactly one Python object:
>
> * If the argument is a tuple, the tuple is unpacked: it behaves like func(*arg)
> * Otherwise, it behaves like func(arg)
>
> This case is not documented in the C API !
> https://docs.python.org/dev/c-api/object.html#c.PyObject_CallFunction
>
>
> The following C functions have the special case:
>
> * PyObject_CallFunction(), _PyObject_CallFunction_SizeT()
> * PyObject_CallMethod(), _PyObject_CallMethod_SizeT()
> * _PyObject_CallMethodId(), _PyObject_CallMethodId_SizeT()
>
>
> I guess that it's a side effect of the implementation: the code uses
> Py_BuildValue() and then checks if the value is a tuple or not.
> Py_BuildValue() is a little bit surprising:
>
> * "i" creates an integer object
> * "ii" creates a tuple
> * "(i)" and "(ii)" create a tuple.
>
> Getting a tuple or not depends on the length of the format string. It
> is not obvious when you have nested tuples like "O(OO)".
>
> Because of the special case, passing a tuple as the only argument
> requires to write "((...))" instead of just "(...)".
>
>
> In the past, this special behaviour caused a bug in
> generator.send(arg), probably because the author of the C code
> implementing generator.send() wasn't aware of the special case. See
> the issue:
> http://bugs.python.org/issue21209
>
> I found code using "O" format in the new _asyncio module, and I'm
> quite sure that unpacking special case is not expected. So I opened an
> issue:
> http://bugs.python.org/issue28920
>
>
> Last days, I patched functions of PyObject_CallFunction() family to
> use internally fast calls. I implemented the special case to keep
> backward compatibility.
>
> I replaced a lot of code using PyObject_CallFunction() with
> PyObject_CallFunctionObjArgs() when the format string was only made of
> "O", PyObject* arguments. I made this change to optimize the code, but
> indirectly, it avoids also the special case for code which used
> exactly "O" format. See:
> http://bugs.python.org/issue28915
>
> When I made these changes, I found some functions which rely the
> unpacking feature!
>
> * time_strptime() (change 49a7fdc0d40a)
> * unpickle() of _ctypes (change ceb22b8f6d32)
>
> I don't know well what we are supposed to do. I don't think that
> changing the behaviour of PyObject_CallFunction() to remove the
> special case is a good idea. It would be an obvious backward
> incompatible change which can break applications.
>
> I guess that the minimum is to document the special case?
>
> Victor