[Python-Dev] PyObject_CallFunction(func, "O", arg) special case

[Victor Stinner]

Hi,

The PyObject_CallFunction() function has a special case when the
format string is "O", to pass exactly one Python object:

* If the argument is a tuple, the tuple is unpacked: it behaves like func(*arg)
* Otherwise, it behaves like func(arg)

This case is not documented in the C API !
https://docs.python.org/dev/c-api/object.html#c.PyObject_CallFunction


The following C functions have the special case:

* PyObject_CallFunction(), _PyObject_CallFunction_SizeT()
* PyObject_CallMethod(), _PyObject_CallMethod_SizeT()
* _PyObject_CallMethodId(), _PyObject_CallMethodId_SizeT()


I guess that it's a side effect of the implementation: the code uses
Py_BuildValue() and then checks if the value is a tuple or not.
Py_BuildValue() is a little bit surprising:

* "i" creates an integer object
* "ii" creates a tuple
* "(i)" and "(ii)" create a tuple.

Getting a tuple or not depends on the length of the format string. It
is not obvious when you have nested tuples like "O(OO)".

Because of the special case, passing a tuple as the only argument
requires to write "((...))" instead of just "(...)".


In the past, this special behaviour caused a bug in
generator.send(arg), probably because the author of the C code
implementing generator.send() wasn't aware of the special case. See
the issue:
http://bugs.python.org/issue21209

I found code using "O" format in the new _asyncio module, and I'm
quite sure that unpacking special case is not expected. So I opened an
issue:
http://bugs.python.org/issue28920


Last days, I patched functions of PyObject_CallFunction() family to
use internally fast calls. I implemented the special case to keep
backward compatibility.

I replaced a lot of code using PyObject_CallFunction() with
PyObject_CallFunctionObjArgs() when the format string was only made of
"O", PyObject* arguments. I made this change to optimize the code, but
indirectly, it avoids also the special case for code which used
exactly "O" format. See:
http://bugs.python.org/issue28915

When I made these changes, I found some functions which rely the
unpacking feature!

* time_strptime() (change 49a7fdc0d40a)
* unpickle() of _ctypes (change ceb22b8f6d32)

I don't know well what we are supposed to do. I don't think that
changing the behaviour of PyObject_CallFunction() to remove the
special case is a good idea. It would be an obvious backward
incompatible change which can break applications.

I guess that the minimum is to document the special case?

Victor