2016-12-09 18:46 GMT+01:00 Victor Stinner victor.stinner@gmail.com:

Last days, I patched functions of PyObject_CallFunction() family to use internally fast calls. (...) http://bugs.python.org/issue28915

Oh, I forgot to mention the performance results of these changes. Python slots are now a little bit faster. Extract of the issue: http://bugs.python.org/issue28915#msg282748

Microbenchmark on a simple class with an __int__() method, call int(o): int(o): Median +- std dev: [ref] 239 ns +- 13 ns -> [patch] 219 ns +- 14 ns: 1.10x faster (-9%)

Microbenchmark on a simple class with an __getitem__() method, call o[100]: o[100]: Median +- std dev: [ref] 211 ns +- 11 ns -> [patch] 172 ns +- 11 ns: 1.23x faster (-19%)

Comparison between Python 2.7, 3.5, 3.7 and 3.7+patch, 3.5 is used as the reference:

int(o) ======

Median +- std dev: [3.5] 271 ns +- 15 ns -> [3.7] 239 ns +- 13 ns: 1.13x faster (-12%) Median +- std dev: [3.5] 271 ns +- 15 ns -> [patch] 219 ns +- 14 ns: 1.24x faster (-19%) Median +- std dev: [3.5] 271 ns +- 15 ns -> [2.7] 401 ns +- 21 ns: 1.48x slower (+48%)

o[100] ======

Median +- std dev: [3.5] 206 ns +- 5 ns -> [3.7] 211 ns +- 11 ns: 1.02x slower (+2%) Not significant! Median +- std dev: [3.5] 206 ns +- 5 ns -> [patch] 172 ns +- 11 ns: 1.20x faster (-17%) Median +- std dev: [3.5] 206 ns +- 5 ns -> [2.7] 254 ns +- 15 ns: 1.23x slower (+23%)

Victor

On 09.12.16 19:46, Victor Stinner wrote:

The PyObject_CallFunction() function has a special case when the format string is "O", to pass exactly one Python object:

• If the argument is a tuple, the tuple is unpacked: it behaves like func(*arg)
• Otherwise, it behaves like func(arg)

This case is not documented in the C API ! https://docs.python.org/dev/c-api/object.html#c.PyObject_CallFunction

It is documented for Py_BuildValue(), and the documentation of PyObject_CallFunction() refers to Py_BuildValue().

I just found that in spite of your changes of parameter names, the documentation still have old names:

PyObject* PyObject_CallMethod(PyObject *o, const char *method, const char *format, ...)

Oh, I found another recent bug fixed because of the "O" weird behaviour: http://bugs.python.org/issue26478 "dict views don't implement subtraction correctly"

Victor

2016-12-09 18:46 GMT+01:00 Victor Stinner victor.stinner@gmail.com:

Hi,

The PyObject_CallFunction() function has a special case when the format string is "O", to pass exactly one Python object:

• If the argument is a tuple, the tuple is unpacked: it behaves like func(*arg)
• Otherwise, it behaves like func(arg)

This case is not documented in the C API ! https://docs.python.org/dev/c-api/object.html#c.PyObject_CallFunction

The following C functions have the special case:

• PyObject_CallFunction(), _PyObject_CallFunction_SizeT()
• PyObject_CallMethod(), _PyObject_CallMethod_SizeT()
• _PyObject_CallMethodId(), _PyObject_CallMethodId_SizeT()

I guess that it's a side effect of the implementation: the code uses Py_BuildValue() and then checks if the value is a tuple or not. Py_BuildValue() is a little bit surprising:

• "i" creates an integer object
• "ii" creates a tuple
• "(i)" and "(ii)" create a tuple.

Getting a tuple or not depends on the length of the format string. It is not obvious when you have nested tuples like "O(OO)".

Because of the special case, passing a tuple as the only argument requires to write "((...))" instead of just "(...)".

In the past, this special behaviour caused a bug in generator.send(arg), probably because the author of the C code implementing generator.send() wasn't aware of the special case. See the issue: http://bugs.python.org/issue21209

I found code using "O" format in the new _asyncio module, and I'm quite sure that unpacking special case is not expected. So I opened an issue: http://bugs.python.org/issue28920

Last days, I patched functions of PyObject_CallFunction() family to use internally fast calls. I implemented the special case to keep backward compatibility.

I replaced a lot of code using PyObject_CallFunction() with PyObject_CallFunctionObjArgs() when the format string was only made of "O", PyObject* arguments. I made this change to optimize the code, but indirectly, it avoids also the special case for code which used exactly "O" format. See: http://bugs.python.org/issue28915

When I made these changes, I found some functions which rely the unpacking feature!

• time_strptime() (change 49a7fdc0d40a)
• unpickle() of _ctypes (change ceb22b8f6d32)

I don't know well what we are supposed to do. I don't think that changing the behaviour of PyObject_CallFunction() to remove the special case is a good idea. It would be an obvious backward incompatible change which can break applications.

I guess that the minimum is to document the special case?

Victor