On Mon, 19 Nov 2018 11:53:42 +0100
Antoine Pitrou
On Mon, 19 Nov 2018 11:28:46 +0100 Victor Stinner
wrote: I would expect that the most common source of speed up of a C extension is the removal of the cost of bytecode evaluation (ceval.c loop).
Well, I don't. All previous experiments showed that simply compiling Python code to C code using the "generic" C API yielded a 30% improvement.
Conversely, the C _pickle module can be 100x faster than the pure Python pickle module. It's doing it *not* by using the generic C API, but by special-casing access to concrete types. You don't get that level of performance simply by removing the cost of bytecode evaluation:
# C version $ python3 -m timeit -s "import pickle; x = list(range(1000))" "pickle.dumps(x)" 100000 loops, best of 3: 19 usec per loop
# Python version $ python3 -m timeit -s "import pickle; x = list(range(1000))" "pickle._dumps(x)" 100 loops, best of 3: 2.25 msec per loop
And to show that this is important for third-party C extensions as well, PyArrow (*) has comparable performance using similar techniques: $ python -m timeit -s "import pyarrow as pa; x = list(range(1000))" "pa.array(x, type=pa.int64())" 10000 loops, best of 5: 27.2 usec per loop (*) https://arrow.apache.org/docs/python/ Regards Antoine.