[Python-Dev] Comparing PEP 576 and PEP 580

[INADA Naoki]

On Fri, Jul 6, 2018 at 7:50 PM Jeroen Demeyer <J.Demeyer at ugent.be> wrote:
>
> On 2018-07-05 14:20, INADA Naoki wrote:
> > like you ignored my advice about creating realistic benchmark for
> > calling 3rd party callable before talking about performance...
>
> I didn't really want to ignore that, I just didn't know what to do.
>
> As far as I can tell, the official Python benchmark suite is
> https://github.com/python/performance
> However, that deals only with pure Python code, not with the C API.
> So those benchmarks are not relevant to PEP 580.
>

These benchmarks uses 3rd party extension modules.  For example,
mako benchmark uses Mako, and Mako uses MarkupSafe.
I optimized MarkupSafe based on the benchmark.
https://github.com/pallets/markupsafe/pull/64

If bm_mako or some other existing benchmarks is OK to demonstrate
METH_FASTCALL benefits, you can just customize 3rd party
library and compare performance.

If it's not enough, you should write new benchmark to demonstrate it.
One important point is the benchmark should demonstrate "application"
performance.   Comparing just overhead of METH_VARARGS vs METH_FASTCALL
is useless, because we did it already.

What you should demonstrate is: METH_FASTCALL (or METH_FASTCALL | METH_KEYWORDS)
really boost real world application performance.