[Python-ideas] Faster PyArg_ParseTupleAndKeywords kwargs

[Nathaniel Smith]

On Fri, May 23, 2014 at 8:22 PM,  <dw+python-ideas at hmmz.org> wrote:
> Early while working on py-lmdb I noticed that a huge proportion of
> runtime was being lost to PyArg_ParseTupleAndKeywords, and so I
> subsequently wrote a specialization for this extension module.
>
> In the current code[0], parse_args() is much faster than
> ParseTupleAndKeywords, responsible for a doubling of performance in
> several of the library's faster code paths (e.g.
> Cursor.put(append=True)). Ever since adding the rewrite I've wanted to
> go back and either remove it or at least reduce the amount of custom
> code, but it seems there really isn't a better approach to fast argument
> parsing using the bare Python C API at the moment.
>
>     [0] https://github.com/dw/py-lmdb/blob/master/lmdb/cpython.c#L833
>
> In the append=True path, parse_args() yields a method that can complete
> 1.1m insertions/sec on my crappy Core 2 laptop, compared to 592k/sec
> using the same method rewritten with PyArg_ParseTupleAndKeywords.
>
> Looking to other 'fast' projects for precedent, and studying Cython's
> output in particular, it seems that Cython completely ignores the
> standard APIs and expends a huge amount of .text on using almost every
> imagineable C performance trick to speed up parsing (actually Cython's
> output is a sheer marvel of trickery, it's worth study). So it's clear
> the standard APIs are somewhat non-ideal, and those concerned with
> performance are taking other approaches.

As another data point about PyArg_ParseTupleAndKeywords slowness,
Numpy has tons of barely-maintainable hand-written argument parsing
code. I haven't read the proposal below in detail, but anything that
helps us clean that up is ok with me...

You should check out Argument Clinic (PEP 436) if you haven't seen it.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org