[Python-ideas] Why CPython is still behind in performance for some widely used patterns ?

[Nick Coghlan]

On 30 January 2018 at 14:24, Barry Warsaw <barry at python.org> wrote:
> This is often undervalued, but shouldn't be!  Moore's Law doesn't apply
> to humans, and you can't effectively or cost efficiently scale up by
> throwing more bodies at a project.  Python is one of the best languages
> (and ecosystems!) that make the development experience fun, high
> quality, and very efficient.

I'll also note that one of the things we (and others) *have* been
putting quite a bit of time into is the question of "Why do people
avoid using extension modules for code acceleration?".

While there are definitely still opportunities to speed up the CPython
interpreter itself, they're never going to compete for raw speed with
the notion of "Let's just opt out of Python's relatively expensive
runtime bookkeeping for lower level code units, and use native machine
types instead". (Before folks say "But what about PyPy/Numba/etc?":
this is what those tools do as well, they're just able to analyse your
running code and do it on the fly, rather than having an outer
feedback loop of humans doing it explicitly in version control based
on benchmarks, flame graphs, and other performance profiling tools)

And on that front, things have progressed quite a bit in the past few years:

* wheel files & the conda ecosystem mean that precompiled binaries are
often readily available for Windows/Mac OS X/Linux x86_64
* tools like pysip and milksnake are working to reduce the
combinatorial explosion of potential wheel build targets
* tools like Cython work to lower barriers to entry between working
with dynamic compilation and working with precompiled systems (this is
also where Grumpy fits in, but targeting Go rather than C/C++)
* at the CPython interpreter level, we continue to work to reduce the
differences between what extension modules can do and what regular
source and bytecode modules can do

There are still lots of other things it would be nice to have (such as
transplanting the notion of JavaScript source maps so that debuggers
can more readily map Python pyc files and extension modules back to
the corresponding lines of source code), but the idea of "What about
precompiling an extension module?" is already markedly less painful
than it used to be.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia