[Python-Dev] Let's change to C API!

[Antoine Pitrou]

On Tue, 31 Jul 2018 02:29:42 +0200
Victor Stinner <vstinner at redhat.com> wrote:
> >> Last year, I gave a talk at the Language Summit (during Pycon) to
> >> explain that CPython should become 2x faster to remain competitive.
> >> IMHO all attempts to optimize Python (CPython forks) have failed
> >> because they have been blocked by the C API which implies strict
> >> constraints.  
> >
> > Well, that's your opinion, but did you prove it?  
> 
> I discussed with many Python developers who agree with me that the C API
> blocked CPython forks. For example, I heard that Pyston was very fast and
> very promising before starting to support the C API.

What exactly in the C API made it slow or non-promising?

> The C API requires that your implementations make almost all the same
> design choices that CPython made 25 years ago (C structures, memory
> allocators, reference couting, specific GC implementation, GIL, etc.).

Yes, but those choices are not necessarily bad.

> Multiple PyPy developers told me that cpyext remains a blocker issue to use
> PyPy.

Probably, but we're talking about speeding up CPython here, right?

If we're talking about making more C extensions PyPy-compatible, that's
a different discussion, and one where I think Stefan is right that we
should push people towards Cython and alternatives, rather than direct
use of the C API (which people often fail to use correctly, in my
experience).

But the C API is still useful for specialized uses, *including* for
development tools such as Cython.

> Oh, by the way, I will not promise anything about any potential performance
> gain. When I write "2x faster", I mean that our current approach for
> optimization failed to make Python 2x faster over the last 10 years. Python
> 3 is more or less as fast, or a little bit faster, than Python 2. But
> Python 2 must not be used as an example of performance. People hesitate
> between Go, Javascript and Python. And Python is not the winner in term of
> performance.

I agree about the overall diagnosis.  I just disagree that changing the
C API will open up easy optimization opportunities.

Actually I'd like to see a list of optimizations that you think are
held up by the C API.

> I have to confess that helping Larry is part of my overall plan.

Which is why I'd like to see Larry chime in here.

Regards

Antoine.