[Python-Dev] Let's change to C API!

[Victor Stinner]

>> Last year, I gave a talk at the Language Summit (during Pycon) to
>> explain that CPython should become 2x faster to remain competitive.
>> IMHO all attempts to optimize Python (CPython forks) have failed
>> because they have been blocked by the C API which implies strict
>> constraints.
>
> Well, that's your opinion, but did you prove it?

I discussed with many Python developers who agree with me that the C API
blocked CPython forks. For example, I heard that Pyston was very fast and
very promising before starting to support the C API.

The C API requires that your implementations make almost all the same
design choices that CPython made 25 years ago (C structures, memory
allocators, reference couting, specific GC implementation, GIL, etc.). More
efficient technonogies appeared in the meanwhile.

Multiple PyPy developers told me that cpyext remains a blocker issue to use
PyPy.

I am not sure how I am supposed to "prove" these facts.

Oh, by the way, I will not promise anything about any potential performance
gain. When I write "2x faster", I mean that our current approach for
optimization failed to make Python 2x faster over the last 10 years. Python
3 is more or less as fast, or a little bit faster, than Python 2. But
Python 2 must not be used as an example of performance. People hesitate
between Go, Javascript and Python. And Python is not the winner in term of
performance.

> The one area I know of where the C API is a strong barrier to
> improvement is removing the GIL, and I'd rather let Larry speak about
> it.

IMHO Gilectomy is going to fail without our help to change the C API.

Again, I don't want to promise anything here. Removing reference counting
inside CPython is a giant project. But at least, I know that borrowed
references are very complex to support if CPython doesn't use reference
counting. I heard that PyPy has issues to implement borrowed references. If
we succeed to remove them, PyPy should benefit directly of that work.

Note: PyPy will still have to support borrowed references for C extensions
usong the old C API. But I expect that PyPy will be more reliable, maybe
even faster, to run C extensions using the new C API without reference
counting.

I have to confess that helping Larry is part of my overall plan. But I
dislke making promise that I cannot do, and I dislike working on giant
multiyear Python roject. My hope is to have a working new C API next year
which will hide some implementation details, but not all of them.

I want to work incrementally using popular C extensions in the feedback
loop. Building a new C API is useless if nobody can use it. But it will
take time to adjust the backward compatibility cursor.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180731/bac00fd2/attachment.html>