>> Last year, I gave a talk at the Language Summit (during Pycon) to
>> explain that CPython should become 2x faster to remain competitive.
>> IMHO all attempts to optimize Python (CPython forks) have failed
>> because they have been blocked by the C API which implies strict
> Well, that's your opinion, but did you prove it?
I discussed with many Python developers who agree with me that the C API blocked CPython forks. For example, I heard that Pyston was very fast and very promising before starting to support the C API.
The C API requires that your implementations make almost all the same design choices that CPython made 25 years ago (C structures, memory allocators, reference couting, specific GC implementation, GIL, etc.). More efficient technonogies appeared in the meanwhile.
Multiple PyPy developers told me that cpyext remains a blocker issue to use PyPy.
I am not sure how I am supposed to "prove" these facts.
> The one area I know of where the C API is a strong barrier to
> improvement is removing the GIL, and I'd rather let Larry speak about
IMHO Gilectomy is going to fail without our help to change the C API.
Again, I don't want to promise anything here. Removing reference counting inside CPython is a giant project. But at least, I know that borrowed references are very complex to support if CPython doesn't use reference counting. I heard that PyPy has issues to implement borrowed references. If we succeed to remove them, PyPy should benefit directly of that work.
Note: PyPy will still have to support borrowed references for C extensions usong the old C API. But I expect that PyPy will be more reliable, maybe even faster, to run C extensions using the new C API without reference counting.
I have to confess that helping Larry is part of my overall plan. But I dislke making promise that I cannot do, and I dislike working on giant multiyear Python roject. My hope is to have a working new C API next year which will hide some implementation details, but not all of them.
I want to work incrementally using popular C extensions in the feedback loop. Building a new C API is useless if nobody can use it. But it will take time to adjust the backward compatibility cursor.