[Python-Dev] C API changes

[Steve Dower]

On 29Nov2018 2206, Armin Rigo wrote:
> On Thu, 29 Nov 2018 at 18:19, Steve Dower <steve.dower at python.org> wrote:
>> quo. We continue to not be able to change CPython internals at all,
>> since that will break people using option B.
> 
> No?  That will only break users if they only have an option-B
> ``foo.cpython-318m-x86_64-linux-gnu.so``, no option-A .so and no
> source code, and want to run it elsewhere than CPython 3.18.  That's
> the same as today.  If you want option-B .so for N versions of
> CPython, recompile the source code N times.
> 
> Just to be clear, if done correctly there should be no need for
> #ifdefs in the source code of the extension module.

The problem is that if option B remains as compatible as it is today, we 
can't make option A faster enough to be attractive. The marketing pitch 
for this looks like: "rewrite all your existing code to be slower but 
works with PyPy, or don't rewrite your existing code and it'll be 
fastest with CPython and won't break in the future". This is status quo 
(where option A today is something like CFFI or Cython), and we can 
already see how many people have made the switch (FWIW, I totally prefer 
Cython over pure C for my own projects :) ).

My proposed marketing pitch is: "rewrite your existing code to be 
forward-compatible today and faster in the future without more work, or 
be prepared to rewrite/update your source code for each CPython release 
to remain compatible with the low level API". The promise of "faster in 
the future" needs to be justified (and I think there's plenty of 
precedent in PyPy, Larry's Gilectomy and the various JavaScript VMs to 
assume that we can do it).

We've already done enough investigation to know that making the runtime 
faster requires changing the low level APIs, and otherwise we're stuck 
in a local optima. Offering a stable, loosely coupled option A and then 
*planning* to break the low level APIs each version in the name of 
performance is the only realistic way to change what we're currently doing.

Cheers,
Steve