[Python-Dev] C API changes

[Steve Dower]

On 28Nov2018 2208, Armin Rigo wrote:
> Hi Steve,
> 
> On Tue, 27 Nov 2018 at 19:14, Steve Dower <steve.dower at python.org> wrote:
>> On 27Nov2018 0609, Victor Stinner wrote:
>>> Note: Again, in my plan, the new C API would be an opt-in API. The old
>>> C API would remain unchanged and fully supported. So there is no
>>> impact on performance if you consider to use the old C API.
>>
>> This is one of the things that makes me think your plan is not feasible.
> 
> I can easily imagine the new API having two different implementations
> even for CPython:
> 
> A) you can use the generic implementation, which produces a
> cross-python-compatible .so.  All function calls go through the API at
> runtime.  The same .so works on any version of CPython or PyPy.
> 
> B) you can use a different set of headers or a #define or something,
> and you get a higher-performance version of your unmodified
> code---with the issue that the .so only runs on the exact version of
> CPython.  This is done by defining some of the functions as macros.  I
> would expect this version to be of similar speed than the current C
> API in most cases.
> 
> This might give a way forward: people would initially port their
> extensions hoping to use the option B; once that is done, they can
> easily measure---not guess--- the extra performance costs of the
> option A, and decide based on actual data if the difference is really
> worth the additional troubles of distributing many versions.  Even if
> it is, they can distribute an A version for PyPy and for unsupported
> CPython versions, and add a few B versions on top of that.

This makes sense, but unless it results in PyPy drastically gaining 
popularity as a production runtime, it basically leaves us in the status 
quo. We continue to not be able to change CPython internals at all, 
since that will break people using option B.

Though potentially if we picked an official option for A, we could 
deprecate the stability of option B (over a few releases) and require 
people using it to thoroughly test, update and #ifdef their code for 
each version. That would allow us to make changes to the runtime while 
preserving option A as the reliable version.

You might want to have a look at https://github.com/Microsoft/xlang/ 
which is not yet ready for showtime (in particular, there's no "make it 
look Pythonic" support yet), but is going to extend our existing 
cross-language ABI to Python (alongside C++/.NET/JS) and non-Windows 
platforms. It's been in use for years in Windows and has been just fine. 
(Sample generated output at 
https://github.com/devhawk/pywinrt-output/tree/master/generated/pyrt/src 
but the design docs at the first link are probably most interesting.)

Cheers,
Steve