On 28Nov2018 2208, Armin Rigo wrote:
On Tue, 27 Nov 2018 at 19:14, Steve Dower email@example.com wrote:
On 27Nov2018 0609, Victor Stinner wrote:
Note: Again, in my plan, the new C API would be an opt-in API. The old C API would remain unchanged and fully supported. So there is no impact on performance if you consider to use the old C API.
This is one of the things that makes me think your plan is not feasible.
I can easily imagine the new API having two different implementations even for CPython:
A) you can use the generic implementation, which produces a cross-python-compatible .so. All function calls go through the API at runtime. The same .so works on any version of CPython or PyPy.
B) you can use a different set of headers or a #define or something, and you get a higher-performance version of your unmodified code---with the issue that the .so only runs on the exact version of CPython. This is done by defining some of the functions as macros. I would expect this version to be of similar speed than the current C API in most cases.
This might give a way forward: people would initially port their extensions hoping to use the option B; once that is done, they can easily measure---not guess--- the extra performance costs of the option A, and decide based on actual data if the difference is really worth the additional troubles of distributing many versions. Even if it is, they can distribute an A version for PyPy and for unsupported CPython versions, and add a few B versions on top of that.
This makes sense, but unless it results in PyPy drastically gaining popularity as a production runtime, it basically leaves us in the status quo. We continue to not be able to change CPython internals at all, since that will break people using option B.
Though potentially if we picked an official option for A, we could deprecate the stability of option B (over a few releases) and require people using it to thoroughly test, update and #ifdef their code for each version. That would allow us to make changes to the runtime while preserving option A as the reliable version.
You might want to have a look at https://github.com/Microsoft/xlang/ which is not yet ready for showtime (in particular, there's no "make it look Pythonic" support yet), but is going to extend our existing cross-language ABI to Python (alongside C++/.NET/JS) and non-Windows platforms. It's been in use for years in Windows and has been just fine. (Sample generated output at https://github.com/devhawk/pywinrt-output/tree/master/generated/pyrt/src but the design docs at the first link are probably most interesting.)