On Jul 31, 2018, at 23:23, Stefan Behnel <python_capi@behnel.de> wrote:
Just to make that point clearer, people commonly use Cython for three different main use cases:
a) Wrap native libraries for CPython (and less commonly for PyPy) with a pythonic interface. The cool feature there is that you can write python(-like) code that compiles down into the C layer and runs there, i.e. you can trivially make the wrapper as thin or thick as you want, without sacrificing performance. (This is pretty much the FFI case.)
b) Speed up Python code by compiling and optimising it using C data types. Here, Cython supports a very smooth transition from Python semantics and features down to low-level C speed and semantics, while still writing Python code (or very Python-like code, if you want) all along the way. You can even use Python type annotations for that these days.
c) Write C/C++ code without having to write C code. From the perspective of someone who has to write native code, statically typed Cython code compiles to the expected C code (when disabling the safety belts), but has a much nicer syntax, native access to the complete (C)Python and C ecosystems, and all Python features built into the language. Quite a number of people use it to write the C/C++ code they need but without the complex and error prone syntax.
Thanks for this write-up Stefan. FWIW, although all of these uses cases are important ones which Cython fulfills nicely, none specifically touch on the reason why I think we want a Cython-like tool in the stdlib. c) probably gets the closest though.
The use case for such a tool built into Python is this:
Victor has a goal/plan for improving the performance of CPython by 2x, and there have been numerous attempts at removing the GIL (gilectomy), swapping reference counting for a gc, reducing the use of ABI-locking macros, etc. Most of these get thwarted by the realization that Python's C API will have to change, very likely in a backward incompatible way. And there is a *ton* of C extensions out there.
So let’s say that we decide these are important and achievable goals, but the API breakage necessitates a version bump to Python 4, let’s say in 5 years. The question is: what is the migration plan so that we can minimize the disruption to the extension module ecosystem?
One approach would be to discourage extension module authors from writing extensions directly against the C API, but providing them with the power they need to call into the C API, and marry that with third party C libraries of all ilk, in a higher level language (type annotated Python?) along with a code generator that has the intimate knowledge of the C API. If we get significant buy-in, we can think about large-scale evolution of the C API in sync with evolving the code generation tool. That would be the ideal solution: source code compatibility across C API and runtime changes.
As far as adopting Cython vs. writing a lightweight tool, I think if we were to go down this path, we’d need something that comes with CPython by default. Thus the classic “the stdlib is where code goes to die” conundrum. There’s also the question of whether the Cython syntax is best suited for this use case, and whether the tool is more than we need to accomplish this goal. Even if we had a separate tool, that wouldn’t eliminate the need for Cython, since it still addresses the use cases you mention.
Cheers, -Barry