[Cython] Rewriting/compiling parts of CPython's stdlib in Cython

Stefan Behnel stefan_ml at behnel.de
Tue Mar 22 08:59:04 CET 2011


Robert Bradshaw, 22.03.2011 08:14:
> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote:
>> there seems to be quite some interest in a project to get parts of CPython
>> and specifically its stdlib rewritten in Cython. [...]
>> In short, we have strong supporters, but Guido has understandable doubts
>> against a new (and quite large) dependency and potential semantic
>> deviations.
>
> Reading the list, I think others on the list overestimate the semantic
> differences. Mostly we're talking about things like is vs. equality
> for floating point numbers and tracebacks (at least for un-annotated
> code).

I think so too, that's what I tried to make clearer with my last reply. I 
think Cython is actually pretty close to Python semantics overall, and 
almost all deviations are explicitly triggered by type annotations in the code.


> It's a valid point that Cython is still under such active development.

Absolutely. Eventually, we'd have to settle on a specific version for the 
compiler used in the stdlib, and support that at least as long as the 
CPython version that uses it.


>> It would be helpful to get support from the side of external distributors
>> that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If
>> they agreed to test the Cython generated stdlib modules in their
>> distributions, we could get user feedback that would allow python-dev to
>> take a well founded decision.
>>
>> Do we have any volunteers for trying this out? Both on the side of
>> distributors and implementors?
>
> I think Sage might be willing to give it a try. I'll ask tomorrow as
> part of a talk I'm giving.

Cool.


> Note that due to the way Python's import
> mechanism works, it would be easy (as a first pass) to make a
> "cythonize this Python install" which would just compile (a subset of)
> the .py files and drop .so files next to them. This would require no
> messing with the Python build system or distribution, easy to test and
> benchmark, and be easy to clean up.

Sure, I implemented that in pyximport ages ago, when Cython was really far 
from being able to compile much in the stdlib. We should totally give it a 
try once Vitja's branch is in shape.


>> At the current state of affairs, the implementation could still be financed
>> by a Python backed GSoC project, although it would be cool if more users
>> could just step up and simply try to compile and optimise stdlib modules
>> (preferably without major changes to the code). It's certainly a great way
>> to show off your Cython skills :). I gave it a try with difflib and it
>> turned out to be quite easy.
>>
>> http://blog.behnel.de/index.php?p=155
>>
>> Reimplementing existing C modules in Cython might, however, be more
>> interesting for python-dev, but also be a larger undertaking. So a GSoC
>> might be worth running on that.
>
> I think that's a great idea. Would you be willing to mentor such a project.

As usual, I'm not sure I'll have the time, but if no-one else steps up, I'd 
consider it.

Stefan


More information about the cython-devel mailing list