[Python-ideas] discouraging direct use of the C-API

[Nathaniel Smith]

On May 7, 2015 10:24 AM, "Stefan Behnel" <stefan_ml at behnel.de> wrote:
>
> Paul Moore schrieb am 07.05.2015 um 10:47:
>
> > (I refrained from adding scipy and numpy to that list, as that would
> > make this post seem like a troll attempt, which it isn't, but has
> > anyone thought of the implications of a recommendation like this on
> > those projects? OK, they'd probably just ignore it as they have a
> > genuine need for direct use of the C API, but we would be sending
> > pretty mixed messages).
>
> Much of scipy and its surrounding tools and libraries are actually written
> in Cython. At least much of their parts that interact with Python, and
> often a lot more than just the interface layer. New code in the scientific
> computing community is commonly written in Cython these days, or uses
other
> tools for JIT or AOT compilation (Numba, numexpr, ...), many of which were
> themselves partly written in Cython.

Yeah, I think if anyone talks to the developers of those libraries they
will get a very *un*mixed message saying, don't do what we did :-). One of
scipy's GSoC projects this year is even porting a c extension to Cython,
and I've been actively investigating the possibility of porting numpy into
Cython as well. Mostly for the immediate benefits, but certainly it has
occurred to me that in the long run this could potentially provide an
escape hatch from CPython. (Numerical people are *very* interested in
JITs... and something like Cython provides the unique possibility that if a
project like PyPy or pyston added direct support for the language, then one
could write a single source file that was fast on cpython b/c it compiled
to C, and was even faster on other interpreters because the same source got
jitted.)

The main obstacle to porting numpy, btw, is that Cython currently assumes
that each source file will generate one python extension, and any
communication between source files will be via python-level imports. NumPy,
of course, has 100,000 lines of C across lots of files that are all built
into one extension module, and which happily communicate via direct C
function calls. So incrementally porting is impossible without teaching
Cython to handle this case a bit better. NumPy is an extreme outlier in
this regard though. In particular this is absolutely not a reason to steer
*new* projects away from Cython.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150507/fda2b995/attachment.html>