[pypy-dev] Questions on the pypy+numpy project

[Stefan Behnel]

Alex Gaynor, 17.10.2011 18:14:
> On Mon, Oct 17, 2011 at 12:01 PM, Stefan Behnel wrote:
>> Maciej Fijalkowski, 17.10.2011 17:46:
>>   - pypy's numpy *will* integrate in some sort of way with existing
>>> C/fortran libraries, but this way *will* be different than current
>>> CPython C API. It's really just too hard to get both.
>>
>> Why reinvent yet another wheel when you could make Cython a common language
>> to write extensions and wrapper code for both? Even if that requires a few
>> feature restrictions for Cython users or adaptations to their code to keep
>> it portable, it's still better than forcing users into a complete vendor
>> lock-in on both sides.
>
> There's no fundamental objection to Cython, but there are practical ones.

I'm very well aware of that. There are both technical and practical issues. 
I didn't hide the fact that the Python+ctypes backend for Cython is quite 
far from being ready for use, for example.


>   a) Most of NumPy isn't Cython, so just having Cython gives us little.

There has been the move towards a smaller core for NumPy, and we perceive 
substantial interest, both inside and outside of the Scientific Python 
community, in writing new wrapper code in Cython and even in rewriting 
existing code in Cython to make it more maintainable. Even generated 
wrappers were and are being rewritten, e.g. to get rid of SWIG. Rewriting 
several hundred to thousand lines of C code in Cython can often be done 
within a few days, depending on test coverage and code complexity, and from 
what we hear, this is actually being done or at least seriously considered 
in several projects. It's helped by the fact that CPython users do not have 
to make the switch right away, but can often migrate or add a module at a time.

I agree that simply supporting Cython is not going to magically connect 
huge amounts of foreign code to PyPy. It just makes it a lot easier to get 
closer to that goal than by inventing yet another way of interfacing that 
is not supported by anything else.

Also note that there isn't just NumPy. A relatively large part of Sage is 
written in Cython, for example, especially those parts that glue the rest 
together, which consists of huge amounts of C, C++ and Fortran code. After 
all, Cython's predecessor Pyrex has been around for almost ten years now.


>  b)
> Is the NumPy on Cython house in order?  AFAIK part of the MS project
> involved rewriting parts of NumPy in Cython and modularising Cython for
> targets besides CPython.  And that this was *not* merged.  For me to be
> convinced Cython is a good target, I'd need belief that there's an interest
> in it being a common platform, and when I see that there's work done, by
> core developers, which sits unmerged (with no timeline) I can't have faith
> in that.

I understand that objection. The Cython project is largely driven by the 
interest of core developers and users (now, how unexpected is that?), and 
none of the developers currently uses IronPython or PyPy. So, while we'd 
like to see Cython support other targets (and the core developers agree on 
that goal), there isn't really a strong incentive for ourselves to move it 
into that direction. It's a bit of a chicken and egg problem - why support 
other platforms that no-one uses it for, and who'd use it on a platform 
that's not as well supported as CPython?

I'd personally like to get the ctypes backend merged, but it's not exactly 
in a state that is ready-to-merge soonish. There's a branch, and Romain 
(our GSoC student for the project) is still working on it, but obviously 
with much less time for it, so I'm sure he could use another helping hand.

https://github.com/hardshooter/CythonCTypesBackend

The IronPython port is a different beast. It ran almost completely in cloak 
mode, outside of the scope of the core developers, and it's neither clear 
what the exact design goals were, nor what was eventually achieved or in 
what status the code branch currently is. The project itself died from 
sudden lack of interest on the side of the financial supporters (MS) at 
some point, and it appears that there is currently no-one who can easily 
take it over. Sad, but really nothing to blame the Cython developers for. 
I'd be happy to see it revived, if there is any interest.

https://bitbucket.org/cwitty/cython-for-ironpython/overview

Stefan