[Numpy-discussion] NumPy re-factoring project

[Dag Sverre Seljebotn]

Sturla Molden wrote:
> Den 11.06.2010 09:14, skrev Sebastien Binet:
>> it of course depends on the granularity at which you wrap and use
>> numpy-core but tight loops calling ctypes ain't gonna be pretty
>> performance-wise.
>>    
> 
> Tight loops in Python are never pretty.
> 
> The purpose of vectorization with NumPy is to avoid tight Python loops.
> 
>> I really think using cython would be a better option.
>> one'd get the python2<->  python3 transition for "free".
>>
>>    
> 
> Perhaps. Cython has nice syntax for PEP 3118 buffers. But there are 
> downsides to this:

IMO I think the syntax is not useful in this setting.

> - Cython is not C. We want the core to be a C library independent of Python.

Cython could fill the role you propose that ctypes takes. The advantage is

  a) (IMO) nicer syntax...well, at least can be less verbose than ctypes

  b) Avoids having to structure the whole API around what can be done in 
Python+ctypes. A pure ctypes+pure C solution would have a tendency to 
push stuff that logically belongs Python-module-side into pure C side, 
which could be a very bad thing.

  c) Possible to iteratively move more and more from C-extension/Cython 
over to C side, rather than a full rewrite in ctypes + C.

There's disadvantages re compilation, but see below.

I believe this is rather long term though. Stuff like this has to be 
done iteratively; one must not rewrite from scratch.

> - Cython depends on CPython.
> - Compiling Cython generated C takes time.
> - Linkage can be a PITA (I use 64-bit Python now, and import libraries 
> for gcc are missing.)

Well, if you can't link CPython extension modules you're going to have 
major problem with about every single scientific Python library out there.

Unless you actually tackle the root of the problem, rewriting NumPy with 
ctypes won't help -- you also need to rewrite matplotlib, mayavi, 
PyTables, SciPy, mpi4py etc. etc. etc. etc. in ctypes + pure C without 
the CPython API.

In other words, I don't think this is a very reasonable argument. (Well, 
perhaps NumPy is in some sense more "basic" than these others.)

Dag Sverre