[Numpy-discussion] Numpy on Python3

[Dag Sverre Seljebotn]

Pauli Virtanen wrote:
> Mon, 23 Nov 2009 08:58:47 +0100, Sturla Molden wrote:
>   
>> Pauli Virtanen skrev:
>>     
>>> XXX: 3K: numpy.random is disabled for now, uses PyString_* XXX: 3K:
>>> numpy.ma is disabled for now -- some issues
>>>   
>>>       
>> I thought numpy.random uses Cython? Is it just a matter of recompiling
>> the pyx-file?
>>     
>
> The Cython file uses the C-api directly, so we'll need a .h file with the 
> necessary compile-time conditionals.
>
>   
>>>   I remember Dag was working on this a bit: how far did it go?
>>>       
>> Cython's include file numpy.pxd has an ndarray class that extend
>> PyArrayObject with PEP 3118 buffer compatibility.
>>     
>
> Great! I believe I will just steal whatever I can and rewrite it in C -- 
> for now, it seems possible to keep Numpy's core in plain C.
>   
I did sit down with David to learn enough to do this and had a brief 
start on doing it properly for NumPy on SciPy 2009 (with seperate 
testcases and the buffer format string stored directly in the NumPy 
dtype structs on creation). I meant to come back to it in November but 
due to becoming sick etc. etc. that's no longer possible. If nothing 
happens by the mid/end of January I still hope to be able to do this 
then. Feel free to ask any questions about the buffer PEP if you do go 
forward with this as I've used it a lot in Cython (and wrote the 
"implementation on behalf of NumPy" there).

The Cython numpy.pxd does:
 - Temporarily allocate buffers for the format string, this is 
inefficient as NumPy can store them directly in the dtype when the dtype 
is constructed
 - Not support non-native endian (and some other relevant packing 
formats I believe)
 - Not support the string types
 - Not support dtypes with nested sub-arrays within records

What is done: David added some code (at least in some branch) that 
ensures that sizeof(npy_intp) == sizeof(Py_ssize_t) on Python 2.6+. 
(I.e. if that assumption is violated NumPy won't compile, so we're free 
to assume it until the issue of using npy_intp for indices is fixed on a 
more fundamental level in NumPy). This means that the shapes/strides in 
Py_buffer can be directed directly to the dimensions/strides in the 
NumPy array struct (whereas Cython's numpy.pxd has to make a copy on 
some platforms for Python 2.4).

Dag Sverre