Project for Cython integration with NumPy
I am going to apply for a Google Summer of Code project about "Developing Cython towards better NumPy integration" (Cython: http://cython.org). Anyone interested in how this is done can have a look at the links below, any feedback is welcome. Unfortunately I don't have much time to spare before the application deadline, so the focus now is only on high-level stuff, the application and so on; thinks like syntax or sorting out the exact supported NumPy features will (unfortunately) have to wait until at least next week. The application I am going to submit (to Python Foundation): http://wiki.cython.org/DagSverreSeljebotn/soc It links to a details page with more concrete information. The specification for the NumPy support itself is here: http://wiki.cython.org/enhancements/numpy As you might be able to see, my thoughts so far has primarily been on how to engineer Cython and not so much about what would be the most convenient NumPy syntax possible - but have a look if you are interested. -- Dag Sverre Seljebotn
I am going to apply for a Google Summer of Code project about "Developing Cython towards better NumPy integration" (Cython: http://cython.org). Anyone interested in how this is done can have a look at the links below, any feedback is welcome.
The application I am going to submit (to Python Foundation): http://wiki.cython.org/DagSverreSeljebotn/soc
I now have time to actively discuss and improve it so any feedback from the NumPy community is greatly appreciated. See especially: http://wiki.cython.org/enhancements/numpy (I have submitted the application to Google, but they have extended the application period by one week, and most of the NumPy specifics are in the Cython wiki anyway). Dag Sverre
Hi Dag On Tue, Apr 1, 2008 at 12:52 AM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
I am going to apply for a Google Summer of Code project about "Developing Cython towards better NumPy integration" (Cython: http://cython.org). Anyone interested in how this is done can have a look at the links below, any feedback is welcome.
The application I am going to submit (to Python Foundation): http://wiki.cython.org/DagSverreSeljebotn/soc
I now have time to actively discuss and improve it so any feedback from the NumPy community is greatly appreciated.
Sorry for only responding now; I was traveling back from Europe. The project your propose has important implications for NumPy. Currently, much of the core is implemented in C and very few people have in-depth knowledge of its inner workings. We therefore have fewer eyes on that code than we'd like. Furthermore, there are only two ways of extending the core: 1) use Python callbacks, which are slow or 2) learn the C API and deal with reference counting -- too much to ask from a scientist who simply wishes to implement a fast algorithm. There are also some other projects, like scipy.ndimage, that could do with a fair bit of refactoring, and this task would be so much easier if Cython could generate the C-level ndarray code; we have precious little developer resources available as it is, and would rather spend it on solving problems than on arranging low-level API calls. Having the following implemented would already be a vast improvement on the current situation: - Array indexing x[1,2] x[1:2,3:4] x[1,...] x[1,:] - Assignment x[1,2] = 3 x[1,...] = 1 ... - Calling array methods: y = x.sum() Doing this would be easier with Travis Oliphant's numpy developer's reference at hand, and, if I recall the page on scipy.org (which is currently down) correctly, he would provide it to you free of charge on request. I am very excited about your proposal, and I really hope it takes off. Let us know if there is anything we can do to help. Best of luck, Stéfan
Hi Dag On Tue, Apr 1, 2008 at 12:52 AM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
I am going to apply for a Google Summer of Code project about "Developing Cython towards better NumPy integration" (Cython: http://cython.org). Anyone interested in how this is done can have a look at the links below, any feedback is welcome.
The application I am going to submit (to Python Foundation): http://wiki.cython.org/DagSverreSeljebotn/soc
I now have time to actively discuss and improve it so any feedback from the NumPy community is greatly appreciated. See especially:
One more comment about the constructor described on the page above. It would be good if we could have the same syntax as the current numpy.ndarray, and then simply call through to the underlying C constructor. We'd also need zeros, empty, ones_like and some other select functions. The goal is to remain as faithful as possible to the original API, so that there is hardly a distinction between Python and Cython. Cheers Stéfan
Hi Dag On Tue, Apr 1, 2008 at 11:06 AM, Dag Sverre Seljebotn <dagss@student.matnat.uio.no> wrote:
One more comment about the constructor described on the page above. It would be good if we could have the same syntax as the current numpy.ndarray, and then simply call through to the underlying C constructor. We'd also need zeros, empty, ones_like and some other select functions. The goal is to remain as faithful as possible to the original API, so that there is hardly a distinction between Python and Cython.
This will be automatic from how Cython operates :-) And that is, I think, what makes all of this great. Objects are Python objects all the time and by default retain all Python implementation, and we only override this implementation when the behaviour is retained.
I'll walk through this code line by line: cdef c_numpy.ndarray(c_numpy.float, 2) x x = exp(4 + numpy.zeros([10, 10], dtype=numpy.float)) y = x + 3 print x[1, 3] print y[3, 5]
Thanks for this example; I didn't notice that you were making a type declaration, so my original comment was not valid.
#3: (x + 3) will treat x as a Python object once again, so y will be an untyped variable referencing an ndarray like normal Python.
I can foresee certain situations under which we can predict the type of the result of operations like this one. Would it be possible to then handle 'y' as an ndarray as well, instead of reverting to Python object calls? Regards Stéfan
I can foresee certain situations under which we can predict the type of the result of operations like this one. Would it be possible to then handle 'y' as an ndarray as well, instead of reverting to Python object calls?
Indeed - plans are underway to add automatic type inference to Cython. It is all about developer resources etc. (the preliminaries can be done in another GSoC project that we hope to get accepted). When/if that happens, a lot of typing will not be necesarry, including this case. This will be an automatic consequence of Cython compiler development and not something that would need to be specifically for the case of NumPy (well, one declares would some more operator and function type signatures for NumPy, and then it would work). Basically one would then only need to declare the type of arguments to the function, while local variables would be inferred automatically. (Getting around this is impossible, though we might add a different syntax for type declaration (using decorators) so that the same code can also be run using the Python interpreter.) Perhaps in a year or so, if the current GSoCs are accepted :-) no promises though. -- Dag Sverre
On Tue, Apr 01, 2008 at 10:48:30PM +0200, Dag Sverre Seljebotn wrote:
though we might add a different syntax for type declaration (using decorators) so that the same code can also be run using the Python interpreter.)
That would be very neat. I can see how you can get around dynamical typing in a very nice way using this. As the pypy projects says, giving a dynamically-typed language to people does not necessarily means they type dynamically-typed code. I must say I like the idea a lot, you are really pushing the concept all the way. Gaël
That would be very neat. I can see how you can get around dynamical typing in a very nice way using this. As the pypy projects says, giving a dynamically-typed language to people does not necessarily means they type dynamically-typed code.
I must say I like the idea a lot, you are really pushing the concept all the way.
We try to focus hard on immediate gains first, but definitely have a plan for going all the way with what we do. BTW the guy applying for the requires GSoC has pypy development experience and we might use some of their stuff. So I'm really hoping we both get GSoCs. -- Dag Sverre
(Getting around this is impossible, though we might add a different syntax for type declaration (using decorators) so that the same code can also be run using the Python interpreter.)
I meant to say: Getting around this is impossible for functions that are exported from a module and can be used from any Python code. However it is possible to automatically infer type of parameters to intra-module calls or inner functions. -- Dag Sverre
participants (3)
-
Dag Sverre Seljebotn
-
Gael Varoquaux
-
Stéfan van der Walt