Numpy/Cython Google Summer of Code project idea
Hi all, after the Scipy/Sage Days 8 meeting, we were all very impressed by the progress made by Cython. For those not familiar with it, Cython: http://www.cython.org/ is an evolved version of Pyrex (which is used by numpy and scipy) with lots of improvements. We'd like to position Cython as the preferred way of writing most, if not all, new extension code written for numpy and scipy, as it is easier to write, get right, debug (when you still get it wrong) and maintain than writing to the raw Python-C API. A specific project along these lines, that would be very beneficial for numpy could be: - Creating new matrix types in cython that match the cvxopt matrices. The creation of new numpy array types with efficient code would be very useful. - Rewriting the existing ndarray subclasses that ship with numpy, such as record arrays, in cython. In doing this, benchmarks of the relative performance of the new code should be obtained. Another possible project would be the addition to Cython of syntactic support for array expressions, multidimensional indexing, and other features of numpy. This is probably more difficult than the above, as it would require fairly detailed knowledge of both the numpy C API and the Cython internals, but would ultimately be extremely useful. Any student interested in this should quickly respond on the list; such a project would likely be co-mentored by people on the Numpy and Cython teams, since it is likely to require expertise from both ends. Cheers, f
Fernando, When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression? I don't want to spend time learning another set of syntax to do something I already know how to do. John On Thu, Mar 6, 2008 at 12:15 PM, Fernando Perez <fperez.net@gmail.com> wrote:
Hi all,
after the Scipy/Sage Days 8 meeting, we were all very impressed by the progress made by Cython. For those not familiar with it, Cython:
is an evolved version of Pyrex (which is used by numpy and scipy) with lots of improvements. We'd like to position Cython as the preferred way of writing most, if not all, new extension code written for numpy and scipy, as it is easier to write, get right, debug (when you still get it wrong) and maintain than writing to the raw Python-C API.
A specific project along these lines, that would be very beneficial for numpy could be:
- Creating new matrix types in cython that match the cvxopt matrices. The creation of new numpy array types with efficient code would be very useful.
- Rewriting the existing ndarray subclasses that ship with numpy, such as record arrays, in cython. In doing this, benchmarks of the relative performance of the new code should be obtained.
Another possible project would be the addition to Cython of syntactic support for array expressions, multidimensional indexing, and other features of numpy. This is probably more difficult than the above, as it would require fairly detailed knowledge of both the numpy C API and the Cython internals, but would ultimately be extremely useful.
Any student interested in this should quickly respond on the list; such a project would likely be co-mentored by people on the Numpy and Cython teams, since it is likely to require expertise from both ends.
Cheers,
f _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
-- John Ollinger University of Wisconsin Waisman Center, T233 1500 Highland Ave Madison, WI 53711 http://brainimaging.waisman.wisc.edu/~jjo/
On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger <ollinger@wisc.edu> wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression?
No. Using Cython or Pyrex eliminates an enormous amount of the boilerplate involved in writing a C extension especially when it comes to implementing types. They are definitely not the same kind of tool that SWIG is. Instead, they are tools to make hand-wrapping less painful, perhaps even pleasant. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger <ollinger@wisc.edu> wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression?
No. Using Cython or Pyrex eliminates an enormous amount of the boilerplate involved in writing a C extension especially when it comes to implementing types. They are definitely not the same kind of tool that SWIG is. Instead, they are tools to make hand-wrapping less painful, perhaps even pleasant.
Has any thought been spent on using boost.python for this ? If so, what were / are the arguments against it ? (http://boost.org/libs/python/doc/tutorial/doc/html/index.html) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
On Thu, Mar 6, 2008 at 7:57 PM, Stefan Seefeld <seefeld@sympatico.ca> wrote:
Robert Kern wrote:
On Thu, Mar 6, 2008 at 7:29 PM, John Ollinger <ollinger@wisc.edu> wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression?
No. Using Cython or Pyrex eliminates an enormous amount of the boilerplate involved in writing a C extension especially when it comes to implementing types. They are definitely not the same kind of tool that SWIG is. Instead, they are tools to make hand-wrapping less painful, perhaps even pleasant.
Has any thought been spent on using boost.python for this ? If so, what were / are the arguments against it ?
Well, boost.python certainly had the goal to make hand-wrapping less painful, but in my experience, it just made it more C++y. If you're heavily invested in C++ and the advanced features of C++ that the Boost project uses, then this might be a win for you. For me, it was just an exchange of unpleasantness. The excessive compile time was the killer for me; I just couldn't develop quickly enough using it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
John Ollinger wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression? Yes, I think this is accurate. The big win, however with using cython is maintainability. Many more people can improve the code. Complicated things can be a bit of a pain to do in Cython yet, and there is a desperate need for multi-dimensional syntax, but I've been impressed with how many people have contributed to the Sage project and wonder if some of that is the fact that much of it is written in Cython. (There are other things the SciPy project can learn from the Sage project, as well).
Best regards, -Travis Oliphant
On Fri, Mar 7, 2008 at 5:22 AM, Travis E. Oliphant <oliphant@enthought.com> wrote:
John Ollinger wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression? Yes, I think this is accurate. The big win, however with using cython is maintainability. Many more people can improve the code. Complicated things can be a bit of a pain to do in Cython yet, and there is a desperate need for multi-dimensional syntax, but I've been impressed with how many people have contributed to the Sage project and wonder if some of that is the fact that much of it is written in Cython. (There are other things the SciPy project can learn from the Sage project, as well).
Like release early, release often? :) Ondrej
Hi John, On Thu, Mar 6, 2008 at 5:29 PM, John Ollinger <ollinger@wisc.edu> wrote:
Fernando,
When I wrap a C package I spend 95% of my time modifying the package to run as a library and 5% of the time hand-wrapping it for python. I find that hand-wrapping makes me more aware of the interface than using swig, so I can recode portions of the library to make the overall package work better. A quick look at the cyrex documentation makes me wonder if it is primarily written for those who haven't written much C-code.. Is this an accurate impression? I don't want to spend time learning another set of syntax to do something I already know how to do.
Others have given you their perspective on this, but for me a huge issue is reducing the bus factor of our code. The python C API isn't particularly easy to use, and code written to it (even assuming an expert who does a perfect job of testing, documenting and describing the code) will inevitably be hard to maintain and extend in the future. Cython is close enough to Python to be fairly easy to pick up, and if anything, the Sage project has proved that such an infrastructure (among other things) encourages contributions and eases the maintainability problem. Cheers f
participants (6)
-
Fernando Perez -
John Ollinger -
Ondrej Certik -
Robert Kern -
Stefan Seefeld -
Travis E. Oliphant