[Numpy-discussion] Numpy's future again (was OpenMP ...)

[Stéfan van der Walt]

On Mon, Mar 24, 2008 at 6:37 PM, David Cournapeau
<david at ar.media.kyoto-u.ac.jp> wrote:
>  That's one of the reason why I was thinking about a gradual move of most
>  "core functionalities of the core" toward a separate C library, with a
>  simple and crystal clear interface, without any reference to any python
>  API, just plain C with plain pointers. We could then force this core,
>  "pure" C library to be used only through dereferencing of an additional
>  pointer, thus enabling dynamic change of the actual functions (at least
>  when numpy is started).
>
>  I have to say I really like the idea of more explicit separation between
>  the actual computation code and the python wrapping; it can only help if
>  we decide to write some of the wrappers in Cython/ctypes/whatever
>  instead of pure C as today. It has many advantages in terms of
>  reliability, maintainability and performance (testing performance would
>  be much easier I think, since it could be done in pure C).

I like the suggestion of pulling out the core functionality into a
separate library.  David mentions that it would help if (meaning
"when", right :) we move over to Cython for the core/ufuncs/wrappers.
Cython code would certainly alleviate the pressure on the few
developers on board, by providing us with a) an easier interface to
the internals (read: more contributions) and b) fewer bugs cropping
up, especially with regards to reference counting.  Besides that, it
also has a number of fantastic features, including introspection of
compiled extensions (no, I'm serious), made possible by heavy
annotation of the generated code.

Given that, I think the numpy community should watch the following
Google SOC project very carefully (ndimage developers also take note):

  http://wiki.cython.org/DagSverreSeljebotn/soc/details

It also looks like a significant amount of Cython activity will take
place during the SAGE developers' days 1:

  http://wiki.sagemath.org/dev1

We can only benefit from close interaction with their group, so it
would be worth popping in to discuss this part of the project (please
let us know if you're going so that we can further exchange ideas).

I am very glad to see all the interest in this thread (that I just
broke, sorry) regarding optimisation of different parts of the numpy
codebase (I hope my concerns regarding the status of the test coverage
didn't make you believe otherwise).  As soon as we release 1.0.5 we'll
be in a position to switch to the nose testing framework, which will
simplify the expansion of our test base significantly.  In turn, that
will allow us to further explore these suggestions in terms of
patches, rather than mere discussions.

I'm tempted to also talk about the idea for a wiki-to-source-roundtrip
for documentation system, but I should stop before I get carried away.

Happy hacking,
Stéfan