[Numpy-discussion] Numeric3

[Peter Verveer]

> "What scipy is" is quite open to interpretation.  The idea of scipy is 
> just to have a super-package where different packages can live and 
> have some kind of inter-operability.   It is made so that sub-packages 
> can be easily added and built upon request.   We try hard to make sure 
> that the only real dependency is scipy_base which should not be hard 
> to install anywhere (does not require ATLAS or FORTRAN).
> It would be great to add nd_image as a sub-package to SciPy.    In 
> fact, nd_image was one of the things that most alarmed me about the 
> emerging split in the community, as it reproduces some of what is 
> already in SciPy (N-D convolution)  while adding some great things 
> that I've always wanted in SciPy (morphology).

What I find unfortunate is that due to the differences between the 
packages, nd_image cannot be compiled for both Numeric and Numarray at 
the moment. I did not forsee that the split between the packages would 
exist for so long. I do however not agree that it is an example of 
duplication of work. The N-D convolutions are based on a 
filter-framework that is an essential part of nd_image, which would 
have to be implemented anyway (e.g. the morphology operators are also 
based on it.) So one should be too quick to judge if something is 
duplication and a wast of resources.

> I'd like to see more of the development work that goes on happen under 
> a common standard, that is all.  SciPy has done a lot of work to 
> provide an infrastructure within which to distribute packages.  The 
> field is built, but people don't seem to want to come.

A common standard is a very good idea. But right now I don't find SciPy 
attractive as a framework, because 1) it is too big and not easily 
installed. 2) it is not very well documented. Thus, I prefer to write 
my package against a smaller code-base, in this case Numarray, but it 
could have also been Numeric. That has the advantage that people can 
install it more easily, while it still can be included in things like 
SciPy if desired.

>  I understand that the Numeric / Numarray dichotomy is a big source of 
> this problem.  That is the whole reason I'm devoting a big portion of 
> my time to the problem right now.

I can only agree with that. Regardless if I want to use SciPy or not, 
or in whatever form, I would like to see this problem go away, so that 
my software can become available for everybody.

> My very ambitious goal with Numeric3 is to replace both Numeric and 
> Numarray (heavily borrowing code from each).  When I say replace them, 
> I'm talking about the array object and umath module.  I'm not talking 
> about the other packages that use them.

I have to admit that I was very sceptical when I read your announcement 
for Numeric3, since I thought it would not be a good idea to have yet 
another array package. However, now it is clearer to me what you mean 
to do, and I think this is in fact exactly how I would like to see it 
happen. To me it would be perfect if there is some array package that 
forms a minimal basis to build on. I would program the nd_image package 
to that and offer the result for inclusion in both Numarray and SciPy, 
and allow it to be installed standalone. But it seems to me that there 
is a danger for the "yet another package" effect to occur. I think I 
will remain sceptical unless you achieve three things: 1) It has the 
most important improvements that numarray  has. 2) It has a good API 
that can be used to write packages that work with Numeric3/SciPy and 
Numarray (the latter probably will not go away). 3) Inclusion in the 
main Python tree, so that it is guaranteed to be available.

> I want those to live under a common standard (i.e. scipy).   It would 
> be fine with me if that common standard include a scipy_lite or 
> something like that which was easy to install, if that is desired.

Jochem Küpper just outlined very well how it could look like: A small 
core, plus a common project with packages at different levels. I think 
it is a very good idea, and probably similar to what SciPy is trying to 
do now. But he suggests an explicit division between independent 
packages: basic packages, packages with external library dependencies 
like FFTW, and advanced packages. Maybe something like that should be 
set up if we get an arraybobject into the Python core.

BTW it was mentioned before that it would be a problem to remove 
packages like LinearAlgebra and FFT from the core Numeric. matplotlib 
was mentioned as an example of a package that depends on them. I think 
that points however to a fundamental problem with matplotlib: why does 
a plotting library need FFTs and linear algebra? So I don't think 
anybody can really argue that things like an FFT should be in a core 
array package.

> But, if the problem is really just availability of binary copies, then 
> why don't we solve that instead of worrying about installation on 
> platforms that nobody uses.   In addition, even though I've developed 
> a lot of scipy, I'm willing to throw it all out or alter it 
> considerably in order to satisfy the demands that exist for package 
> developers to want to put their weight behind something like SciPy.   
> So, don't equate history with an unwillingness to cooperate.

That sounds great. If you go the route of a separate Numeric3 that goes 
into the python core and keep a common (simple) API at both Python and 
C level, then I will support that by rewriting nd_image against it. The 
result should be able to replace the current nd_image package in 
numarray, and hopefully be included in SciPy.

> While I do recognize the value of repeated work.  When it comes to 
> interoperability, single standards seem to work best.  So, the longer 
> this unnecessary division of resources goes on, the farther behind 
> Python will get for doing anything but niche work.

Agreed about the single standard thing. But I am not willing to just 
'join' the SciPy project to achieve it (at least for nd_image). I am 
however very interested in the possibility of writing against a small 
high-quality array package that is included in the pyhton core. That 
would be all the standard I need. If you manage to make SciPy into a 
useful larger standard on top of that, great, more power to all of us!

Peter