[Numpy-discussion] Numeric3

[Travis Oliphant]

>
> I have in the past tried to install SciPy on OSX and I had big 
> problems with it. Motivated by the recent discussions on the list I 
> tried again today, and found it much improved, but still quite 
> involved. It would still be a pain to distribute software that has a 
> dependency on SciPy to 'average users' that would have trouble 
> installing SciPy. There does not seem to be a binary distribution on 
> scipy.org for OS X, which would solve that problem. This is not 
> unimportant, since I guess OS X is gaining popularity in scientific 
> computing.

A binary for OS X is a desireable thing.  I think more of a stink should 
be made, so that one gets provided by default.  I know that Robert Kern 
has worked on such a thing and it is not out of the question, but is on 
the horizon.   More users asking for it would help.

>>   I'm very glad to hear these comments, as I constantly wonder why 
>> people seem to be duplicating SciPy's efforts instead of helping SciPy.
>
>
> I agree with that, but fact is that the split between Numarray and 
> Numeric prevented me from contributing to SciPy. I have developed for 
> numarray (the image processing package nd_image, part of numarray, is 
> done by me) because I felt numarray was much better then Numeric for 
> that application, and that also ruled out SciPy. There seems now to be 
> an effort going to either try to merge the two or to bring Numeric up 
> to numarray standard. Thats great, and if that works out I will try to 
> rewrite my package so it works with both. Would  be happy to 
> contribute it to SciPy. But I would make sure it remains a package 
> that can be installed separately from  SciPy if required.
>
"What scipy is" is quite open to interpretation.  The idea of scipy is 
just to have a super-package where different packages can live and have 
some kind of inter-operability.   It is made so that sub-packages can be 
easily added and built upon request.   We try hard to make sure that the 
only real dependency is scipy_base which should not be hard to install 
anywhere (does not require ATLAS or FORTRAN). 

It would be great to add nd_image as a sub-package to SciPy.    In fact, 
nd_image was one of the things that most alarmed me about the emerging 
split in the community, as it reproduces some of what is already in 
SciPy (N-D convolution)  while adding some great things that I've always 
wanted in SciPy (morphology).

I believe scipy can be easily installed as a collection of packages if 
that is desired.  Pearu has done an incredible job with scipy_disutils 
to make that process simple.   Making a single super package helps users 
see that they have only a part of what's available (but maybe all they 
need), and you get the added benefit of standard documentation 
efforts.    Nobody has taken up the ball to actually try and distribute 
individual packages of scipy, but it is a welcome effort.   
www.scipy.org has places where such packages could live.  Enthought has 
mainly been interested in distributing "batteries-included" packages for 
windows because it is what their customers need.  Other are welcome to 
pick up the slack.

Perhaps it would be good to get a list of which platforms actually need 
binaries for people to look more favorably at SciPy.   I can see that OS 
X is on the top of that list, surely somebody could distribute a copy of 
a binary for that platform (I know of people who have it installed).     
Also, scipy is still version 0.3 not because the code is unstable but 
because the layout of the packages is still open to change.  Come and 
discuss your visions of what it should look like with us on the 
scipy-dev at scipy.net  mailing list.

> Making SciPy modular is in my opinion a very good approach. Why do I 
> need to install all of it, if I really  only  need a few parts? If I 
> can install only the subset I need, the likelihood that I will have an 
> easier job doing that will increase. That would make it also possible 
> to have sub-sets that are free of dependencies such as FORTRAN.

>
> On the other hand, I would think it would be a good idea to write 
> software that does not  have dependencies on big packages such as 
> SciPy when possible, but only on Numeric/Numarray. That does not 
> exclude them from being included in SciPy. It seems to me that it 
> would be a desirable property if a package can be installed just with 
> Numeric (or numarray), with an appropriate sub-set of SciPy, or come 
> with the big SciPy package. I would rather see a "collection of SciPy 
> packages" then a "SciPy package".

I'd like to see more of the development work that goes on happen under a 
common standard, that is all.  SciPy has done a lot of work to provide 
an infrastructure within which to distribute packages.  The field is 
built, but people don't seem to want to come.    I understand that the 
Numeric / Numarray dichotomy is a big source of this problem.  That is 
the whole reason I'm devoting a big portion of my time to the problem 
right now.  

My very ambitious goal with Numeric3 is to replace both Numeric and 
Numarray (heavily borrowing code from each).  When I say replace them, 
I'm talking about the array object and umath module.  I'm not talking 
about the other packages that use them.  I want those to live under a 
common standard (i.e. scipy).   It would be fine with me if that common 
standard include a scipy_lite or something like that which was easy to 
install, if that is desired.   But, if the problem is really just 
availability of binary copies, then why don't we solve that instead of 
worrying about installation on platforms that nobody uses.   In 
addition, even though I've developed a lot of scipy, I'm willing to 
throw it all out or alter it considerably in order to satisfy the 
demands that exist for package developers to want to put their weight 
behind something like SciPy.   So, don't equate history with an 
unwillingness to cooperate. 

I think you'll find the people involved with scipy to be incredibly 
willing to cooperate to achieve mutual success.  If you have users of a 
package that demand a certain binary of scipy to be installed, then 
bring them over.  I can almost guarantee you that a copy of scipy will 
be available for them. 

While I do recognize the value of repeated work.  When it comes to 
interoperability, single standards seem to work best.  So, the longer 
this unnecessary division of resources goes on, the farther behind 
Python will get for doing anything but niche work. 

-Travis