[SciPy-dev] Re: [SciPy-user] Future directions for SciPy in light of meeting at Berkeley

[John Hunter]

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> It would seem that while the scipy conference demonstrates
    Travis> a continuing and even increasing use of Python for
    Travis> scientific computing, not as many of these users are scipy
    Travis> devotees.  Why?

Hi Travis,

I like a lot of your proposal, and I want to throw a couple of
additional ideas into the mix.  There are two ideas about what scipy
is: a collection of scientific algorithms and a general purpose
scientific computing environment.  On the first front, scipy has been
a great success; on the second, less so. I think the following would
be crucial to make such an effort a success (some of these are just
restatements of your ideas with additional comments)

  * Easy to install: - it would be probably be important to have a
    fault-tolerant install so that even if a component fails, the
    parts that don't depend on that can continue.  Matthew Knepley's
    build system might be an important tool to make this work right
    for source installs, rather than trying to push distutils too
    hard.

  * A package repository and a way of specifying dependencies between
    the packages and allow automated recursive downloads ala apt-get,
    yum, etc....  So basically we have to come up with a package
    manager, and probably one that supports src as well as binary
    installs.  Everyone knows this is a significant problem in python,
    and we're in a good place to tackle it in that we have experience
    distributing complex packages across platforms which are a mixture
    of python/C/C++/FORTRAN, so if we can make it work, it will
    probably work for all of python.  I think we would want
    contributions from people who do packaging on OSX and win32, eg
    Bob Ippolito, Joe Cooper, Robert Kern, and others.

  * Transparent support for Numeric, numarray and Numeric3 built into
    a compatibility layer, eg something like matplotlib.numerix which
    enables the user to be shielded from past and future changes in
    the array package.  If you and the numarray developers can agree
    on that interface, that is an important start, because no matter
    how much success you have with Numeric3, Numeric 23.x and numarray
    will be in the wild for some time to come.  Having all the major
    players come together and agree on a core interface layer would be
    a win.  In practice, it works well in matplotlib.numerix.

  * Buy-in from the developers of all the major packages that people
    want and need to have the CVS / SVN live on a single site which
    also has mailing lists etc.  I think this is a possibility,
    actually; I'm open to it at least.

  * Good tutorial, printable documentation, perhaps following a "dive
    into python" model with a "just-in-time" model of teaching the
    language; ie, task oriented.


A question I think should be addressed is whether scipy is the right
vehicle for this aggregation.  I know this has been a long-standing
goal of yours and appreciate your efforts to continue to make it
happen.  But there is a lot of residual belief that scipy is hard to
install, and this is founded in an old memory that refuses, sometimes
irrationally, to die, and in part from people's continued
difficulties.  If we make a grand effort to unify into a coherent
whole, we might be better off with a new name that doesn't carry the
difficult-to-install connotation.  And easy-to-install should be our
#1 priority.

Another reason to consider a neutral name is that it wouldn't scare
off a lot of people who want to use these tools but don't consider
themselves to be scientists.  In matplotlib, there are people who just
want to make bar and pie charts, and in talks I've given many people
are very happy when I tell them that I am interested in providing
plotting capabilities outside the realm of scientific plotting.

This is obviously a lot to bite off but it could be made viable with
some dedicated effort; python is like that.  Another concern I have,
though, is that it seems to duplicate a lot of the enthought effort to
build a scientific python bundle -- they do a great job already for
win32 and I think an enthought edition for linux and OSX are in the
works.  The advantage of your approach is that it is modular rather
than monolithic.  To really make this work, I think enthought would
need to be on board with it.  Eg mayavi2 and traits2 are both natural
candidates for inclusion into this beast, but both live in the
enthought subversion tree.  Much of what you describe seems to be
parallel to the enthought python, which also provides scipy, numeric,
ipython, mayavi, plotting, and so on.

I am hesitant to get too involved in the packaging game -- it's really
hard and would take a lot of work.  We might be better off each making
little focused pieces, and let packagers (pythonmac, fink, yum,
debian, enthought, ...) do what they do well.  Not totally opposed,
mind you, just hesitant....

JDH