[Matrix-SIG] advocacy

[Joe Harrington]

I too have been searching for a free data analysis environment.  I
currently use the abominable IDL.  For the average scientist, IDL is
still the best that's out there, and this makes me glum.  Python has
the promise to be *much* more, but it's not there yet.  Mostly this is
a problem with the add-ons, not the language itself.  When I tried
doing a few projects with Numeric, here's what I ran into:

Plotting capability is primitive compared to IDL, better plotting
	might be available but is tedious/difficult to integrate, and
	there are licensing issues.
The breadth of numerical routines is small, and most have very
	limited implementations.
Data types I use (FITS) are poorly supported, if at all.
To do anything, you need to do a lot of legwork pulling together
	disparate packages and integrating them into Python.
Documentation is almost nonexistent -- basically you must use the
	force, read the source.
Things are very disorganized; different packages offer overlapping
	routines that don't work together.

To a hacker interested in Python, much of the above is not a big deal.
Just do the legwork, read the source code comments for documentation,
write what you need.  To most scientists, this is unacceptable.
They'd rather support a commercial organization they don't generally
like (RSI) to make a poor language like IDL have a slick appearance, a
broad, well-implemented set of numerical and plotting routines, and
good docs.  It's the lesson of Windows vs. Mac: to most people the
core system doesn't matter, the applications do.

A year and a half ago, Paul Barrett and I put in a proposal to NASA
with a number of list members as Collaborators.  The proposal would
have covered some programmers' and doc writers' salaries to make and
distribute a coherent package, but it wasn't selected for funding.

Given these problems, I sadly put Python aside, though I've been
lurking for the past year in hopes of seeing a change or catching an
opportunity to do something.  Today's digest contained both your
message and the announcement of some FITS routines (I wish Paul
Barrett would release his class library!).  Perhaps this is a good
time to step back and ask the community a few questions about where it
wants Numerical Python to go.

It's great to have the flexibility of modules, but to be successful
among non-hacker scientists there has to be a "full version" with all
the packages built, a well-organized structure, good docs, plotting
capability, and support for all popular data formats.  The folks at
LLNL have the basics going, but progress at the current rate won't get
NumPy to compete with IDL (or several others) any time soon.  This is
not at all intended as a criticism of LLNL or the list's current level
of activity.  Making a fleshed-out package will be a lot of work, and
will require some real commitment and a lot of late hours from a lot
of people.  It would mean real coordination, people volunteering time
and being tasked with sometimes boring jobs, establishing a means to
make overall design decisions and sticking with them, etc.  Are list
members willing and interested in an organized development effort
aimed at creating a competitor to IDL?

--jh--
Joe Harrington
Cornell University Space Sciences