[Numpy-discussion] Numeric3

[John Hunter]

>>>>> "Travis" == Travis Oliphant <oliphant at ee.byu.edu> writes:

    Travis> Matplotlib as well is reinventing stuff already available
    Travis> elsewhere.  SciPy has always tried to bring a matlab-like
    Travis> environment to the user.  That's always been my goal.  It
    Travis> is frustrating to see so many with the same goal work so
    Travis> independently of each other.  Sometimes I guess it is
    Travis> necessary so that new ideas can be tried out.  But, it
    Travis> sure seems to happen more often than I would like with
    Travis> Python and Science / Engineering.

True there has been some reinvention, but as Perry pointed out 95% of
what we do on matplotlib is plotting.  The "matlab-like" environment
is created primarily by simply importing symbols from Numeric, MLab,
FFT and so on.  Along the way I've added *a few* analysis functions,
eg matplotlib.mlab's hist, psd, polyfit, but these were usually to
support some plotting feature or just to have some fun with linear
algebra. 

The other thing to emphasize vis-a-vis the matplotlib duplicating
scipy effort discussion is that the matlab-like environment is only
one thing matplotlib tries to do, and in terms of size of the code
base it is a small part of it.  We also provide an API to embed
matplotlib in every major python GUI (WX, GTK, Tk, FLTK, and QT)
including basic event handling.  We support dynamic images, eg for
building data monitoring applications (performance can be an issue
here, since matplotlib is too slow for some applications).  And we try
to appeal, with some success, to non-scientists -- financial plots,
simple pie and bar charts for business types, webapp developers...  I
would like to see it become a comprehensive 2D plotting toolkit, not
just a toolkit for scientists and engineers.

Thus, is scipy the best place for it?

    Travis> Is it ownership that is a concern?  None of us involved
    Travis> with scipy have an alterior motive for trying to bring
    Travis> packages together except the creation of a single standard
    Travis> and infrastructure for scientific computing with Python.

Ownership is a concern, but perhaps not in the sense you mean it.  I
think I pointed out before that when I started looking at plotting
solutions, I seriously considered putting my effort into
chaco/kiva/agg.  I choose not to, because it is a complex package that
was hard for me to get my head around.  I was something of a python
newbie at the time, though I had coded extensively in other languages.
I didn't really get how all the parts worked together, didn't know
much or anything about swig, agg, quartz and freetype, etc.  It's a
daunting code base to walk up to, and I don't say this pejoratively
because it's an amazing piece of work, it's just a fact.

By writing my own plotting library, I owned the code in the sense that
I fully understood it, knew how the pieces fit together, knew how to
extend and fix it.  A newbie walking up to matplotlib today might have
the same reaction I did to chaco -- they might rather roll their own
plotting solution tailored to their own needs.  I call it the curse of
python -- coding in python is so fun and easy that it facilitates if
not encourages you to roll your own solutions.

If that is right, then it really argues for making libraries as
modular as possible, so that people who want to reuse just some part
of the code, eg the chaco freetype engine, or the matplotlib mathtext
engine or W3C font manager, or the scipy constrained optimizer, can do
so.  In practice, as library developers, we want people to use (and
contribute) to our code, not to take out some juicy morsel and use it
as the basis to build a competing project.  So in addition to the
extra work it takes to make a library truly modular (matplotlib
certainly isn't) there may be a disincentive to do it.

But if it is true that new developers will shy away from largish and
complex projects, and I think scipy, chaco and matplotlib are all in
that category, then we might all benefit by making our packages more
modular.  To me, this basically means that there is an easy,
advertised and documented way with examples on how to take a piece of
the library out and plug it into your own project.  I don't know that
any of us really have the time to do it, since everyone I know is
maximally committed already to providing what they provide in
conjunction with their paying work.  Or maybe scipy already does this
and I'm not aware of it.

    Travis> I keep hearing that people like the idea of a small subset
    Travis> of packages, but that is exactly what scipy has always
    Travis> been trying to be.  Perhaps those involved with scipy have
    Travis> not voiced our intentions enough or loud enough.

I think there are two ideas about what scipy should be.  Many view it
as an awesome repository of high performance numerical algorithms that
are array aware.  You seem to view it as an integrated matlab-like
environment, eg with plotting.  I tend toward the former view in part
because I was unsatisfied with the plotting capabilities, but as you
note there are ways around that, and I'm open minded about integration
if that is the best solution.

An integrated matlab like environment, as you suggested earlier, is a
big job, and in my view should be a project that is separate from
scipy *and* matplotlib.  I don't view scipy and matplotlib as
competitors in providing a matlab-like environment, but as two
components that interoperate well and can be used together with other
software to build such an environment.  There is really very little
duplication of effort in scipy and matplotlib, and we both share a
goal of providing a matlab-like environment.  

In my view, building a matlab-like environment should be an effort
separate from scipy, matplotlib, and chaco, which incorporate scipy's
algorithms, matplotlib and/or chaco's 2D vis, mayavi's 3D vis, the
ipython shell, a good code editor, a notebook interface, and a bit
more.  Ie, we should all focus on our core competencies :-) .  A bit
like what envisage appears to be doing but envisage is a framework and
this is an application -- perhaps envisage is the framework that will
build this application.

JDH