[IPython-dev] IPython development news and prospects

[Fernando Perez]

Hi all,

I wanted to get a bit of feedback from others on dev plans, as well as
update everyone on a few things.  This is necessarily a long email,
sorry about that.

IPython continues to grow, thanks to the hard work of many (myself
notably excluded), but it's obvious to all of us that the current
codebase has problems and is hard to extend cleanly.  I think we
finally have both the manpower and the pieces in place to do something
about it.  First of all, there's enough of us working on the project
that I think we can tackle more changes, and second I'll have more
time soon to devote to some of this effort, as I'll be able to
directly use more of it in my research.

Next month I'm switching jobs, and at my new job we'll be using enough
of ipython's capabilities (the trunk as well as the parallel work)
that I'll be able to spend some more time on it. I know Brian will
also have soon more time for it, which is fantastic, and we've also
picked up new developers.  All this means that I expect this year to
make a lot of progress.  Ville and the rest of the team have done a
fantastic job keeping the main project alive while Brian and Min did
the distributed computing work, but I think we're at the point where
we can join forces to push through a fully integrated, well structured
project.

For this to be successful, there are several areas that will change,
so that the project evolves in a more sustainable way.

- The first is source control.  All this work will likely require a
fair amount of messing around with multiple parts of the codebase, by
multiple people.  Hence I'm thinking this is the time to switch out to
a DVCS.  I know we've talked about it, and it's something that seems
to be "in the air".  By this I mean that right now, the same
discussion is going on in the Emacs-dev, python-dev, matplotlib-dev
and numpy/scipy-dev lists.  I honestly don't want to get dragged into
a git/hg/bzr debate here, *all* of the above lists have to some extent
done that already.

My proposal is very simple: we go with hg.  Other related projects to
us are already using it (Sage & sympy), I know some of you already use
it yourselves, some of the MPL devs personally use it, I've used it
and liked it, and Robert Kern already uses it.  To me, that's enough
to let me make a choice.  It really seems like *both* bzr and hg are
pretty good, even if they have small differences here and there.  I
think we'll benefit a lot more from having *A* dvcs than from having
yet another 70-email-long thread on the subject.

So the question is: does anyone here *fundamentally object* to
switching out to hg for the project?

If not, then the big remaining issue is Trac support and SVN
migration.  If we all agree on the above, I'll look into how to make
that part happen, but I'd be *extremely grateful* if any of you has
already done this and offers a bit of help.

- Next comes documentation.  I'll work on transferring the manual over
to a bunch of reST files, so we can have reST based manual(s).  After
we do this, we will hopefully start doing a very, very good job of
always documenting things correctly in the reST sources, rather than
having the hodgepodge of outdated manual + random wiki pages we have
today.

Again, anyone who has done a lyx/latex->reST conversion in the past
and is willing to pitch in with help, let me know.

- Testing: we'll just go with nose.  All the ipython1 code that needs
Twisted has tests (Brian and Min have done a stellar job here) that
use twisted's test support to properly manage the reactor, but
fortunately they work out of the box with Twisted.  Nose makes it
really simple to have little snippets get picked up as tests, so this
means that we can have easier test development, and continue to use
the twisted test support where reactor management is needed.

At some point, we may need to turn some of the fancier doctest support
that SnakeOil has into a nose plugin to support pure ipython
functionality.  We'll cross that bridge when we get to it.

- Configuration: for this, we've more or less decided to just going
with TConfig:

http://projects.scipy.org/ipython/ipython/browser/ipython1/trunk/sandbox/tconfig

The only downside tconfig has is a dependence on Enthought Traits,
which has C code.  The upsides are vast, including a *clean* way of
handling many of the configuration-related problems that pop up here
all the time.  We can have a separate discussion thread for this topic
later if you want, because I think it's the only potentially
contentious one and we don't want to ram it down anyone's throat.


Having the above in place, what's left is actual code and project
evolution.  In the ipython1 subproject we already have the beginnings
of a cleaner separation of modular components for the interactive
component.  Eric Jones, Gael and I have taken (short) stabs at this,
and as of March I expect to spend some serious time on getting this
running.  Laurent's recent work would much more naturally fit here.

If any of you (in addition to Gael and Laurent) is in or near Paris
March 22/23, please let us know.  Gael and I are tentatively planning
a Paris IPython sprint for those dates.  If it's just the three of us
the logistics are one thing, but if more developers can come we can
look into a different setup.


What I'd like to know is if Ville and the others who have a stake on
trunk are willing to basically join in together, starting mid to late
March, on this effort.  My idea is that we'll eventually have ALL the
functionality that ipython trunk has today (a lot of which Ville has
shepherded), though some of the APIs may necessarily change.

But there will be enough work to be done here, and it touches on what
all have contributed so much, that I want to find the best way to
manage a transition so that:

- nobody feels like their work is dropped/ignored

- we don't end up with *two* projects.

This is still just one project, that will have multiple components but
all of which work in concert to provide various levels of
functionality (a terminal-based shell, a gui-based shell, a
javascript-based one, a remote shell, a parallel computing system, an
ide-like system for scientific work, etc).

Python is growing a lot, and in the scientific arena it's really,
really becoming a major player fast.  I think if we round up our joint
effort together, with the proper ground infrastructure, we can provide
something that will be tremendously useful to lots of people.  I'd
like to hear your thoughts on the matter.

If there's broad agreement on the above, we can plan specifics in
subsequent emails (hg transition, a 'final release plan' to put trunk
in maintenance mode with a good view for when we'll get the new system
functioning, etc).

I'd like to close by thanking all of you who donate your time and
energy to ipython.  It started as a 'quick afternoon hack' back in
2001 when I wanted an excuse to not write my dissertation and was
tired of Perl. Years later we have something that's used in major
research labs around the world, included in all the major Linux
distros, and shipped in every laptop from the OLPC program to kids
everywhere.  It's really been a great run so far, one that couldn't
have happended without all of your help.  Now I think we can make the
future of the project even better!

Regards,

f