[IPython-dev] IPython development: course adjustment required

[Fernando Perez]

Hi all,

after much thought and hand-wringing, Brian and I would like to bring
up an idea for a change of plans in the development process that will
hopefully make us all happier, more efficient, and will lead to more
usable code sooner, which is after all what we want.  Please pitch in
with any comments you may have, ideas, dissent, etc.

The current ipython development process is, we think, far less
efficient than it could be.  Coupled with the fact that we have a
rather small core team, this is leading to sometimes near paralysis.
It is particularly unfair to Ville, who is tasked with maintaining an
actively (and very widely) used project on a codebase with no clear
future, and yet we have in ipython1 a codebase with split-personality
disorder: lovely parallel computing interfaces but a bunch of "we'll
get there soon" components that are all half-finished.

In all this, Brian and Min try to push ip1 forward, Barry Wark works
on Cocoa, Laurent works on the WX sell, Ondrej helps us with docs but
that work is half-repeated in ip0 and ip1, etc.  And in the meantime,
I'm feeling like I'm straddling a chasm (ip0-ip1) that keeps on
growing wider, with the so far unfulfilled promised of building a
bridge across it, and no realistic plan for such a bridge to really be
built any time soon.  I still come back to issues with ip0 on
occasion, and I try to work on ip1, but the split is too much for my
limited time.  This has left me personally disenchanted with the
project, feeling that I'm not really doing anything of quality on any
front.

So our current rethinking is: forge ahead with what we've been calling
IPython0, and begin integrating the various key (and functional)
components of IPython1 into it one by one.  The IPython0/1 split will
end, and we will use all the good pieces of IPython1 to add
functionality to ip0 without losing its current features.  At each 0.X
release we'll clearly document any api breakage that might happen.

This would bring us a number of benefits:

 * We would be forced to only commit _finished_ stuff to ipython0
because otherwise there will be lots of pissed off IPython0 users.

 * We go from a situation where few people are trying our new code, to
one where thousands of people are trying it.

 * If we moved stuff in IPython.* -> IPython.core.*, we could then
move things in ipython1.kernel.* -> IPython.kernel.*.  It one simple
step, _every_ IPython user has parallel computing at their finger
tips.  With our current approach, the parallel computing stuff will
always be "over there" for your average IPython user and won't be used
until ipython1 is a full replacement for ipython0.

 * It would be sooo much simpler to manage one code base and documentation set.

 * It would be much easier to explain to people that state of ipython.
 "IPython is just IPython and now it has parallel computing"  The
ambitious plans for refactoring, notebooks, frontends are underway,
but IPython is still just IPython.

 * The parallel computing stuff would instantly make it into all the
Linux distros.

 * Design problems will emerge much faster as we will always have a
completely working terminal based IPython to test everything on.
Things like -pylab can't be broken, not even for a second.  Whereas in
ipython1, we are a long way off from even thinking about such things.

 * We don't have to handle the questions like "I want to implement
such and such in IPython, should I use ipython0 or ipython1"

 * Our entire combined effort (limited as it may be, we have some
great people on board) can be focused on a single problem, and we can
all trust that we're working together for the same goal.


In retrospect, I take full blame for any mistakes that I pushed for.
While having a clean slate for thinking IPython1 ideas was probably
beneficial, and none of that work is going to be lost (we'll just fold
it piece by piece into the trunk), some of my ideas led to this
paralysis.  Live and learn.

So, any comments?  We'd like to move forward *quickly* with this idea.
We'd try to make a series of much more frequent releases, one for each
key component we add in, so that little by little the baby can
actually grow.   There will be an initial period where I would prepare
the ground in ip0 for the ip1 components to land in while Brian
finishes a couple of things he has in his local branch, but we're
talking a couple of weeks, not years.

Cheers,

f