[IPython-dev] Development plans update

Fernando Perez fperez.net at gmail.com
Wed Jan 30 02:57:10 EST 2008

Hi all,

I'd like to drop a quick update on the current plans, as well as
propose a few specific steps to take from here on.  It was great to
hear so much enthusiasm coming from different corners, so it's obvious
that we must now make good use of such 'capital', lest we lose it to
other projects :)

- First, an update on the HG situation: Ryan Earl, from Enthought, has
kindly agreed to help up set up hg on their servers, with a likely
timeline of mid- to late-February.  In the meantime we'll continue
operating on our existing SVN, but I'm sure it will be very handy to
have hg once we start making deeper changes and keeping more parallel
lines of development running.

The plan is to initially only set up hg without any Trac coupling, so
we'll continue to use Trac for tickets but will simply point to the hg
changeset URLs instead of having the convenience of integrated
source/ticket management.    This is a small, and hopefully temporary
loss that I'm sure we can live with.  Full Trac integration remains in
the plans, but for later as time permits.  We may simply modify the
Trac config to stop showing the 'browse source' part of the tree to
avoid confusing new visitors.

I spent a few hours today familiarizing myself with hg in more depth,
and I think we can start with a work model where we serve multiple hg
repos via hgwebdir, and set the server permissions so that current
devs can commit to them.  We have a small and well behaved enough team
that I don't foresee having to implement more granular controls, since
each of us can just touch only what we're supposed to at any point in
time.  These repos will serve as the 'official' ones for the public to
download, contribute patches, etc.  Obviously with hg being fully
distributed, anyone can simply set up a publicly visible repo to offer
material for experimental development, and that type of work is
absolutely welcome.  But I think that having an official repo (or
group thereof) is a good idea so that newcomers can know what to use
from the start.

Some references on the matter for the curious:


Comments on hg-based workflows are welcome.

- Next, what to do: we've seen many new ideas coming around and offers
for collaboration, so I think our key priority is to organize things a
bit for that to happen smoothly.  Up until now we've had basically two
lines of work proceeding in parallel: the trunk, dutifully maintained
by Ville and the ipython1 work, mostly done by Brian and Min.  I'm not
sure what exactly I've been doing :)

Our idea has always been that the work in ipython1 would eventually
absorb the trunk, since there's no question whatsoever about ever
dropping the command-line ipython we use today as a tool we offer.
But with finite resources, and the trunk continuing to evolve, I think
we might make more definitive progress by following a slightly
different strategy.  What I'd like to see happen, if all the devs are
interested, would be to all focus over the next few months on actually
pushing out a 'new ipython' that would be developed off the current
ipython1/ codebase, but trying to incorporate, as fast as possible,
all the existing functionality.

Specifically, this is my proposal:

1. We push out an 0.8.3 release with a fair amount of care.  We
actually go through Trac and find anything we can reasonably expect to
fix and try to close it.  This would likely be the 'end of life' of
the series.  I'm willing to participate on this work directly, since I
know the codebase pretty well, but obviously it's not a decision I'll
push without the agreement of Ville, Jorgen, Walter and the rest of
the team.

2. Once this release is out, we really pretty much stop working on the
trunk altogether.  *All of us*. We put it in pure maintenance mode,
and only fix critical bugs.  New feature requests get tickets made for
them, but their fulfillment will actually happen in the new series.

3. We all start working on getting the code in ipython1/ in a usable
form.  I expect by the time we get here, the hg repos will be up, so
we can partition the work in logical components that various teams can
work on.   I'll outline roughly what those pieces are below, but this
message is getting long already, so I won't go into too much detail
yet.  If this plan is agreeable to all, I'll probably start putting
these ideas as reST docs in the codebase itself, so we can discuss
them as design documents.

The basic system is a core capable of doing what ipython does today in
a terminal, but with a cleanly defined API for all of its public
functionality, and subcomponents well separated.  The raw draft of
this is already in place:


The first order of business would be, using this, to get working:

- a terminal-based, single-process ipython, much like today.
- in-terminal GUI support like our -Xthread options of today.
- a gui embedded one, likely Laurent's WX tool.

If the above goals are met, likely it will be because the right
abstractions will have been made (things like separating prompt and
readline handling from stdin/out, etc).

At this point, we'd have specific projects that are either already
functioning but without a 'true ipython backend' or that don't exist

- All of the distributed computing facilities that Brian and I are
very interested in, and for which he's already working on here, in
addition to all the existing ipython1 machinery:

- A JavaScript notebook, for which we already have also prototype code:

- The Qt/Cocoa frontends and other ideas that some of you have
expressed interest in.

- The Leo integration work that has been recently discussed.

- End-user interactive functionality like a better help system (Steve
Roger's project),

- Integration with Enthought's Envisage

- Etc.

In order to achieve all this, the basic guidelines we have in mind are:

- We'll try to keep compatibility with existing APIs (like ipapi)
where feasible, but there are no promises.  We'll break backwards
compatibility wherever we need to in order to do the right thing.

- No untested code.  Between nose, doctests and Twisted's trial (for
the network parts), we have more than enough for properly testing new

- Good docstrings using epydoc/reST for everything.

- A reST-based set of manuals from which we can produce at any time
HTML and PDF documentation.  I'll do the work of exporting the current
lyx user manual out to reST, but that will be just a start.

I'd like to make a release 1.0 of ipython once the 'first order of
business' items above are reasonably met, even if we have a few
functionality regressions on the current terminal-based ipython.
After that, as the various components and subprojects improve, we'll
continue releasing.  This release would also include at least all of
the existing ipython1 distributed computing code, since that already
works quite well for many uses.

Do these sound like reasonable steps to all of you, especially to the
other devs?  If so, we can get going on Trac ticket triage for the
0.8.3 release as best we all can.  I should add that my
moving/changing jobs will make me more useless than usual for the next
few weeks, which is why at least I wanted to get this message out.
I'll do my best for this period, though I'm sure that won't be much.
But once I'm settled in I intend to commit more time to this effort,
since the distributed computing aspects of it will be immediately
needed in my research.

Ok, I better stop.  Thanks to anyone still reading :)  That was long...



More information about the IPython-dev mailing list