[IPython-dev] Starting to plan for 0.11 (this time for real)

[Brian Granger]

Min,

On Fri, Oct 29, 2010 at 10:55 PM, MinRK <benjaminrk at gmail.com> wrote:
> This is more agressive than I expected, I'll have to get the new parallel
> stuff in gear.

If you stopped writing great code, we wouldn't be tempted to do crazy
things like this ;-)

> The main roadblock for me is merging work into the Kernels.  I plan to spend
> tomorrow working on getting the new parallel code ready for review, and
> identifying what needs to happen with code in master in order for this to go
> into 0.11.  The only work that needs merge rather than drop-in is in Kernels
> and Session.  I expect that just using the new Session will just be fine
> after a rewview, but getting the existing Kernels to provide what is
> necessary for the parallel code will be some work, and I'll try to identify
> exactly what that will look like.

Are you thinking of having only one top-level kernel script that
handles both the parallel computing stuff and the interactive IPython?
 I think the idea of that is fantastic, but I am not sure we need to
have all of that working to merge your stuff.  I am not opposed to
attempting this before/during the merge, but I don't view it as
absolutely needed.  Also, it may make sense to review your code
standalone first and then discuss merging the kernel and session stuff
with what we already have.

> The main things I know already:
> * Names should change (GH-178). It's really a coincidence that we had just
> one action per socket type, and the parallel code has several sockets of the
> same type, and some actions that can be on different socket types, depending
> on the scheduler.

Yep.

> * Use IOLoop/ZMQStream - this isn't necessarily critical, and I can probably
> do it with a subclass if we don't want it in the main kernels.

At this point I think that zmqstream has stablized enough that we
*should* be using it in the kernel and kernel manager code anyways.  I
am completely fine with this.

> * apply_request. This should be all new code, and shouldn't collide with
> anything.

Ok.

One other point that Fernando and I talked about is actually shipping
the rest of tornado with pyzmq.  I have been thinking more about the
architecture of the html notebook that James has been working on and
it is an absolutely perfect fit for implementing the server using our
zmq enabled Tornado event loop with tornado's regular http handling.
It would also give us ssl support, authentication and lots of other
web server goodies like websockets.  If we did this, I think it would
be possible to have a decent prototype of James' html notebook in
0.11.  What do you think about this Min?  We are already shipping a
good portion of tornado already with pyzmq and the rest is just a
dozen or so .py files (there is one .c file that we don't need for
python 2.6 and up).
Eventually I would like to contribute our ioloop.py and zmqstream to
tornado itself, but I don't think we have to worry about that yet.

Also, moving tornado into pyzmq would allow us to so secure https
connections for the parallel computing client - controller connection.

Cheers,

Brian

> Let me know what I can do to help things along.
> -MinRK
>
> On Fri, Oct 29, 2010 at 20:28, Fernando Perez <fperez.net at gmail.com> wrote:
>>
>> On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger <ellisonbg at gmail.com>
>> wrote:
>> > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in
>> > place as a prototype.
>> >
>> > Here is my logic:
>> >
>> > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone
>> > has stable code running on it, they should be using 0.10.
>> > * If someone is happy to run non-production ready code, there is no
>> > reason they should be using the Twisted stuff, they should use the
>> > pyzmq stuff.
>> > * Twisted is a *massive* burden on our code base:
>> >  - For package managers, it brings in Twisted, Foolscap and
>> > zope.interface.
>> >  - It makes our test suite unstable and fragile because we have to
>> > run tests in subprocesses and use trial sometimes and nose other
>> > times.
>> >  - It is a huge # of LOC.
>> >  - It means that most of our codebase is Python 3 ready.
>> >
>> > There are lots of cons to this proposal:
>> >
>> > * That is really quick to drop support for the Twisted stuff.
>> > * We may piss some people off.
>> > * It possibly means maintaining the 0.10 series longer than we imagined.
>> > * We don't have a security story for the pyzmq parallel stuff yet.
>>
>> I have to say that I simply didn't have Brian's boldness to propose
>> this, but I think it's the right thing to do, ultimately.  It *is*
>> painful in the short term, but it's also the honest approach.  I keep
>> forgetting but Brian reminded me that even the Twisted-based code in
>> 0.11 has serious regressions re. the 0.10.x series, since in the big
>> refactoring for 0.11 not quite everything made it through.
>>
>> The 0.10 maintenance doesn't worry me a whole lot: as long as we limit
>> it to small changes, by now merging them as self-contained pull
>> requests is really easy (as I just did recently with the ones Paul and
>> Tom sent).  And rolling out a new release when the total delta is
>> small is actually not that much work.
>>
>> So I'm totally +1 on this radical, but I think ultimately beneficial,
>> approach.  It's important to keep in mind that doing this will lift a
>> big load off our shoulders, and we're a small enough team that this
>> benefit is significant.  It will let us concentrate on moving the new
>> machinery forward quickly without having to worry about the large
>> Twisted code.  It will also help Thomas with his py3 efforts, as it's
>> one less thing he has to keep getting out of his way.
>>
>> Concrete plan:
>>
>> - Wait a week or two for feedback.
>> - If we decide to move ahead, make a shared branch on the main repo
>> where we can do this work and review it, with all having the chance to
>> contribute while it happens.
>> - Move all twisted-using code (IPython/kernel and some code in
>> IPython/testing) into IPython/deathrow.  This will let anyone who
>> reall wants it find it easily, without having to dig through version
>> control history.  Note that deathrow does *not* make it into official
>> release tarballs.
>>
>> Cheers,
>>
>> f
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com