[IPython-dev] Starting to plan for 0.11 (this time for real)

[MinRK]

This is more agressive than I expected, I'll have to get the new parallel
stuff in gear.

The main roadblock for me is merging work into the Kernels.  I plan to spend
tomorrow working on getting the new parallel code ready for review, and
identifying what needs to happen with code in master in order for this to go
into 0.11.  The only work that needs merge rather than drop-in is in Kernels
and Session.  I expect that just using the new Session will just be fine
after a rewview, but getting the existing Kernels to provide what is
necessary for the parallel code will be some work, and I'll try to identify
exactly what that will look like.

The main things I know already:

* Names should change
(GH-178<http://github.com/ipython/ipython/issues/#issue/178>).
It's really a coincidence that we had just one action per socket type, and
the parallel code has several sockets of the same type, and some actions
that can be on different socket types, depending on the scheduler.
* Use IOLoop/ZMQStream - this isn't necessarily critical, and I can probably
do it with a subclass if we don't want it in the main kernels.
* apply_request. This should be all new code, and shouldn't collide with
anything.


Let me know what I can do to help things along.

-MinRK

On Fri, Oct 29, 2010 at 20:28, Fernando Perez <fperez.net at gmail.com> wrote:

> On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger <ellisonbg at gmail.com>
> wrote:
> > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in
> > place as a prototype.
> >
> > Here is my logic:
> >
> > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone
> > has stable code running on it, they should be using 0.10.
> > * If someone is happy to run non-production ready code, there is no
> > reason they should be using the Twisted stuff, they should use the
> > pyzmq stuff.
> > * Twisted is a *massive* burden on our code base:
> >  - For package managers, it brings in Twisted, Foolscap and
> zope.interface.
> >  - It makes our test suite unstable and fragile because we have to
> > run tests in subprocesses and use trial sometimes and nose other
> > times.
> >  - It is a huge # of LOC.
> >  - It means that most of our codebase is Python 3 ready.
> >
> > There are lots of cons to this proposal:
> >
> > * That is really quick to drop support for the Twisted stuff.
> > * We may piss some people off.
> > * It possibly means maintaining the 0.10 series longer than we imagined.
> > * We don't have a security story for the pyzmq parallel stuff yet.
>
> I have to say that I simply didn't have Brian's boldness to propose
> this, but I think it's the right thing to do, ultimately.  It *is*
> painful in the short term, but it's also the honest approach.  I keep
> forgetting but Brian reminded me that even the Twisted-based code in
> 0.11 has serious regressions re. the 0.10.x series, since in the big
> refactoring for 0.11 not quite everything made it through.
>
> The 0.10 maintenance doesn't worry me a whole lot: as long as we limit
> it to small changes, by now merging them as self-contained pull
> requests is really easy (as I just did recently with the ones Paul and
> Tom sent).  And rolling out a new release when the total delta is
> small is actually not that much work.
>
> So I'm totally +1 on this radical, but I think ultimately beneficial,
> approach.  It's important to keep in mind that doing this will lift a
> big load off our shoulders, and we're a small enough team that this
> benefit is significant.  It will let us concentrate on moving the new
> machinery forward quickly without having to worry about the large
> Twisted code.  It will also help Thomas with his py3 efforts, as it's
> one less thing he has to keep getting out of his way.
>
> Concrete plan:
>
> - Wait a week or two for feedback.
> - If we decide to move ahead, make a shared branch on the main repo
> where we can do this work and review it, with all having the chance to
> contribute while it happens.
> - Move all twisted-using code (IPython/kernel and some code in
> IPython/testing) into IPython/deathrow.  This will let anyone who
> reall wants it find it easily, without having to dig through version
> control history.  Note that deathrow does *not* make it into official
> release tarballs.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101029/f04154a8/attachment.html>