[IPython-dev] Starting to plan for 0.11 (this time for real)

Brian Granger ellisonbg at gmail.com
Fri Oct 29 14:23:22 EDT 2010

> As for non-blockers, we have:
> - the parallel code is not in a good situation right now: we have a
> few regressions re. the Twisted 0.10.1 code (e.g. the SGE code isn't
> ported yet), the Twisted winhpc scheduler is only in 0.11, and while
> the new zmq tools are looking great, they are NOT production-ready
> quite yet.  In summary, we'll have to warn in bright, blinking pink
> letters 1995-style, everyone who uses the parallel code in production
> systems to stick with the 0.10 series for a little longer.  Annoying,
> yes, but unfortunately such is life.

I want to at least proposed the following solution to this:

Remove all of the twisted stuff from 0.11 and put the new zmq stuff in
place as a prototype.

Here is my logic:

* The Twisted parallel stuff is *already* broken in 0.11 and if anyone
has stable code running on it, they should be using 0.10.
* If someone is happy to run non-production ready code, there is no
reason they should be using the Twisted stuff, they should use the
pyzmq stuff.
* Twisted is a *massive* burden on our code base:
  - For package managers, it brings in Twisted, Foolscap and zope.interface.
  - It makes our test suite unstable and fragile because we have to
run tests in subprocesses and use trial sometimes and nose other
  - It is a huge # of LOC.
  - It means that most of our codebase is Python 3 ready.

There are lots of cons to this proposal:

* That is really quick to drop support for the Twisted stuff.
* We may piss some people off.
* It possibly means maintaining the 0.10 series longer than we imagined.
* We don't have a security story for the pyzmq parallel stuff yet.

I am not convinced this is the right thing to do, but the benefits are



Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the IPython-dev mailing list