[IPython-dev] Qt/Curses interfaces future: results of the weekend mini-sprint (or having fun with 0mq)

[Fernando Perez]

Hi all,

I realize that we have a significant task ahead in sorting some thorny
issues out to get 0.11 ready for release, especially with respect to
GUI handling.  But the big refactor of 0.11 starts to give us a clean
codebase on which new/better interfaces using ipython can be built,
and for that several people want to  contribute right away, so it's
best that we build a path forward for those contributions, even if
some of us still have to finish the 0.11 struggle.

Wendell is looking into curses and Gerardo and Omar in Colombia are
looking at Qt, and it would be fantastic to be able to get these
developments moving forward soon.  So Brian and I got together over
the weekend and  did a design/coding sprint that turned out to be
surprisingly fun and productive.  This is the summary of those
results.  I hope this thread can serve as a design brief we'll later
formalize in the docs and which can be used to plan for a possible
GSOC submission, for example (which the Qt guys have in mind).

The basic issue we need to solve is the ability to have out-of-process
interfaces that are efficient, simple to develop,  and that support
fully asynchronous operation.  In today's ipython, you type code into
a program that is the same tasked with executing the code,  so that if
your code crashes, it takes the interface down with it.  So we need to
have a two-process system where the user-facing client and the kernel
that executes code live in separate processes (we'll retain a minimal
in-process interface for embedding,  no worries, but the bulk of the
real-world use should be in two processes).

We want the user-facing client (be it readline-, curses- or qt-based)
to remain responsive when the kernel is executing code, and to survive
a full kernel crash.  So client/kernel need to communicate, and the
communication should hopefully be possible *even when the kernel is
busy*, at least to the extent that low-level messaging should continue
to function even if the kernel is busy with Python  code.

Up until now our engines use Twisted, and the above requirements can
simply not be met with Twisted (not to mention Twisted's complexity
and the concern we have with it not being ported soon to py3).  We
recently stumbled on the 0mq messaging library:

http://www.zeromq.org/

and Brian was able to quickly build a set of Python bindings for it
(see  link at the 0mq site, I'm writing this offline) using Cython.
They are fast, we have  full  control over them, and since Cython is
python-3 compliant, it means we can get a py3 version anytime we need.

0mq is a really amazing library: I'd only read about it recently and
only used it for the first time this weekend (I started installing it
at Brian's two days ago), and I was blown away by it.  It does all the
messaging in C++ system threads that are 100% Python-threads safe, so
the library is capable of queuing messages until the Python layer is
available to handle them.  The api is dead-simple, it's blazingly
fast, and we were able to get in two intense days a very real
prototype that solves a number of problems that we were never able to
make a dent into with Twisted.  Furthermore, with Twisted it was only
really Brian and Min who ever wrote major amounts of code  for
Ipython: twisted is really hard to grasp and has layers upon layers of
abstraction,  making it a very difficult library to  pick up without a
major commitment.  0mq is exactly the opposite: Brian explained the
basic concepts to me in a few minutes (I haven't read a single doc
yet!), we did some queuing tests interactively (by just making objects
at an ipython prompt) and we then started writing a real prototype
that now works.  We are very much considering abandoning twisted as we
move forward and using 0mq for everything, including the distributed
computing support (while keeping the user-facing apis unchanged).

So what's in this example?  For now, you'll need to install 0mq and
pyzmq from git; for 0mq, clone the repo at:

git://github.com/sustrik/zeromq2.git

then run

./autogen.sh
./configure --prefix=your_favorite_installation_prefix
make
make install

This should give you a fully working 0mq.  Then for the python
bindings, clone Brian's repo and get the kernel branch:

git clone git://github.com/ellisonbg/pyzmq.git
cd pyzmq
git co -b kernel origin/kernel

then

cp setup.cfg.template setup.cfg

and edit setup.cfg to indicate where you put your libraries.  This is
basically the prefix above with /lib and /include appended.

Then you can do the usual

python setup.py install --prefix=your_favorite_installation_prefix


The prototype we wrote lives in examples/kernel.  To play with it,
open 3 terminals:

- T1: run kernel.py, just leave it there.
- T2 and T2: run frontend.py

Both T2 and T3 are simple interactive prompts that run python code.
You can quit them and restart them, type in both of them and they both
manage the same namespace from the kernel in T1.  Each time you hit
return, they synchronize with each other's input/output, so you can
see what each client is sending to the kernel.  In a terminal this is
done in-line and only when you hit return, but a curses/qt client with
a real event loop can actually fire events when data arrives and
display output from other clients as it is produced.

You can 'background' inputs by putting ';' as the last character, and
you can keep typing interactively while the kernel continues to
process.  If you type something that's taking a long time, Ctrl-C will
break out of  the wait but will leave the code running in the
background (like Ctrl-Z in unix).

This is NOT meant to be production code, it has no ipython
dependencies at all,  no tab-completion yet,  etc.  It's meant to:

- let us understand the basic design questions,
- settle on the messaging api between clients and kernel,
- establish what common api all clients can use: a base to be shared
by readline/curses/qt clients, on top of which the frontend-specific
code will go.

So I'd encourage those of you who are interested in this problem to
have a look and let us know how it goes.  For now the code lives in
pyzmq because it makes for a great zmq example, but we're almost ready
to start from it putting real ipython machinery.  For thinking about
this design though, it's a lot easier to work with a tiny prototype
that fits in 3 files than to deal with all of ipython's complexity.

Cheers,

f