[IPython-dev] Qt/Curses interfaces future: results of the weekend mini-sprint (or having fun with 0mq)

[Brian Granger]

All,

As Fernando has summarized our work very well. I too am very excited
about this development.  One thing that I don't think Fernando
mentioned is that stdout/stderr and Out (displayhook) are all handled
asynchronously AND broadcast to all users.

Thus, if you run the following

Py>>> for i in range(10):
  ...   print i
  ...   i**2
  ...   time.sleep(1)
  ...

You get the result asynchronously:

0
Out : 0
Out : None
[then wait 1 second]
1
Out : 1
Out : None
[then wait 1 second]
2
Out : 4
Out : None
[then wait 1 second]
3
Out : 9
Out : None
[then wait 1 second]
4
Out : 16
Out : None
[then wait 1 second]

etc.  If another user is connected to the kernel, They will also
receive these (along with the corresponding input) asynchronously.  In
a terminal based frontend these things are a little bit difficult to
demonstrate, but in a nice GUI frontend, we could imagine a nice a
interface to represent these things.

Cheers,

Brian

PS: Fernando, do you notice that time.sleep(1) (which returns None)
also triggers displayhook?  That is a bit odd.  Do we want to filter
out None from the displayhook?


On Tue, Mar 23, 2010 at 2:01 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi all,
>
> I realize that we have a significant task ahead in sorting some thorny
> issues out to get 0.11 ready for release, especially with respect to
> GUI handling.  But the big refactor of 0.11 starts to give us a clean
> codebase on which new/better interfaces using ipython can be built,
> and for that several people want to  contribute right away, so it's
> best that we build a path forward for those contributions, even if
> some of us still have to finish the 0.11 struggle.
>
> Wendell is looking into curses and Gerardo and Omar in Colombia are
> looking at Qt, and it would be fantastic to be able to get these
> developments moving forward soon.  So Brian and I got together over
> the weekend and  did a design/coding sprint that turned out to be
> surprisingly fun and productive.  This is the summary of those
> results.  I hope this thread can serve as a design brief we'll later
> formalize in the docs and which can be used to plan for a possible
> GSOC submission, for example (which the Qt guys have in mind).
>
> The basic issue we need to solve is the ability to have out-of-process
> interfaces that are efficient, simple to develop,  and that support
> fully asynchronous operation.  In today's ipython, you type code into
> a program that is the same tasked with executing the code,  so that if
> your code crashes, it takes the interface down with it.  So we need to
> have a two-process system where the user-facing client and the kernel
> that executes code live in separate processes (we'll retain a minimal
> in-process interface for embedding,  no worries, but the bulk of the
> real-world use should be in two processes).
>
> We want the user-facing client (be it readline-, curses- or qt-based)
> to remain responsive when the kernel is executing code, and to survive
> a full kernel crash.  So client/kernel need to communicate, and the
> communication should hopefully be possible *even when the kernel is
> busy*, at least to the extent that low-level messaging should continue
> to function even if the kernel is busy with Python  code.
>
> Up until now our engines use Twisted, and the above requirements can
> simply not be met with Twisted (not to mention Twisted's complexity
> and the concern we have with it not being ported soon to py3).  We
> recently stumbled on the 0mq messaging library:
>
> http://www.zeromq.org/
>
> and Brian was able to quickly build a set of Python bindings for it
> (see  link at the 0mq site, I'm writing this offline) using Cython.
> They are fast, we have  full  control over them, and since Cython is
> python-3 compliant, it means we can get a py3 version anytime we need.
>
> 0mq is a really amazing library: I'd only read about it recently and
> only used it for the first time this weekend (I started installing it
> at Brian's two days ago), and I was blown away by it.  It does all the
> messaging in C++ system threads that are 100% Python-threads safe, so
> the library is capable of queuing messages until the Python layer is
> available to handle them.  The api is dead-simple, it's blazingly
> fast, and we were able to get in two intense days a very real
> prototype that solves a number of problems that we were never able to
> make a dent into with Twisted.  Furthermore, with Twisted it was only
> really Brian and Min who ever wrote major amounts of code  for
> Ipython: twisted is really hard to grasp and has layers upon layers of
> abstraction,  making it a very difficult library to  pick up without a
> major commitment.  0mq is exactly the opposite: Brian explained the
> basic concepts to me in a few minutes (I haven't read a single doc
> yet!), we did some queuing tests interactively (by just making objects
> at an ipython prompt) and we then started writing a real prototype
> that now works.  We are very much considering abandoning twisted as we
> move forward and using 0mq for everything, including the distributed
> computing support (while keeping the user-facing apis unchanged).
>
> So what's in this example?  For now, you'll need to install 0mq and
> pyzmq from git; for 0mq, clone the repo at:
>
> git://github.com/sustrik/zeromq2.git
>
> then run
>
> ./autogen.sh
> ./configure --prefix=your_favorite_installation_prefix
> make
> make install
>
> This should give you a fully working 0mq.  Then for the python
> bindings, clone Brian's repo and get the kernel branch:
>
> git clone git://github.com/ellisonbg/pyzmq.git
> cd pyzmq
> git co -b kernel origin/kernel
>
> then
>
> cp setup.cfg.template setup.cfg
>
> and edit setup.cfg to indicate where you put your libraries.  This is
> basically the prefix above with /lib and /include appended.
>
> Then you can do the usual
>
> python setup.py install --prefix=your_favorite_installation_prefix
>
>
> The prototype we wrote lives in examples/kernel.  To play with it,
> open 3 terminals:
>
> - T1: run kernel.py, just leave it there.
> - T2 and T2: run frontend.py
>
> Both T2 and T3 are simple interactive prompts that run python code.
> You can quit them and restart them, type in both of them and they both
> manage the same namespace from the kernel in T1.  Each time you hit
> return, they synchronize with each other's input/output, so you can
> see what each client is sending to the kernel.  In a terminal this is
> done in-line and only when you hit return, but a curses/qt client with
> a real event loop can actually fire events when data arrives and
> display output from other clients as it is produced.
>
> You can 'background' inputs by putting ';' as the last character, and
> you can keep typing interactively while the kernel continues to
> process.  If you type something that's taking a long time, Ctrl-C will
> break out of  the wait but will leave the code running in the
> background (like Ctrl-Z in unix).
>
> This is NOT meant to be production code, it has no ipython
> dependencies at all,  no tab-completion yet,  etc.  It's meant to:
>
> - let us understand the basic design questions,
> - settle on the messaging api between clients and kernel,
> - establish what common api all clients can use: a base to be shared
> by readline/curses/qt clients, on top of which the frontend-specific
> code will go.
>
> So I'd encourage those of you who are interested in this problem to
> have a look and let us know how it goes.  For now the code lives in
> pyzmq because it makes for a great zmq example, but we're almost ready
> to start from it putting real ipython machinery.  For thinking about
> this design though, it's a lot easier to work with a tiny prototype
> that fits in 3 files than to deal with all of ipython's complexity.
>
> Cheers,
>
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com