[IPython-dev] Qt/Curses interfaces future: results of the weekend mini-sprint (or having fun with 0mq)

[Barry Wark]

Congratulations Brian and Fernando! This is a huge advance for UI
integration (and possible for parallel ipython as well).

Having written several two-process UI/kernel systems, I suspect that
the direction things are heading will make it quite easy to implement
a UI frontend for the kernel using modern UI toolkits (e.g. Qt, Cocoa,
WPF, etc.)

I suppose this new paradigm brings to the fore the ongoing discussion
of protocol for communication between frontend and the kernel. As this
could affect format for a persistent "notebook" format (it would be
nice to be able to send a notebook or notebook section to the kernel),
it might be worth considering the two issues together. The previous
discussion settled, I think, leaning towards an XML notebook. Assuming
an entity that describes a block of python code, the UI->kernel
message could be an XML-serialized block and the response could be the
corresponding XML-serialized output. The other possibility is to
separate notebook format (XML) from UI<->kernel protocol. In this
case, something like Google's protocol buffers make sense. These
protocol buffer messages are somewhat easier (and much faster) to work
with from many languages than XML, but are not as easily human
readable (if at all) and would add yet an other non-stdlib dependency.
Just starting the discussion...

Looking forward to hacking some UIs,
Barry



On Tue, Mar 23, 2010 at 3:05 PM, Brian Granger <ellisonbg at gmail.com> wrote:
> All,
>
> As Fernando has summarized our work very well. I too am very excited
> about this development.  One thing that I don't think Fernando
> mentioned is that stdout/stderr and Out (displayhook) are all handled
> asynchronously AND broadcast to all users.
>
> Thus, if you run the following
>
> Py>>> for i in range(10):
>  ...   print i
>  ...   i**2
>  ...   time.sleep(1)
>  ...
>
> You get the result asynchronously:
>
> 0
> Out : 0
> Out : None
> [then wait 1 second]
> 1
> Out : 1
> Out : None
> [then wait 1 second]
> 2
> Out : 4
> Out : None
> [then wait 1 second]
> 3
> Out : 9
> Out : None
> [then wait 1 second]
> 4
> Out : 16
> Out : None
> [then wait 1 second]
>
> etc.  If another user is connected to the kernel, They will also
> receive these (along with the corresponding input) asynchronously.  In
> a terminal based frontend these things are a little bit difficult to
> demonstrate, but in a nice GUI frontend, we could imagine a nice a
> interface to represent these things.
>
> Cheers,
>
> Brian
>
> PS: Fernando, do you notice that time.sleep(1) (which returns None)
> also triggers displayhook?  That is a bit odd.  Do we want to filter
> out None from the displayhook?
>
>
> On Tue, Mar 23, 2010 at 2:01 PM, Fernando Perez <fperez.net at gmail.com> wrote:
>> Hi all,
>>
>> I realize that we have a significant task ahead in sorting some thorny
>> issues out to get 0.11 ready for release, especially with respect to
>> GUI handling.  But the big refactor of 0.11 starts to give us a clean
>> codebase on which new/better interfaces using ipython can be built,
>> and for that several people want to  contribute right away, so it's
>> best that we build a path forward for those contributions, even if
>> some of us still have to finish the 0.11 struggle.
>>
>> Wendell is looking into curses and Gerardo and Omar in Colombia are
>> looking at Qt, and it would be fantastic to be able to get these
>> developments moving forward soon.  So Brian and I got together over
>> the weekend and  did a design/coding sprint that turned out to be
>> surprisingly fun and productive.  This is the summary of those
>> results.  I hope this thread can serve as a design brief we'll later
>> formalize in the docs and which can be used to plan for a possible
>> GSOC submission, for example (which the Qt guys have in mind).
>>
>> The basic issue we need to solve is the ability to have out-of-process
>> interfaces that are efficient, simple to develop,  and that support
>> fully asynchronous operation.  In today's ipython, you type code into
>> a program that is the same tasked with executing the code,  so that if
>> your code crashes, it takes the interface down with it.  So we need to
>> have a two-process system where the user-facing client and the kernel
>> that executes code live in separate processes (we'll retain a minimal
>> in-process interface for embedding,  no worries, but the bulk of the
>> real-world use should be in two processes).
>>
>> We want the user-facing client (be it readline-, curses- or qt-based)
>> to remain responsive when the kernel is executing code, and to survive
>> a full kernel crash.  So client/kernel need to communicate, and the
>> communication should hopefully be possible *even when the kernel is
>> busy*, at least to the extent that low-level messaging should continue
>> to function even if the kernel is busy with Python  code.
>>
>> Up until now our engines use Twisted, and the above requirements can
>> simply not be met with Twisted (not to mention Twisted's complexity
>> and the concern we have with it not being ported soon to py3).  We
>> recently stumbled on the 0mq messaging library:
>>
>> http://www.zeromq.org/
>>
>> and Brian was able to quickly build a set of Python bindings for it
>> (see  link at the 0mq site, I'm writing this offline) using Cython.
>> They are fast, we have  full  control over them, and since Cython is
>> python-3 compliant, it means we can get a py3 version anytime we need.
>>
>> 0mq is a really amazing library: I'd only read about it recently and
>> only used it for the first time this weekend (I started installing it
>> at Brian's two days ago), and I was blown away by it.  It does all the
>> messaging in C++ system threads that are 100% Python-threads safe, so
>> the library is capable of queuing messages until the Python layer is
>> available to handle them.  The api is dead-simple, it's blazingly
>> fast, and we were able to get in two intense days a very real
>> prototype that solves a number of problems that we were never able to
>> make a dent into with Twisted.  Furthermore, with Twisted it was only
>> really Brian and Min who ever wrote major amounts of code  for
>> Ipython: twisted is really hard to grasp and has layers upon layers of
>> abstraction,  making it a very difficult library to  pick up without a
>> major commitment.  0mq is exactly the opposite: Brian explained the
>> basic concepts to me in a few minutes (I haven't read a single doc
>> yet!), we did some queuing tests interactively (by just making objects
>> at an ipython prompt) and we then started writing a real prototype
>> that now works.  We are very much considering abandoning twisted as we
>> move forward and using 0mq for everything, including the distributed
>> computing support (while keeping the user-facing apis unchanged).
>>
>> So what's in this example?  For now, you'll need to install 0mq and
>> pyzmq from git; for 0mq, clone the repo at:
>>
>> git://github.com/sustrik/zeromq2.git
>>
>> then run
>>
>> ./autogen.sh
>> ./configure --prefix=your_favorite_installation_prefix
>> make
>> make install
>>
>> This should give you a fully working 0mq.  Then for the python
>> bindings, clone Brian's repo and get the kernel branch:
>>
>> git clone git://github.com/ellisonbg/pyzmq.git
>> cd pyzmq
>> git co -b kernel origin/kernel
>>
>> then
>>
>> cp setup.cfg.template setup.cfg
>>
>> and edit setup.cfg to indicate where you put your libraries.  This is
>> basically the prefix above with /lib and /include appended.
>>
>> Then you can do the usual
>>
>> python setup.py install --prefix=your_favorite_installation_prefix
>>
>>
>> The prototype we wrote lives in examples/kernel.  To play with it,
>> open 3 terminals:
>>
>> - T1: run kernel.py, just leave it there.
>> - T2 and T2: run frontend.py
>>
>> Both T2 and T3 are simple interactive prompts that run python code.
>> You can quit them and restart them, type in both of them and they both
>> manage the same namespace from the kernel in T1.  Each time you hit
>> return, they synchronize with each other's input/output, so you can
>> see what each client is sending to the kernel.  In a terminal this is
>> done in-line and only when you hit return, but a curses/qt client with
>> a real event loop can actually fire events when data arrives and
>> display output from other clients as it is produced.
>>
>> You can 'background' inputs by putting ';' as the last character, and
>> you can keep typing interactively while the kernel continues to
>> process.  If you type something that's taking a long time, Ctrl-C will
>> break out of  the wait but will leave the code running in the
>> background (like Ctrl-Z in unix).
>>
>> This is NOT meant to be production code, it has no ipython
>> dependencies at all,  no tab-completion yet,  etc.  It's meant to:
>>
>> - let us understand the basic design questions,
>> - settle on the messaging api between clients and kernel,
>> - establish what common api all clients can use: a base to be shared
>> by readline/curses/qt clients, on top of which the frontend-specific
>> code will go.
>>
>> So I'd encourage those of you who are interested in this problem to
>> have a look and let us know how it goes.  For now the code lives in
>> pyzmq because it makes for a great zmq example, but we're almost ready
>> to start from it putting real ipython machinery.  For thinking about
>> this design though, it's a lot easier to work with a tiny prototype
>> that fits in 3 files than to deal with all of ipython's complexity.
>>
>> Cheers,
>>
>> f
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>