[IPython-dev] Qt/Curses interfaces future: results of the weekend mini-sprint (or having fun with 0mq)

[Mikhail Terekhov]

On Tue, Mar 23, 2010 at 5:01 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> The basic issue we need to solve is the ability to have out-of-process
> interfaces that are efficient, simple to develop,  and that support
> fully asynchronous operation.  In today's ipython, you type code into
> a program that is the same tasked with executing the code,  so that if
> your code crashes, it takes the interface down with it.  So we need to
> have a two-process system where the user-facing client and the kernel
> that executes code live in separate processes (we'll retain a minimal
> in-process interface for embedding,  no worries, but the bulk of the
> real-world use should be in two processes).
>
> We want the user-facing client (be it readline-, curses- or qt-based)
> to remain responsive when the kernel is executing code, and to survive
> a full kernel crash.  So client/kernel need to communicate, and the
> communication should hopefully be possible *even when the kernel is
> busy*, at least to the extent that low-level messaging should continue
> to function even if the kernel is busy with Python  code.
>
> Up until now our engines use Twisted, and the above requirements can
> simply not be met with Twisted (not to mention Twisted's complexity
> and the concern we have with it not being ported soon to py3).  We
> recently stumbled on the 0mq messaging library:
>
> http://www.zeromq.org/
>
> and Brian was able to quickly build a set of Python bindings for it
> (see  link at the 0mq site, I'm writing this offline) using Cython.
> They are fast, we have  full  control over them, and since Cython is
> python-3 compliant, it means we can get a py3 version anytime we need.
>
> 0mq is a really amazing library: I'd only read about it recently and
> only used it for the first time this weekend (I started installing it
> at Brian's two days ago), and I was blown away by it.  It does all the
> messaging in C++ system threads that are 100% Python-threads safe, so
> the library is capable of queuing messages until the Python layer is
> available to handle them.  The api is dead-simple, it's blazingly
> fast, and we were able to get in two intense days a very real
> prototype that solves a number of problems that we were never able to
> make a dent into with Twisted.  Furthermore, with Twisted it was only
> really Brian and Min who ever wrote major amounts of code  for
> Ipython: twisted is really hard to grasp and has layers upon layers of
> abstraction,  making it a very difficult library to  pick up without a
> major commitment.  0mq is exactly the opposite: Brian explained the
> basic concepts to me in a few minutes (I haven't read a single doc
> yet!), we did some queuing tests interactively (by just making objects
> at an ipython prompt) and we then started writing a real prototype
> that now works.  We are very much considering abandoning twisted as we
> move forward and using 0mq for everything, including the distributed
> computing support (while keeping the user-facing apis unchanged).
>
> IMHO it is a great idea to separate the main IPython engine from the
frontend.
But while implementing an RPC framework over 0mq from ground up should
not be a very difficult task and will definitely bring you a lot of fun,
have you
considered something preexisting like RPyC (http://rpyc.wikidot.com/) for
example. The reason is that IPython already has a lot of useful and exciting
functionality and yet another RPC framework is somewhat too much. Plus,
you don't have to think about these too low level details like communication

protocols, serialization etc.

Regards,
-- 
Mikhail Terekhov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100324/ae8ac683/attachment.html>