[IPython-dev] Curses Frontend

[Wendell Smith]

Thanks again for your response!

OK, I have very little experience with twisted, and none with 0MQ, and 
would very much appreciate if someone could give me some help there.

My plan so far is:
1. Figure out whether to use curses or urwid
2. Make the necessary widgets/etc. for curses/urwid, completely 
independent of ipython (mainly, a nice text editing box that can handle 
coloring, a main scrollable text box for the output, and a pop-up window 
for completions)
3. Figure out how to handle coloring - pygments, pycolorize, etc. 
Ipython's built in colorizer is not good enough: firstly, curses doesn't 
take terminal escapes, and secondly, I want to colorize the input as one 
types, which requires handling unfinished input - which the built in 
lexer can't handle. It looks like pygments is the way to go, and I'll 
need to write a formatter for curses (pygments doesn't have one, and 
curses is a special case, anyway)
4. Start working on integration with ipython. While of course ipython 
will be on my mind for the above, I would like the widgets to be sort of 
their own thing, that could be used independently of ipython, for 2 
reasons: firstly, someone might find them useful anyway, and secondly, 
it should make it easier to do bug fixes and isolate bugs.

I hope to get #1 done today or tomorrow, and get  back to #2. I've 
already gotten a good start on a curses based text-editing, 
color-capable widget, but if urwid looks good, I may drop it and go for 
theirs.

A couple of questions/problems:
1. Curses likes to block while waiting for input - is this ok? Should I 
try and get around this?
2. Colorizing. Someone already mentioned that they were thinking of 
switching ipython over to pygments. I could do this - it would make 
things easier for me in the long run. If I don't go that route, then it 
gets really complicated. As coloring the input pretty much requires 
pygments (built-in lexer can't handle unfinished input), there then 
become several options:
     a. Require pygments for the curses frontend.
     b. Don't require pygments, but be completely colorless without it.
     c. Write some fancy way of using the built in colorizer to handle 
the output, as well as one for pygments for the input. That way, without 
pygments installed, there is still color in the output portion.
I would, of course, prefer to just convert all ipython to pygments - 
which I think I can do - but of course that's not my decision to make, 
I'm new here. But if that is a good way to go, I'll happily go there.
Otherwise, I'm leaning towards (b). It's barely more complicated than 
(a) while having advantages over (a), and (c) is just too complicated.

OK, that was a lot. I'm going to get back to looking at urwid, but if 
anyone else has any answers/recommendations/comments/etc., I would love 
to hear them...

-Wendell


On 03/12/2010 08:49 PM, Brian Granger wrote:
> Wendell,
>
> I have been busy with other things lately, so sorry I haven't gotten
> back to you sooner.  I wanted to say a few more things about the
> overall design of your curses based frontend.  This also applies to
> other frontends...
>
> * For a long time we have wanted to move IPython to a 2 process model
> that is similar to how Mathematica works.  The idea is to have 1) a
> simple lightweight frontend GUI/terminal/curses based process that
> handles user input, prints output, etc and 2) a kernel/engine process
> that actually executes the code.
>
> * The GIL has made this effort very difficult to achieve.  The problem
> is that if the kernel executes extension code that doesn't release the
> GIL, the kernel's will appear dead for the networking.  Threads don't
> help this and Twisted (which we use) doesn't either.
>
> * Our solution so far has been to use Twisted and introduce an
> additional process called the "controller" that manages traffic
> between the frontend and kernel/engine.  Twisted is amazing in many
> ways, but it has a number of downsides:
>
> - Twisted tends to be an all or nothing thing.  Thus, it has been
> quite difficult to integrate with other approaches and it is also
> difficult to *force* everyone to use Twisted.  Also, using Twisted
> forces us to have a very constrained threading model that is super
> infelxible.  I can give more details on this if needed.
> - It looks like Twisted will make the transition to Python 3k *very*
> slowly (after zope, after everyone drops python 2.x support, etc.).
> IPython, on the other hand needs to move to Python 3k quickly as so
> many people use it.
> - As we have spent more time looking at network protocols, we are more
> and more convinced that a pure RPC style network approach is really
> insufficient.  What we really need is more of an asynchronous
> messaging architecture that supports different messaging topologies.
> - Twisted is slow for how we are using.  For some of the things we
> would like to do, we need C/C++ speed networking.
>
> So.....for the last while we have been looking at alternatives.
> Initially we were hopeful about AMQP:
>
> http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol
>
> AMQP looks really nice, but 1) is very complex and heavyweight, 2)
> adopts a very specific model of messaging that looks to be too limited
> for what we want.
>
> Recently, however, we learned of 0MQ:
>
> http://www.zeromq.org/
>
> Zeromq has been developed by some of the same folks as AMQP, but with
> a different emphasis:
>
> * As fast as can be (written in C++).  For somethings it is faster
> than raw TCP sockets!
> * Super simple API, wire protocol, etc.
> * Lightweight and easy to install.
> * All the network IO and message queuing are done in a C++ thread, so
> it can happen without the GIL being involved.
>
> This last point is the most important thing about 0MQ.  It means that
> a process can run non-GIL releasing extension code and still do all
> the network IO and message queuing.
>
> I have tested out 0MQ quite a bit and have created new Cython based
> bindings to 0MQ:
>
> http://www.zeromq.org/bindings:python
>
> Here is a Py0MQ example of what an IPython kernel would look like:
>
> http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/
>
> So....my thoughts right now are that this is the direction we are
> headed.  Thus, I think the model you should use in designing the
> frontend is this:
>
> * Cureses/urwid based frontend that ....
> * Talks to IPython kernel over...
> * 0MQ
>
> Obviously, you are free to use a 1 process model where Ipython runs in
> the same process as the curses stuff, but you will run into all the
> same problems that we have had for years.  I can ellaborate further if
> you want.  What do you think about this plan?
>
> Cheers,
>
> Brian
>