[IPython-dev] Message spec draft more fleshed out

[Fernando Perez]

Hi Wendell,

On Fri, Aug 13, 2010 at 8:40 PM, Wendell Smith <wackywendell at gmail.com> wrote:
> As for curses... I've switched to the urwid library, which, by the way,
> I have already mostly ported to py3k... and the urwid library is set up
> to use any sort of asynchronous main loop you want, with a basic main
> loop written into it, a tornado-based main loop, and a select-based
> mainloop already written, and it's flexible, so one could write a main
> loop on one's own. Input is non-blocking.

This is great news, I'm very happy to hear you've taken the py3k lead
there.  Numpy is just coming out with py3 support, scipy will follow
soon, and I imagine at that point matplotlib will start worrying about
py3k.  So we want to have a clear story on that front.

> As for my previous idea, maybe I'm still not understanding, but perhaps
> we could still have a basic system with a kernelmanager object, a
> send_receive function, and two queues, in and out. The send_receive
> function reads messages from the out_queue and sends them, and then
> receives messages from zmq through the ports and then puts them on the
> in_queue, never really looking at what messages are coming in and out,
> just putting them on the queues. The kernel manager object could then be
> set up exactly as I said before, except that it has an additional
> method, process_messages, in which it reads a message from the in_queue,
> determines which method to call, and calls it; the methods do their
> magic, printing, receiving input, whatever, and some put messages on the
> out_queue.
>
> As I see it, this sounds great: the queues can be from Queue.Queue, and
> then everything is thread safe, as the send_receive function cold be on
> one thread, the kernelmanager methods on the other, and the two would
> interact only through the thread-safe queues. For an asynchronous
> approach, you have an option to set max_msgs and timeout for both the
> send_receive function and the process_messages method, and call them
> alternately with max_msgs = 1, timeout = 0. This then would make the
> frontend programmer's job easy: all they need to do is get their main
> loop to frequently call send_receive and kernelmanager.process_messages,
> with max_num and timeout set appropriately, and then fill in the other
> methods from the kernel manager to provide output and input.

Brian already gave you several key details, so I won't repeat too much.

I do agree that having a functional api is desirable, especially to
make calling some messages easier from code, with the functional api
ensuring that all fields in the message header are properly filled
without every caller having to fill every field manually.  So we'll
need to play with this as we understand better the needs and
commonalities of all the frontends.

Please do keep us posted of your progress and don't hesitate to ask.
It's a bit unfortunate in a sense that we have 2 Google Summer of Code
students, Evan and you simultaneously needing this without Brian, Min
and I having had the time to complete the architectural cleanup, but
we'll do our best to minimize the feel of duplicated/wasted effort
from everyone.

Also, we're being pretty good about hanging out on #ipython in the
freenode IRC server whenever we work on ipython, so if you have a
quick question and you see any of us there, don't hesitate to ping.

With my finishing of the syntax work and Brian having just started the
kernel code, we're getting closer to a common layer.  With a bit of
patience and a lot of hard work, we should have soon something both
fun and powerful.

Cheers,

f