[IPython-dev] Message spec draft more fleshed out
ellisonbg at gmail.com
Tue Aug 24 00:24:02 EDT 2010
On Tue, Aug 17, 2010 at 9:10 PM, Wendell Smith <wackywendell at gmail.com> wrote:
> Hi Fernando,
> Thanks for the reply... it's been really nice to see how helpful and
> responsive this team has been!
> I'm kinda busy for the next week or two, but then I'll get back to my
> frontend... it still has some issues before integration, and I think I'm
> going to let the API settle a bit before I try and get too involved.
> I do still have one more question... why are we doing so much in the
> channel threads? Why not have the channels simply send and receive, not
> even looking at the messages, and have hookups to call methods on some
> object in the main thread that will do the message parsing?
Currently that is really all that the channels are doing. I don't
think they actually look at the msg formats at all anymore. They used
to do that, but I took that logic out of them.
> That way the
> frontend managers only have to hook up 'message_received' and
> 'send_message' between the two threads, and aren't messing about too
> much with those threads... once they make those simple connections
> (through queues, signals, etc.), then one would only have to fill in
> methods on the main API object. That way seems to have two advantages:
> firstly, the inter-thread communication is limited to only 4 or 5
> methods, making thread errors less likely, and secondly, the main API is
> then on one object, clearly in the main thread.
> So... I assume there's a good reason for doing everything in the channel
> threads... but I can't see it. Could someone explain?
We could do everything in one thread, but that would require a much
more careful thinking about how to integrate our zmq even tloops (that
are currently in the channel threads) with the main event loop on the
GUI. This is a much more subtle and difficult problem that we end up
avoiding with the channel threads. We truly do need event loops for
each channel. We could combine them into 1 event loop, but we like
the idea of keeping them separate, because some clients may want to
only use a subset of the channels.
> On 08/17/2010 09:07 PM, Fernando Perez wrote:
>> Hi Wendell,
>> On Fri, Aug 13, 2010 at 8:40 PM, Wendell Smith<wackywendell at gmail.com> wrote:
>>> As for curses... I've switched to the urwid library, which, by the way,
>>> I have already mostly ported to py3k... and the urwid library is set up
>>> to use any sort of asynchronous main loop you want, with a basic main
>>> loop written into it, a tornado-based main loop, and a select-based
>>> mainloop already written, and it's flexible, so one could write a main
>>> loop on one's own. Input is non-blocking.
>> This is great news, I'm very happy to hear you've taken the py3k lead
>> there. Numpy is just coming out with py3 support, scipy will follow
>> soon, and I imagine at that point matplotlib will start worrying about
>> py3k. So we want to have a clear story on that front.
>>> As for my previous idea, maybe I'm still not understanding, but perhaps
>>> we could still have a basic system with a kernelmanager object, a
>>> send_receive function, and two queues, in and out. The send_receive
>>> function reads messages from the out_queue and sends them, and then
>>> receives messages from zmq through the ports and then puts them on the
>>> in_queue, never really looking at what messages are coming in and out,
>>> just putting them on the queues. The kernel manager object could then be
>>> set up exactly as I said before, except that it has an additional
>>> method, process_messages, in which it reads a message from the in_queue,
>>> determines which method to call, and calls it; the methods do their
>>> magic, printing, receiving input, whatever, and some put messages on the
>>> As I see it, this sounds great: the queues can be from Queue.Queue, and
>>> then everything is thread safe, as the send_receive function cold be on
>>> one thread, the kernelmanager methods on the other, and the two would
>>> interact only through the thread-safe queues. For an asynchronous
>>> approach, you have an option to set max_msgs and timeout for both the
>>> send_receive function and the process_messages method, and call them
>>> alternately with max_msgs = 1, timeout = 0. This then would make the
>>> frontend programmer's job easy: all they need to do is get their main
>>> loop to frequently call send_receive and kernelmanager.process_messages,
>>> with max_num and timeout set appropriately, and then fill in the other
>>> methods from the kernel manager to provide output and input.
>> Brian already gave you several key details, so I won't repeat too much.
>> I do agree that having a functional api is desirable, especially to
>> make calling some messages easier from code, with the functional api
>> ensuring that all fields in the message header are properly filled
>> without every caller having to fill every field manually. So we'll
>> need to play with this as we understand better the needs and
>> commonalities of all the frontends.
>> Please do keep us posted of your progress and don't hesitate to ask.
>> It's a bit unfortunate in a sense that we have 2 Google Summer of Code
>> students, Evan and you simultaneously needing this without Brian, Min
>> and I having had the time to complete the architectural cleanup, but
>> we'll do our best to minimize the feel of duplicated/wasted effort
>> from everyone.
>> Also, we're being pretty good about hanging out on #ipython in the
>> freenode IRC server whenever we work on ipython, so if you have a
>> quick question and you see any of us there, don't hesitate to ping.
>> With my finishing of the syntax work and Brian having just started the
>> kernel code, we're getting closer to a common layer. With a bit of
>> patience and a lot of hard work, we should have soon something both
>> fun and powerful.
> IPython-dev mailing list
> IPython-dev at scipy.org
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com
More information about the IPython-dev