[IPython-dev] Message spec draft more fleshed out

[Wendell Smith]

  Hi Fernando,

Thanks for the reply... it's been really nice to see how helpful and 
responsive this team has been!

I'm kinda busy for the next week or two, but then I'll get back to my 
frontend... it still has some issues before integration, and I think I'm 
going to let the API settle a bit before I try and get too involved.

I do still have one more question... why are we doing so much in the 
channel threads? Why not have the channels simply send and receive, not 
even looking at the messages, and have hookups to call methods on some 
object in the main thread that will do the message parsing? That way the 
frontend managers only have to hook up 'message_received' and 
'send_message' between the two threads, and aren't messing about too 
much with those threads... once they make those simple connections 
(through queues, signals, etc.), then one would only have to fill in 
methods on the main API object. That way seems to have two advantages: 
firstly, the inter-thread communication is limited to only 4 or 5 
methods, making thread errors less likely, and secondly, the main API is 
then on one object, clearly in the main thread.

So... I assume there's a good reason for doing everything in the channel 
threads... but I can't see it. Could someone explain?

thanks,
Wendell


On 08/17/2010 09:07 PM, Fernando Perez wrote:
> Hi Wendell,
>
> On Fri, Aug 13, 2010 at 8:40 PM, Wendell Smith<wackywendell at gmail.com>  wrote:
>> As for curses... I've switched to the urwid library, which, by the way,
>> I have already mostly ported to py3k... and the urwid library is set up
>> to use any sort of asynchronous main loop you want, with a basic main
>> loop written into it, a tornado-based main loop, and a select-based
>> mainloop already written, and it's flexible, so one could write a main
>> loop on one's own. Input is non-blocking.
> This is great news, I'm very happy to hear you've taken the py3k lead
> there.  Numpy is just coming out with py3 support, scipy will follow
> soon, and I imagine at that point matplotlib will start worrying about
> py3k.  So we want to have a clear story on that front.
>
>> As for my previous idea, maybe I'm still not understanding, but perhaps
>> we could still have a basic system with a kernelmanager object, a
>> send_receive function, and two queues, in and out. The send_receive
>> function reads messages from the out_queue and sends them, and then
>> receives messages from zmq through the ports and then puts them on the
>> in_queue, never really looking at what messages are coming in and out,
>> just putting them on the queues. The kernel manager object could then be
>> set up exactly as I said before, except that it has an additional
>> method, process_messages, in which it reads a message from the in_queue,
>> determines which method to call, and calls it; the methods do their
>> magic, printing, receiving input, whatever, and some put messages on the
>> out_queue.
>>
>> As I see it, this sounds great: the queues can be from Queue.Queue, and
>> then everything is thread safe, as the send_receive function cold be on
>> one thread, the kernelmanager methods on the other, and the two would
>> interact only through the thread-safe queues. For an asynchronous
>> approach, you have an option to set max_msgs and timeout for both the
>> send_receive function and the process_messages method, and call them
>> alternately with max_msgs = 1, timeout = 0. This then would make the
>> frontend programmer's job easy: all they need to do is get their main
>> loop to frequently call send_receive and kernelmanager.process_messages,
>> with max_num and timeout set appropriately, and then fill in the other
>> methods from the kernel manager to provide output and input.
> Brian already gave you several key details, so I won't repeat too much.
>
> I do agree that having a functional api is desirable, especially to
> make calling some messages easier from code, with the functional api
> ensuring that all fields in the message header are properly filled
> without every caller having to fill every field manually.  So we'll
> need to play with this as we understand better the needs and
> commonalities of all the frontends.
>
> Please do keep us posted of your progress and don't hesitate to ask.
> It's a bit unfortunate in a sense that we have 2 Google Summer of Code
> students, Evan and you simultaneously needing this without Brian, Min
> and I having had the time to complete the architectural cleanup, but
> we'll do our best to minimize the feel of duplicated/wasted effort
> from everyone.
>
> Also, we're being pretty good about hanging out on #ipython in the
> freenode IRC server whenever we work on ipython, so if you have a
> quick question and you see any of us there, don't hesitate to ping.
>
> With my finishing of the syntax work and Brian having just started the
> kernel code, we're getting closer to a common layer.  With a bit of
> patience and a lot of hard work, we should have soon something both
> fun and powerful.
>
> Cheers,
>
> f