[IPython-dev] Message spec draft more fleshed out

Fri Aug 13 23:40:17 EDT 2010

  Dear Brian,

Thank you for your answer and your patience... you're really helping me 
understand this, and I appreciate it!

As for curses... I've switched to the urwid library, which, by the way, 
I have already mostly ported to py3k... and the urwid library is set up 
to use any sort of asynchronous main loop you want, with a basic main 
loop written into it, a tornado-based main loop, and a select-based 
mainloop already written, and it's flexible, so one could write a main 
loop on one's own. Input is non-blocking.

As for my previous idea, maybe I'm still not understanding, but perhaps 
we could still have a basic system with a kernelmanager object, a 
send_receive function, and two queues, in and out. The send_receive 
function reads messages from the out_queue and sends them, and then 
receives messages from zmq through the ports and then puts them on the 
in_queue, never really looking at what messages are coming in and out, 
just putting them on the queues. The kernel manager object could then be 
set up exactly as I said before, except that it has an additional 
method, process_messages, in which it reads a message from the in_queue, 
determines which method to call, and calls it; the methods do their 
magic, printing, receiving input, whatever, and some put messages on the 
out_queue.

As I see it, this sounds great: the queues can be from Queue.Queue, and 
then everything is thread safe, as the send_receive function cold be on 
one thread, the kernelmanager methods on the other, and the two would 
interact only through the thread-safe queues. For an asynchronous 
approach, you have an option to set max_msgs and timeout for both the 
send_receive function and the process_messages method, and call them 
alternately with max_msgs = 1, timeout = 0. This then would make the 
frontend programmer's job easy: all they need to do is get their main 
loop to frequently call send_receive and kernelmanager.process_messages, 
with max_num and timeout set appropriately, and then fill in the other 
methods from the kernel manager to provide output and input.

This is a bit simplified... we might need prioritized queues, or 
send_receive might need to add tags to the message to say which channel 
they came in on... but that shouldn't be difficult, if the rest of it 
makes sense.

Would this work? If not, what am I missing?

Thank you again for your patience,
Wendell

On 08/13/2010 07:16 PM, Brian Granger wrote:
> Wendell,
>
> On Fri, Aug 13, 2010 at 3:04 PM, Wendell Smith<wackywendell at gmail.com>  wrote:
>>   Hello all,
>>
>> I have been looking all these documents over, and wondering if perhaps
>> we could have some object (descended from KernelManager) that would be
>> constructed to perfectly match the message spec, such that any message
>> received would translate to a functional call (i.e.
>> MessageManager.execute(self, header, code, silent=False)), to make it
>> easy for someone to simply write an object that descends from
>> MessageManager (or whatever we call it) and fill in the methods. This
>> would also then serve as a message spec - it should be created such that
>> it can receive any valid message and sends only (and can send all) valid
>> messages.
> The KernelManager classes and ZMQChannel classes are about as close as
> we can get for now.  There are a couple of different issues:
>
> 1. The handler methods that you are talking about need to be called in
> the main thread.  But all of the channels are receiving msgs in a
> second thread.  The call_handlers method needs to be overridden in a
> way that causes the true handler methods to be called in the other
> thread.  Each toolkit has its own way of calling functions in other
> threads, so that is difficult to do in a general way.  Furthermore, in
> a terminal where there is no event loop, there really isn't a way of
> calling a method in a different thread.  Thus, the yet-to-be-written
> subclasses of KernelManager and the channels will have to simply put
> the received message in a Queue and the main terminal thread will have
> to poll that queue.  But an important question is this:  does curses
> have a way of calling functions in the main thread?  If not, we will
> have to develop a custom KernelManager and channel classes that use
> Queues and polling.  I have started some of this in
> blockingkernelmanager.
>
> 2. We are moving towards a model where "The message is is API"  Thus,
> we don't want to hide the actual messages from frontend code.  You
> really need all of that information and because of (1), we can't
> really easily put it into handler methods.
>
> 3.  We still don't quite know what is needed by different frontends,
> so it is difficult to identify the common code yet.  As time goes on,
> we may realize all frontends use similar logic and we can abstract
> that out properly at the time.  But for now we are in the
> wild-wild-west.
>
> 4.  Everything is truly asynchronous.  It takes a while to get used to
> this fact as it is *very* different than the old terminal based
> IPython.  Because each frontend will handle that asynchronicity in
> different ways, it is very difficult to come up with abstractions
> beyond the messages that are universal.
>
>> Of course, this may not make sense, and I may not know what I'm talking
>> about - I don't know much about the zmq communication, and was sort of
>> hoping to stay focused on the fancy console frontend without delving too
>> deeply into that, but if others agree with me but no one with better
>> knowledge wants to do it, I would be happy to write the necessary code
>> myself, but again, I'm probably not the one best able to do it.
>>
>> Speaking of combined code, it would also be nice to have a
>> frontend.pygmentize module that covers pygments coloring for input,
>> output, prompts, and tracebacks, providing a lexer and a style from
>> config, (formatters would depend on the frontend), and also perhaps some
>> object that takes a formatter and provides all these tools for the
>> frontend, perhaps even descending from KernelManager and just having
>> methods that manage these. That would be nice. I could work on that too,
>> and would be happy too - I just noticed that at least Evan and I have
>> written pygments code, and it would be nice to avoid code duplication.
> I agree, and if you want to take what Evan has done and create a
> common base class
> that all frontends can use and appropriate frontend subclasses for qt,
> curse, terminal
> that would be great.  Not sure it should (at least yet) be descending
> from KernelManager though.
>
>> Anyways, I just feel like we've got 4 people working away on 4 frontends
>> without too much communication going on about useful common code, and it
>> would be nice to get that sort of work delegated out before we all go
>> and write our own versions of the same tools.
> Part of the challenge we are having is that there is so much code
> being written currently that none of us can keep up with it all.  Our
> current model is that my kernelmanager branch is the "common code
> base" that all 4 frontends are using.  There is definitely repeated
> code in the various frontends and over time we will move that into the
> common base.  But you are stepping in very early in the process,
> before all the APIs are very solid.  But part of what we want and need
> is for the various frontend developers to give it a shot and let us
> know what things they need in the common base.  But I think the way
> that needs to go is that each frontend does it on their own first, to
> see what works best for them and then we try to identify the
> commonalities.
>
>> Please let me know if this makes sense and is a good idea - I am
>> certainly not the most knowledgeable here, and if I seem to be missing
>> something, please let me know!
> No, you have definitely noticed the most important point - this stuff
> is not simple and it is definitely not done!  But please let us know
> if you have questions.
>
> Cheers,
>
> Brian
>
>> -Wendell
>>
>> On 08/13/2010 04:27 PM, Brian Granger wrote:
>>> I have this in a browser tab and will review it soon.
>>>
>>> Cheers,
>>>
>>> Brian
>>>
>>> On Thu, Aug 12, 2010 at 1:13 AM, Fernando Perez<fperez.net at gmail.com>    wrote:
>>>> On Wed, Aug 11, 2010 at 2:50 PM, Brian Granger<ellisonbg at gmail.com>    wrote:
>>>>> Very good points.  I think we should just copy this description into the
>>>>> message spec.
>>>> I just updated the doc and pushed to trunk and a build with Min's text:
>>>>
>>>> http://ipython.scipy.org/doc/nightly/html/development/messaging.html#heartbeat-for-kernels
>>>>
>>>> Modulo final feedback, that design spec is reasonably complete, as far
>>>> as I'm concerned.
>>>>
>>>> Thanks!
>>>>
>>>> f
>>>>
>>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>
>
>