[IPython-dev] Message spec draft more fleshed out

[Brian Granger]

Wendell,

On Fri, Aug 13, 2010 at 3:04 PM, Wendell Smith <wackywendell at gmail.com> wrote:
>  Hello all,
>
> I have been looking all these documents over, and wondering if perhaps
> we could have some object (descended from KernelManager) that would be
> constructed to perfectly match the message spec, such that any message
> received would translate to a functional call (i.e.
> MessageManager.execute(self, header, code, silent=False)), to make it
> easy for someone to simply write an object that descends from
> MessageManager (or whatever we call it) and fill in the methods. This
> would also then serve as a message spec - it should be created such that
> it can receive any valid message and sends only (and can send all) valid
> messages.

The KernelManager classes and ZMQChannel classes are about as close as
we can get for now.  There are a couple of different issues:

1. The handler methods that you are talking about need to be called in
the main thread.  But all of the channels are receiving msgs in a
second thread.  The call_handlers method needs to be overridden in a
way that causes the true handler methods to be called in the other
thread.  Each toolkit has its own way of calling functions in other
threads, so that is difficult to do in a general way.  Furthermore, in
a terminal where there is no event loop, there really isn't a way of
calling a method in a different thread.  Thus, the yet-to-be-written
subclasses of KernelManager and the channels will have to simply put
the received message in a Queue and the main terminal thread will have
to poll that queue.  But an important question is this:  does curses
have a way of calling functions in the main thread?  If not, we will
have to develop a custom KernelManager and channel classes that use
Queues and polling.  I have started some of this in
blockingkernelmanager.

2. We are moving towards a model where "The message is is API"  Thus,
we don't want to hide the actual messages from frontend code.  You
really need all of that information and because of (1), we can't
really easily put it into handler methods.

3.  We still don't quite know what is needed by different frontends,
so it is difficult to identify the common code yet.  As time goes on,
we may realize all frontends use similar logic and we can abstract
that out properly at the time.  But for now we are in the
wild-wild-west.

4.  Everything is truly asynchronous.  It takes a while to get used to
this fact as it is *very* different than the old terminal based
IPython.  Because each frontend will handle that asynchronicity in
different ways, it is very difficult to come up with abstractions
beyond the messages that are universal.

> Of course, this may not make sense, and I may not know what I'm talking
> about - I don't know much about the zmq communication, and was sort of
> hoping to stay focused on the fancy console frontend without delving too
> deeply into that, but if others agree with me but no one with better
> knowledge wants to do it, I would be happy to write the necessary code
> myself, but again, I'm probably not the one best able to do it.
>
> Speaking of combined code, it would also be nice to have a
> frontend.pygmentize module that covers pygments coloring for input,
> output, prompts, and tracebacks, providing a lexer and a style from
> config, (formatters would depend on the frontend), and also perhaps some
> object that takes a formatter and provides all these tools for the
> frontend, perhaps even descending from KernelManager and just having
> methods that manage these. That would be nice. I could work on that too,
> and would be happy too - I just noticed that at least Evan and I have
> written pygments code, and it would be nice to avoid code duplication.

I agree, and if you want to take what Evan has done and create a
common base class
that all frontends can use and appropriate frontend subclasses for qt,
curse, terminal
that would be great.  Not sure it should (at least yet) be descending
from KernelManager though.

> Anyways, I just feel like we've got 4 people working away on 4 frontends
> without too much communication going on about useful common code, and it
> would be nice to get that sort of work delegated out before we all go
> and write our own versions of the same tools.

Part of the challenge we are having is that there is so much code
being written currently that none of us can keep up with it all.  Our
current model is that my kernelmanager branch is the "common code
base" that all 4 frontends are using.  There is definitely repeated
code in the various frontends and over time we will move that into the
common base.  But you are stepping in very early in the process,
before all the APIs are very solid.  But part of what we want and need
is for the various frontend developers to give it a shot and let us
know what things they need in the common base.  But I think the way
that needs to go is that each frontend does it on their own first, to
see what works best for them and then we try to identify the
commonalities.

> Please let me know if this makes sense and is a good idea - I am
> certainly not the most knowledgeable here, and if I seem to be missing
> something, please let me know!

No, you have definitely noticed the most important point - this stuff
is not simple and it is definitely not done!  But please let us know
if you have questions.

Cheers,

Brian

> -Wendell
>
> On 08/13/2010 04:27 PM, Brian Granger wrote:
>> I have this in a browser tab and will review it soon.
>>
>> Cheers,
>>
>> Brian
>>
>> On Thu, Aug 12, 2010 at 1:13 AM, Fernando Perez<fperez.net at gmail.com>  wrote:
>>> On Wed, Aug 11, 2010 at 2:50 PM, Brian Granger<ellisonbg at gmail.com>  wrote:
>>>> Very good points.  I think we should just copy this description into the
>>>> message spec.
>>> I just updated the doc and pushed to trunk and a build with Min's text:
>>>
>>> http://ipython.scipy.org/doc/nightly/html/development/messaging.html#heartbeat-for-kernels
>>>
>>> Modulo final feedback, that design spec is reasonably complete, as far
>>> as I'm concerned.
>>>
>>> Thanks!
>>>
>>> f
>>>
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com