[IPython-dev] messaging protocol

Jason Grout jason-sage at creativetrax.com
Fri Apr 8 04:04:30 EDT 2011

On 4/8/11 1:12 AM, MinRK wrote:
> On Thu, Apr 7, 2011 at 18:27, Jason Grout <jason-sage at creativetrax.com
> <mailto:jason-sage at creativetrax.com>> wrote:
>     I've been pondering the messaging spec quite a bit over the last few
>     weeks and meshing it in with a project in Sage (a single-cell server,
>     like a one-off ipython or sage web notebook).  Our architecture is that
>     we have a big database sitting on the server side that basically acts as
>     a cache of the zeromq messages between the kernel and the web client. It
>     is easy enough to store the messages in the database, but we have a
>     problem that you guys don't have since you use zeromq end-to-end.  Once
>     we put a list of messages into the database, we don't know what the
>     right order for the messages is.  So here is a
>     suggestion/request/proposal for the API:
>     PROPOSAL: Can we add another field to the header of a, a msg_order (or
>     msg_counter?) field, which (across messages with the same
>     header['session']) is guaranteed to be an increasing integer signifying
>     the order of messages?
> The parallel code uses a slightly more advanced Session object (the thing
> that builds messages) that includes the notion of an extensible 'subheader'.
> With this, you can put any extra (jsonable) information you want into
> the header.

Interesting.  I was going off of the documented spec.  Is this subheader 
information going to make it into the documented spec, or is it an 
extension to the spec?

Also, do you envision the various fields (like user_variables or the 
session field of the header) being required?  For example, our 
simplified execute message is simply:

"content":{"code":"CODE USER TYPES IN"}}

where the omitted fields have the "obvious" defaults (silent=false, all 
other fields empty strings, lists, or dictionaries.

>     You might ask why we don't just use the msg_id field for this.  Well, it
>     seems nice for us to make the msg_id be a mongodb id object, which is
>     just a hash.  It seems elegant to let the client or server dictate what
>     how the msg_id's are formed with no insistence on a particular format or
>     structure for the msg_id, other than it be unique among messages in the
>     same session.
> The Client's Session object (or whoever builds the message) creates the
> msg_id,
> and the only restriction is that it be jsonable.  The Client building
> the message
> can use any scheme they like to build msg_ids, same with the Server and
> its replies,
> and they don't even have to be the same as each other.
>   In fact, I'm not even sure there's anywhere in the code that actually
> requires that msg_id be unique
> (though *we* do, for various reasons).

>     You must have had a related problem storing history, though.  How did
>     you sort the messages in the history database?
> the IPython console has an execution_count (the number that determines
> what goes in 'In [nn]'.  the sqlite database stores the session ID and
> execution counter for every entry.

I was thinking more of multiple output messages for a single input 
(assuming that you stored multiple outputs in the database).  That is 
our current problem, since our use-case will always have an 
execution_count of 1.



More information about the IPython-dev mailing list