[IPython-dev] msg spec

Fernando Perez Fernando.Perez at berkeley.edu
Tue Apr 17 21:46:10 EDT 2012


[ Putting this on-list so we have a record of the conversation for the
archives, I also know that Jason Grout has been using the spec in some
of the new sage work, so hey may have a stake here]

On Tue, Apr 17, 2012 at 15:41, MinRK <benjaminrk at gmail.com> wrote:
> I've started to write tests for the message spec, via the KernelManager, and
> I'm finding that we don't actually comply very well with the spec in our doc
> (first simple execute test has two failures).
> execute_reply:
> 1. payload - this is not a dict, it is a *list* of payload dicts

Yes, the docs need updating here: since payloads can be created by
user code at arbitrary times during execution, there's no way to put
them into a single dict.  We probably just hadn't realized that when
we first wrote the spec, and then forgot to update the docs as we
started having more than one payload come back.

> 2. transformed_input - this key is never in an execute_reply, it only
> appears in a specific auto_rewrite *payload*.

Mmh, we may have taken the cheap way out here: conceptually, since
code was sent out for execution to the kernel, so it makes sense that
the transformed code (if it was transformed) should come back as a
top-level entity.  However, implementation-wise that's probably tricky
to do, b/c inside the IPython object, the rewriting happens deep in
the bowels of the execution codepath.

Back in the days of terminal-only client that was no big deal: a
properly positioned 'print' statement was all one needed to see what
had happened.

But now we'd like that transformed code to propagate *back* all the
way to the caller, and that would require that this was also available
in the kernel instead of being dumped to stdout somewhere along the

So we most likely punted, and replaced the 'print' with a payload
generator so at least the transformed output would make its way back
to the calling client, but didn't really honor the spec.

So the question is what to do now:

1. we can say that any rewrite of the input can happen at arbitrary
points and that therefore it makes more sense to let it be delivered
as a payload instead of a top-level return value.

2. or we stick to the spec, and then we have to fix the code.  The
hard way would require messing with the entire execution pipeline to
do it 'right', the easy one would be to put a check right before the
message is sent, pull any 'rewrite' payload we find out of the
payloads list and stick it into the 'transformed_code' field of the
message.  Then the message could be sent according to the current

I was initially thinking that 2 was really the way to do it, but now
I'm not so sure: in principle different parts of the execution
pipeline could apply different transformations of code (imagine our
base rewrite plus user-driven prefiltering).  So I'm now leaning
towards 1 as a sensible option (which has the advantage of being
trivial to implement, as it only requires updating the doc to mention
'rewrite' as a possible payload).

> I've been using traitlets to do the message validation, and I'm running into
> the fact that we are unclear about what keys can be None and what cannot.
>  Should I start with the assumption that *anything* can be None, or not?

I think that's a reasonable starting point. Once the tests are in
solid shape we can revisit this and go over the spec/implementation
more carefully.

Thanks for digging into this, it's one of the big missing pieces we have!



More information about the IPython-dev mailing list