[IPython-dev] msg spec

[Jason Grout]

On 4/17/12 8:46 PM, Fernando Perez wrote:
> Hey,
>
> [ Putting this on-list so we have a record of the conversation for the
> archives, I also know that Jason Grout has been using the spec in some
> of the new sage work, so hey may have a stake here]

Thanks.  Up to this point, we've used the spec for a point of reference, 
but we've developed both the sender and receivers, so we've not been 
affected by non-compliant implementations.  However, we hope to start 
using the new ipython in our work this summer (moving aleph.sagemath.org 
to Google App Engine), so we are starting to look hard again at the spec 
and how to use ipython's native implementation.


So please, keep conversations about the messaging spec on the list.

>
> On Tue, Apr 17, 2012 at 15:41, MinRK<benjaminrk at gmail.com>  wrote:
>> I've started to write tests for the message spec, via the KernelManager, and
>> I'm finding that we don't actually comply very well with the spec in our doc
>> (first simple execute test has two failures).
>>
>> execute_reply:
>>
>> 1. payload - this is not a dict, it is a *list* of payload dicts
>
> Yes, the docs need updating here: since payloads can be created by
> user code at arbitrary times during execution, there's no way to put
> them into a single dict.  We probably just hadn't realized that when
> we first wrote the spec, and then forgot to update the docs as we
> started having more than one payload come back.
>
>> 2. transformed_input - this key is never in an execute_reply, it only
>> appears in a specific auto_rewrite *payload*.
>
> Mmh, we may have taken the cheap way out here: conceptually, since
> code was sent out for execution to the kernel, so it makes sense that
> the transformed code (if it was transformed) should come back as a
> top-level entity.  However, implementation-wise that's probably tricky
> to do, b/c inside the IPython object, the rewriting happens deep in
> the bowels of the execution codepath.
>
> Back in the days of terminal-only client that was no big deal: a
> properly positioned 'print' statement was all one needed to see what
> had happened.
>
> But now we'd like that transformed code to propagate *back* all the
> way to the caller, and that would require that this was also available
> in the kernel instead of being dumped to stdout somewhere along the
> way.
>
> So we most likely punted, and replaced the 'print' with a payload
> generator so at least the transformed output would make its way back
> to the calling client, but didn't really honor the spec.
>
> So the question is what to do now:
>
> 1. we can say that any rewrite of the input can happen at arbitrary
> points and that therefore it makes more sense to let it be delivered
> as a payload instead of a top-level return value.
>
> 2. or we stick to the spec, and then we have to fix the code.  The
> hard way would require messing with the entire execution pipeline to
> do it 'right', the easy one would be to put a check right before the
> message is sent, pull any 'rewrite' payload we find out of the
> payloads list and stick it into the 'transformed_code' field of the
> message.  Then the message could be sent according to the current
> spec.
>
> I was initially thinking that 2 was really the way to do it, but now
> I'm not so sure: in principle different parts of the execution
> pipeline could apply different transformations of code (imagine our
> base rewrite plus user-driven prefiltering).

Like the Sage preparser, right?


> So I'm now leaning
> towards 1 as a sensible option (which has the advantage of being
> trivial to implement, as it only requires updating the doc to mention
> 'rewrite' as a possible payload).
>
>> I've been using traitlets to do the message validation, and I'm running into
>> the fact that we are unclear about what keys can be None and what cannot.
>>   Should I start with the assumption that *anything* can be None, or not?
>
> I think that's a reasonable starting point. Once the tests are in
> solid shape we can revisit this and go over the spec/implementation
> more carefully.
>
> Thanks for digging into this, it's one of the big missing pieces we have!

Thanks also from us in the Sage aleph project for tightening loose ends 
up here!

Jason