[IPython-dev] Coordinating the XREQ and SUB channels

[Evan Patterson]

I've added a 'flush' method to the KernelManager here:

http://github.com/epatters/ipython/commit/2ecde29e8f2a5e7012236f61819b2f7833248553

It works, although there may be a more intelligent way to do it. That being
said, I tried a number of different things, and none of the others worked.

Brian: since the 'flush' method must be called explicitly by clients, this
won't break our model or extra induce latencies for clients that want to
take a more sophisticated approach to SUB channel monitoring.

Evan

On Wed, Jul 14, 2010 at 4:21 PM, Evan Patterson <epatters at enthought.com>wrote:

> On Wed, Jul 14, 2010 at 2:10 PM, Brian Granger <ellisonbg at gmail.com>wrote:
>
>> On Wed, Jul 14, 2010 at 10:56 AM, Evan Patterson <epatters at enthought.com>
>> wrote:
>> > Hi guys,
>> >
>> > I've been making decent progress at connecting my FrontendWidget to a
>> > KernelManager. I have, however, encountered one fairly serious problem:
>> > since the XREQ and SUB channels of the KernelManager are in separate
>> > threads, there is no guarantee about the order in which signals are
>> emitted.
>> > I'm finding that 'execute_reply' signals are frequently emitted *before*
>> all
>> > the output signals have been emitted.
>>
>> Yes, that is definitely possible and we really don't have control over
>> it.  Part of the difficulty is that the SUB/SUB channel does buffering
>> of stdout/stderr (just like sys.stdout/sys.stderr).  While it will
>> make your application logic more difficult, I think this is something
>> fundamental we have to live with.  Also, I wouldn't be surprised if
>> the same were true of the regular python shell because of the
>> buffering of stdout.
>>
>
> I'm not sure it's fair to call this problem fundamental (if we ignore the
> corner case of the threads in the kernel). After all, output and execution
> completion happen in a very predictable order in the kernel; it's only our
> use of multiple frontend-side channel threads that has complicated the
> issue.
>
> In a regular same-process shell, this wouldn't be a problem because you
> would simply flush stdout before writing the new prompt. It makes sense to
> be able to request a flush here, I think. A 'flush' in this case would just
> consist of the making the SubChannel thread active, so that its event loop
> would pick up whatever it needs to. I believe calling time.sleep(0) once in
> the XReqChannel before sending an execute reply will be sufficient. The
> latency introduced should be negligible. I'll experiment with this.
>
>
>> > It seems to me that we should be enforcing, to the extent that we can
>> (i.e.
>> > ignoring threads in the kernel for now), the assumption that when
>> > 'execute_reply' is signaled, all output has been signaled. Is this
>> > reasonable?
>>
>> I don't think so.  I would write the frontend to allow for arbitrary
>> timing of the execute_reply and the SUB messages.  You will have to
>> use the parent_id information to order things properly in the GUI.
>> Does this make sense.  I think if we try to impose the timing of the
>> signals, we will end up breaking the model or introducing extra
>> latencies.  Let us know if you have questions.  I know this will
>> probably be one of the more subtle parts of the frontend logic.
>>
>
> Yes, this is something that will be quite difficult to get right. For
> frontend implementors who are interested only in console-style interaction,
> it doesn't make sense for them to have worry about this.
>
> Evan
>
>
>>
>> Cheers,
>>
>> Brian
>>
>>
>>
>>
>> > Evan
>> >
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100715/a3ce46d8/attachment.html>