[IPython-dev] Coordinating the XREQ and SUB channels

[Evan Patterson]

On Wed, Jul 14, 2010 at 2:10 PM, Brian Granger <ellisonbg at gmail.com> wrote:

> On Wed, Jul 14, 2010 at 10:56 AM, Evan Patterson <epatters at enthought.com>
> wrote:
> > Hi guys,
> >
> > I've been making decent progress at connecting my FrontendWidget to a
> > KernelManager. I have, however, encountered one fairly serious problem:
> > since the XREQ and SUB channels of the KernelManager are in separate
> > threads, there is no guarantee about the order in which signals are
> emitted.
> > I'm finding that 'execute_reply' signals are frequently emitted *before*
> all
> > the output signals have been emitted.
>
> Yes, that is definitely possible and we really don't have control over
> it.  Part of the difficulty is that the SUB/SUB channel does buffering
> of stdout/stderr (just like sys.stdout/sys.stderr).  While it will
> make your application logic more difficult, I think this is something
> fundamental we have to live with.  Also, I wouldn't be surprised if
> the same were true of the regular python shell because of the
> buffering of stdout.
>

I'm not sure it's fair to call this problem fundamental (if we ignore the
corner case of the threads in the kernel). After all, output and execution
completion happen in a very predictable order in the kernel; it's only our
use of multiple frontend-side channel threads that has complicated the
issue.

In a regular same-process shell, this wouldn't be a problem because you
would simply flush stdout before writing the new prompt. It makes sense to
be able to request a flush here, I think. A 'flush' in this case would just
consist of the making the SubChannel thread active, so that its event loop
would pick up whatever it needs to. I believe calling time.sleep(0) once in
the XReqChannel before sending an execute reply will be sufficient. The
latency introduced should be negligible. I'll experiment with this.


> > It seems to me that we should be enforcing, to the extent that we can
> (i.e.
> > ignoring threads in the kernel for now), the assumption that when
> > 'execute_reply' is signaled, all output has been signaled. Is this
> > reasonable?
>
> I don't think so.  I would write the frontend to allow for arbitrary
> timing of the execute_reply and the SUB messages.  You will have to
> use the parent_id information to order things properly in the GUI.
> Does this make sense.  I think if we try to impose the timing of the
> signals, we will end up breaking the model or introducing extra
> latencies.  Let us know if you have questions.  I know this will
> probably be one of the more subtle parts of the frontend logic.
>

Yes, this is something that will be quite difficult to get right. For
frontend implementors who are interested only in console-style interaction,
it doesn't make sense for them to have worry about this.

Evan


>
> Cheers,
>
> Brian
>
>
>
>
> > Evan
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100714/ba14ec44/attachment.html>