[IPython-dev] Coordinating the XREQ and SUB channels

Evan Patterson epatters at enthought.com
Wed Jul 14 17:21:18 EDT 2010

On Wed, Jul 14, 2010 at 2:10 PM, Brian Granger <ellisonbg at gmail.com> wrote:

> On Wed, Jul 14, 2010 at 10:56 AM, Evan Patterson <epatters at enthought.com>
> wrote:
> > Hi guys,
> >
> > I've been making decent progress at connecting my FrontendWidget to a
> > KernelManager. I have, however, encountered one fairly serious problem:
> > since the XREQ and SUB channels of the KernelManager are in separate
> > threads, there is no guarantee about the order in which signals are
> emitted.
> > I'm finding that 'execute_reply' signals are frequently emitted *before*
> all
> > the output signals have been emitted.
> Yes, that is definitely possible and we really don't have control over
> it.  Part of the difficulty is that the SUB/SUB channel does buffering
> of stdout/stderr (just like sys.stdout/sys.stderr).  While it will
> make your application logic more difficult, I think this is something
> fundamental we have to live with.  Also, I wouldn't be surprised if
> the same were true of the regular python shell because of the
> buffering of stdout.

I'm not sure it's fair to call this problem fundamental (if we ignore the
corner case of the threads in the kernel). After all, output and execution
completion happen in a very predictable order in the kernel; it's only our
use of multiple frontend-side channel threads that has complicated the

In a regular same-process shell, this wouldn't be a problem because you
would simply flush stdout before writing the new prompt. It makes sense to
be able to request a flush here, I think. A 'flush' in this case would just
consist of the making the SubChannel thread active, so that its event loop
would pick up whatever it needs to. I believe calling time.sleep(0) once in
the XReqChannel before sending an execute reply will be sufficient. The
latency introduced should be negligible. I'll experiment with this.

> > It seems to me that we should be enforcing, to the extent that we can
> (i.e.
> > ignoring threads in the kernel for now), the assumption that when
> > 'execute_reply' is signaled, all output has been signaled. Is this
> > reasonable?
> I don't think so.  I would write the frontend to allow for arbitrary
> timing of the execute_reply and the SUB messages.  You will have to
> use the parent_id information to order things properly in the GUI.
> Does this make sense.  I think if we try to impose the timing of the
> signals, we will end up breaking the model or introducing extra
> latencies.  Let us know if you have questions.  I know this will
> probably be one of the more subtle parts of the frontend logic.

Yes, this is something that will be quite difficult to get right. For
frontend implementors who are interested only in console-style interaction,
it doesn't make sense for them to have worry about this.


> Cheers,
> Brian
> > Evan
> >
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20100714/ba14ec44/attachment.html>

More information about the IPython-dev mailing list