[Python-ideas] Tulip / PEP 3156 - subprocess events

[Guido van Rossum]

On Wed, Jan 16, 2013 at 10:10 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 16 January 2013 17:52, Guido van Rossum <guido at python.org> wrote:
> > On Wed, Jan 16, 2013 at 7:12 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> >> I've so far been lurking on the tulip/async discussions, as although
> >> I'm interested, I have no specific need for writing high-performance
> >> network code.
> >>
> >> However, I hit a use case today which seems to me to be ideal for an
> >> async-style approach, and yet I don't think it's covered by the
> >> current PEP. Specifically, I am looking at monitoring a
> >> subprocess.Popen object. This is basically an IO loop, but monitoring
> >> the 3 pipes to the subprocess (well, only stdout and stderr in my
> >> case...). Something like add_reader/add_writer would be fine, except
> >> for the fact that (a) they are documented as low-level not for the
> >> user, and (b) they don't work in all cases (e.g. in a select-based
> >> loop on Windows).
> >>
> >> I'd like PEP 3156 to include some support for waiting on IO from (one
> >> or more) subprocesses like this in a cross-platform way. If there's
> >> something in there to do this at the moment, that's great, but it
> >> wasn't obvious to me when I looked...
> >
> > This is a great use case. The right approach would probably be to
> > define a new Transport (and an event loop method to create one) that
> > wraps pipes going into and out of a subprocess. The new method would
> > have a standard API (probably similar to that of subprocess), whereas
> > there would be different implementations of the Transport based on
> > platform and event loop implementation (similar to the way the
> > subprocess module has quite different implementations).
> >
> > Can you check out the Tulip source code (code.google.com/p/tulip) and
> > come up with a patch to do this? I'll gladly review it. It's fine to
> > only cover the UNIX case for now.
>
> I'll have a look. There *is* one problem, though - I imagine it will
> be relatively easy to put something together that works on Unix, as
> waiting on pipes is covered by the existing select/poll mechanisms.
> But I'm on Windows, so I won't be able to test it. And on Windows,
> there's no mechanism in place to wait on arbitrary filehandles, so the
> process wait mechanism is a much harder nut to crack. Chicken and egg
> problem...
>

What does the subprocess module do on Windows? (I'm in the reverse
position, although I have asked the kind IT folks at Dropbox to provide me
with a Windows machine.)


> Maybe I'll start by looking at waiting on arbitrary filehandles, and
> use that to build the process approach. Unfortunately, I don't think
> IOCP is any more able to wait on arbitrary files than select - see my
> followup to an older thread on Richard's work there. Or maybe I'll set
> up a hacking environment in a Linux VM or something. That'd be a fun
> experience in any case.
>

I'm eagerly awaiting Richard's response. AFAIK handles on Windows *are*
more general than sockets...


> I'll have to get my brain round the existing spec as well. I'm finding
> it hard to understand why there are so many methods on the event loop
> that are specific to particular use cases (for this example, your
> suggested method to create the new type of Transport).


This is mainly so that the event loop implementation can control the
Transport class. Note that it isn't enough to define different Transport
classes per platform -- on a single platform there may be multiple event
loop implementations (e.g. on Windows you can use Select or IOCP) and these
may need different Transport implementations. SO this must really be under
control of the event loop object.


> My instinct
> says that this should *also* be a good test case for a user coming up
> with a new type of "event source" and wanting to plug it into the
> event loop. Having to add a new method to the event loop seems to
> imply this isn't possible.
>

If the user is okay with solving the problem only for their particular
platform and event loop implementation they don't need to add anything to
the event loop. But for transports that make it into the PEP, it is
essential that alternate implementations (e.g. one that proxies a Twisted
Reactor) be in control of the Transport construction.

>
> OK, off to do a lot of spec reading and then some coding. With luck,
> you'll be patient with dumb questions from me on the way :-)
>

I will be!

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130116/99260a10/attachment.html>