[Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

Josiah Carlson josiah.carlson at gmail.com
Thu Mar 27 22:52:59 CET 2014


Hopping in to give my take on this, which I've expressed to Antoine
off-list.

When I first built the functionality about 8.5-9 years ago, I personally
just wanted to be able to build something that could replace some of
Expect: http://expect.sourceforge.net/ . The original and perhaps current
API of the GSoC patch were inspired by my experience with asyncore (hence
send() and recv() methods), but I never made an effort to get it
practically working with asyncore - primarily because such would be
functionally impossible on Windows without a lot of work to pull in a chunk
of what was pywin32 libraries (at the time, Windows was a primary target).
On the *nix side of things, performing the integration would be arguably
trivial as select, poll, epoll, etc., all deal with pipes the same way as
any other file handles (on-disk files, domain sockets, network sockets,
etc.), with OS X being the exception. A little work would have been
necessary to handle the two readable file handles and one writable file
handle, but it's not that much different than building a proxy. But I
digress.

At this point I still believe that the functionality is useful from a
scriptable interaction perspective, regardless of platform. I don't believe
that being able to natively support the piping of output from one process
to another is necessary, but a convenient future feature. That said,
discussions about the quality of the existing GSoC patch and its API
basically mean that the existing code to implement async subprocesses
within the subprocess module precludes it from an easy or short acceptance
process. And without substantial efforts from one or more people would doom
the feature request and PEP to rejection.

As an alternative, easily 95% of what most people would use this for can be
written as an example using the asyncio module and included in the docs
just after (or replacing)
http://docs.python.org/3/library/asyncio-subprocess.html#example . Adding a
reference to the subprocess module docs to point off to the asyncio
subprocess example docs would get people a copy/paste snippet that they can
include and update to their heart's content.

Benefits to updating the docs:
* It can happen at any time and doesn't need to wait for a 3.5 release (it
can also happily wait)
* No one likes maintaining code, but everyone loves docs (especially if it
documents likely use-cases)
* Because it is example docs, maybe a multi-week bikeshedding discussion
about API doesn't need to happen (as long as "read line", "read X bytes",
"read what is available", and "write this data" - all with timeouts - are
shown, people can build everything else they want/need)
* An example using asyncio is shorter than the modifications to the
subprocess module
* I would celebrate the closing of a feature request I opened in 2005

Aside from discarding code (Eric's and my own), not supporting Python-side
chained pipes, and potentially angering some purists who *needed* this to
be based on the subprocess module, I'm not sure I can think of any
drawbacks. And arguably 2/3 of those drawbacks are imagined.


Let me know your thoughts. If it gets an "okay", I'll come up with some
example code, update the docs, and post a link to the code review in this
thread.

 - Josiah



On Wed, Mar 26, 2014 at 4:55 AM, Victor Stinner <victor.stinner at gmail.com>wrote:

> Hi,
>
> For your information, asyncio.subprocess.Process is limited. It's not
> possible yet to connect pipes between two processes. Something like
> "cat | wc -l" where the cat stdin comes from Python.
>
> It's possible to enhance the API to implement that, but the timeframe
> was too short to implement it before Python 3.4.
>
> Victor
>
> 2014-03-25 23:19 GMT+01:00 Antoine Pitrou <solipsis at pitrou.net>:
> >
> > Hi,
> >
> > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for
> > subprocess.popen.  I answered that asyncio now has subprocess support
> > (including non-blocking I/O on the three standard stream pipes), so
> > it's not obvious anything else is needed.
> >
> > Should we change the PEP's status to Rejected or Superseded?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140327/e60424ae/attachment.html>


More information about the Python-Dev mailing list