[ python-Feature Requests-1191964 ] asynchronous Subprocess

[SourceForge.net]

Feature Requests item #1191964, was opened at 2005-04-28 13:40
Message generated for change (Comment added) made by josiahcarlson
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1191964&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Josiah Carlson (josiahcarlson)
Assigned to: Peter Åstrand (astrand)
Summary: asynchronous Subprocess

Initial Comment:
It would be terribly nice if the Popen class in the
subprocess module would allow a programmer to easily
say "send some data right now (if I have some to send)
and receive whatever information is available right
now".  Essentially the equivalent of
asyncore.loop(count=1), only that it returns the data
received, instead of placing it in a buffer.

Why would this functionality be useful?  Currently,
when using the subprocess module with pipes, the
interaction with a pipe is limited to "send data if
desired, close the subprocess' stdin, read all output
from the subprocess' stdout, ...". 

Certainly one can pull the pipes out of the Popen
instance, and perform the necessary functions on posix
systems (with select or poll), but the additional magic
on WIndows is a little less obvious (but still possible).

There is a post by Paul Du Bois with an example using
win32pipe.PeekNamedPipe:
http://groups-beta.google.com/group/comp.lang.python/msg/115e9332cc1ca09d?hl=en

And a message from Neil Hodgeson stating that
PeekNamedPipe works on anonymous pipes:
http://mail.python.org/pipermail/python-dev/2000-May/004200.html


With this modification, creating Expect-like modules
for any platform, as well as embedded shells inside any
GUI with a text input widget, becomes easy.  Heck, even
Idle could use it rather than a socket for its
interactive interpreter.

----------------------------------------------------------------------

>Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-09-21 13:55

Message:
Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-09-21 13:51

Message:
Logged In: YES 
user_id=341410

I've implemented this as a subclass of subprocess.Popen in
the Python cookbook, available here:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Essentially this is a request for inclusion in the standard
library for Python 2.5 .

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-06-26 12:47

Message:
Logged In: YES 
user_id=341410

How about if subprocesses have 3 new methods, send(input),
recv(maxlen), and recv_stderr(maxlen).

send(input) would perform like socket.send(), sending as
much as it currently can, returning the number of bytes sent.
recv(maxlen) and recv_stderr(maxlen) would recieve up to the
provided number of bytes from the stdout or stderr pipes
respectively.

I currently have an implementation of the above on Windows
and posix.  I include the context diff against revision 1.20
of subprocess.py in current CVS.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-05-28 17:15

Message:
Logged In: YES 
user_id=341410

I suppose I should mention one side-effect of all this.  It
requires three more functions from pywin32 be included in
the _subprocess driver; win32file.ReadFile,
win32file.WriteFile, and win32pipe.PeekNamedPipe .  Read and
Peek are for reading data from stdout and stderr, and Write
is for the support for partial writes to stdin.

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2005-05-28 16:22

Message:
Logged In: YES 
user_id=341410

I've got a version of subprocess that has this functionality
with pywin32.  Making it work on *nix systems with usable
select is trivial.

About the only question is whether the functionality is
desireable, and if so, what kind of API is reasonable.

Perhaps adding an optional argument 'wait_for_completion' to
the communicate method, which defaults to True for executing
what is currently in the body of communicate.

If the 'wait_for_completion' argument is untrue, then it
does non-waiting asynchronous IO, returning either a 2 or
3-tuple on completion.  If input is None, it is a 2-tuple of
(stdout, stderr).  If input is a string, it is a 3-tuple of
(bytes_sent, stdout, stderr).

How does that sound?

----------------------------------------------------------------------

Comment By: Dave Schuyler (parameter)
Date: 2005-04-28 16:38

Message:
Logged In: YES 
user_id=473331

More control of sub-processes would be great.  Several times
in the past few years I've done stuff with os.system,
popenN, or spawn* and it would be useful to have even more
cross-platform consistency and support in this area. 
Anyway, this is just a vote to encurage other developers.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1191964&group_id=5470