[Python-Dev] PEP 3145 (With Contents)
exarkun at twistedmatrix.com
exarkun at twistedmatrix.com
Tue Sep 15 20:24:56 CEST 2009
On 04:25 pm, eric.pruitt at gmail.com wrote:
>I'm bumping this PEP again in hopes of getting some feedback.
>On Tue, Sep 8, 2009 at 23:52, Eric Pruitt <eric.pruitt at gmail.com>
>>Title: Asynchronous I/O For subprocess.Popen
>>Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
>>Type: Standards Track
>> In its present form, the subprocess.Popen implementation is prone
>> dead-locking and blocking of the parent Python script while waiting
>> from the child process.
>> A search for "python asynchronous subprocess" will turn up numerous
>> accounts of people wanting to execute a child process and
>> it from time to time reading only the data that is available
>> blocking to wait for the program to produce data   . The
>> behavior of the subprocess module is that when a user sends or
>> data via the stdin, stderr and stdout file objects, dead locks are
>> and documented  . While communicate can be used to alleviate
>> the buffering issues, it will still cause the parent process to
>> attempting to read data when none is available to be read from the
>> There is a documented need for asynchronous, non-blocking
>> subprocess.Popen    . Inclusion of the code would
>> utility of the Python standard library that can be used on Unix
>> Windows builds of Python. Practically every I/O object in Python
>> file-like wrapper of some sort. Sockets already act as such and
>> strings there is StringIO. Popen can be made to act like a file by
>> using the methods attached the the subprocess.Popen.stderr, stdout
>> stdin file-like objects. But when using the read and write methods
>> those options, you do not have the benefit of asynchronous I/O. In
>> proposed solution the wrapper wraps the asynchronous methods to
>> file object.
>> I have been maintaining a Google Code repository that contains all
>> changes including tests and documentation  as well as blog
>> the problems I have come across in the development process .
>> I have been working on implementing non-blocking asynchronous I/O
>> subprocess.Popen module as well as a wrapper class for
>> that makes it so that an executed process can take the place of a
>> duplicating all of the methods and attributes that file objects
"Non-blocking" and "asynchronous" are actually two different things.
From the rest of this PEP, I think only a non-blocking API is being
introduced. I haven't looked beyond the PEP, though, so I might be
>> There are two base functions that have been added to the
>> class: Popen.send and Popen._recv, each with two separate
>> one for Windows and one for Unix based systems. The Windows
>> implementation uses ctypes to access the functions needed to
>> in the kernel 32 DLL in an asynchronous manner. On Unix based
>> the Python interface for file control serves the same purpose. The
>> different implementations of Popen.send and Popen._recv have
>> arguments to make code that uses these functions work across
Why does the method for non-blocking read from a pipe start with an "_"?
This is the convention (widely used) for a private API. The name also
doesn't suggest that this is the non-blocking version of reading.
Similarly, the name "send" doesn't suggest that this is the non-blocking
version of writing.
>> When calling the Popen._recv function, it requires the pipe name be
>> passed as an argument so there exists the Popen.recv function that
>> selects stdout as the pipe for Popen._recv by default.
>> selects stderr as the pipe by default. "Popen.recv" and
>> are much easier to read and understand than "Popen._recv('stdout'
>> "Popen._recv('stderr' ..." respectively.
What about reading from other file descriptors? subprocess.Popen allows
arbitrary file descriptors to be used. Is there any provision here for
reading and writing non-blocking from or to those?
>> Since the Popen._recv function does not wait on data to be produced
>> before returning a value, it may return empty bytes.
>> handles this issue by returning all data read over a given time
Oh. Popen.asyncread? What's that? This is the first time the PEP
>> The ProcessIOWrapper class uses the asyncread and asyncwrite
>> allow a process to act like a file so that there are no blocking
>> that can arise from using the stdout and stdin file objects
>> a subprocess.Popen call.
What's the ProcessIOWrapper class? And what's the asyncwrite function?
Again, this is the first time it's mentioned.
So, to sum up, I think my main comment is that the PEP seems to be
missing a significant portion of the details of what it's actually
proposing. I suspect that this information is present in the
implementation, which I have not looked at, but it probably belongs in
More information about the Python-Dev