[Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

I'm bumping this PEP again in hopes of getting some feedback.

Thanks,
Eric

On Tue, Sep 8, 2009 at 23:52, Eric Pruitt <eric.pruitt at gmail.com> wrote:
> PEP: 3145
> Title: Asynchronous I/O For subprocess.Popen
> Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
> Type: Standards Track
> Content-Type: text/plain
> Created: 04-Aug-2009
> Python-Version: 3.2
>
> Abstract:
>
>    In its present form, the subprocess.Popen implementation is prone to
>    dead-locking and blocking of the parent Python script while waiting on data
>    from the child process.
>
> Motivation:
>
>    A search for "python asynchronous subprocess" will turn up numerous
>    accounts of people wanting to execute a child process and communicate with
>    it from time to time reading only the data that is available instead of
>    blocking to wait for the program to produce data [1] [2] [3].  The current
>    behavior of the subprocess module is that when a user sends or receives
>    data via the stdin, stderr and stdout file objects, dead locks are common
>    and documented [4] [5].  While communicate can be used to alleviate some of
>    the buffering issues, it will still cause the parent process to block while
>    attempting to read data when none is available to be read from the child
>    process.
>
> Rationale:
>
>    There is a documented need for asynchronous, non-blocking functionality in
>    subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the
>    utility of the Python standard library that can be used on Unix based and
>    Windows builds of Python.  Practically every I/O object in Python has a
>    file-like wrapper of some sort.  Sockets already act as such and for
>    strings there is StringIO.  Popen can be made to act like a file by simply
>    using the methods attached the the subprocess.Popen.stderr, stdout and
>    stdin file-like objects.  But when using the read and write methods of
>    those options, you do not have the benefit of asynchronous I/O.  In the
>    proposed solution the wrapper wraps the asynchronous methods to mimic a
>    file object.
>
> Reference Implementation:
>
>    I have been maintaining a Google Code repository that contains all of my
>    changes including tests and documentation [9] as well as blog detailing
>    the problems I have come across in the development process [10].
>
>    I have been working on implementing non-blocking asynchronous I/O in the
>    subprocess.Popen module as well as a wrapper class for subprocess.Popen
>    that makes it so that an executed process can take the place of a file by
>    duplicating all of the methods and attributes that file objects have.
>
>    There are two base functions that have been added to the subprocess.Popen
>    class: Popen.send and Popen._recv, each with two separate implementations,
>    one for Windows and one for Unix based systems.  The Windows
>    implementation uses ctypes to access the functions needed to control pipes
>    in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
>    the Python interface for file control serves the same purpose.  The
>    different implementations of Popen.send and Popen._recv have identical
>    arguments to make code that uses these functions work across multiple
>    platforms.
>
>    When calling the Popen._recv function, it requires the pipe name be
>    passed as an argument so there exists the Popen.recv function that passes
>    selects stdout as the pipe for Popen._recv by default.  Popen.recv_err
>    selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err"
>    are much easier to read and understand than "Popen._recv('stdout' ..." and
>    "Popen._recv('stderr' ..." respectively.
>
>    Since the Popen._recv function does not wait on data to be produced
>    before returning a value, it may return empty bytes.  Popen.asyncread
>    handles this issue by returning all data read over a given time
>    interval.
>
>    The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
>    allow a process to act like a file so that there are no blocking issues
>    that can arise from using the stdout and stdin file objects produced from
>    a subprocess.Popen call.
>
>
> References:
>
>    [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
>        http://mail.python.org/pipermail/python-bugs-list/2006-December/
>          036524.html
>
>    [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
>        http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
>
>    [3] How can I run an external command asynchronously from Python? - Stack
>        Overflow
>        http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
>          command-asynchronously-from-python
>
>    [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
>        http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
>
>    [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
>        http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
>
>    [6] Issue 1191964: asynchronous Subprocess - Python tracker
>        http://bugs.python.org/issue1191964
>
>    [7] Module to allow Asynchronous subprocess use on Windows and Posix
>        platforms - ActiveState Code
>        http://code.activestate.com/recipes/440554/
>
>    [8] subprocess.rst - subprocdev - Project Hosting on Google Code
>        http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
>
>    [9] subprocdev - Project Hosting on Google Code
>        http://code.google.com/p/subprocdev
>
>    [10] Python Subprocess Dev
>         http://subdev.blogspot.com/
>
> Copyright:
>
>    This P.E.P. is licensed under the Open Publication License;
>    http://www.opencontent.org/openpub/.