[Python-Dev] subprocess crossplatformness and async communication

anatoly techtonik techtonik at gmail.com
Mon Jan 26 09:44:01 CET 2009


Hi, Guido

You can't launch a process and communicate with it without blocking at
some point. The scenario where you can periodically check output and
error pipes without blocking and send input is not supported.

Scenario One: control console program. This implies reading from input
and writing replies asyncronously
- Popen.stdin, stdout, stderr - stdin.write(), stdout.read() or
stderr.read() block script execution if any of the other OS pipe
buffers fills up and blocks the child process. Advice is to use
communicate(), but it waits for process to exit and hence doesn't
allow to send the input based on reply from previous send.

Scenario Two: waiting for process that terminates with extensive output blocks
- Popen.wait() blocks if the child process generates enough output to
a stdout or stderr pipe, advice is to use Popen.communicate()
http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
- Popen.communicate() - Waits for process to terminate, reads data
from stdout and stderr until end-of-file. Cashes data in memory -
should not be used for if the data size is large or unlimited.

Solution - read(maxsize) and write(maxsize) functions that will return
immediately.

-- 
anatoly t.

On Sat, Jan 24, 2009 at 5:58 PM, Guido van Rossum <guido at python.org> wrote:
> Anatoly,
>
> I'm confused. The subprocess already allows reading/writing its
> stdin/stdout/stderr, and AFAIK it's a platform-neutral API. I'm sure
> there's something missing, but your post doesn't make it clear what
> exactly, and the recipe you reference is too large to digest easily.
> Can you explain what it is that the current subprocess does't have
> beyond saying "async communication" (which could mean many things to
> many people)?
>
> --Guido
>
> On Sat, Jan 24, 2009 at 5:07 AM, anatoly techtonik <techtonik at gmail.com> wrote:
>> Greetings,
>>
>> This turned out to be a rather long post that in short can be summarized as:
>> "please-please-please, include asynchronous process communication in
>> subprocess module and do not allow "available only on ..."
>> functionality", because it hurts the brain".
>>
>> Code to speak for itself: http://code.activestate.com/recipes/440554/
>>
>>
>> The subprocess module was a great step forward to unify various spawn
>> and system and exec and etc. calls in one module, and more importantly
>> - in one uniform API. But this API is partly crossplatform, and I
>> believe I've seen recent commits to docs with more unix-only
>> differences in this module.
>>
>> The main point of this module is to "allows you to spawn new
>> processes, connect to their input/output/error pipes, and obtain their
>> return codes". PEP 324 goal is also to make "make Python an even
>> better replacement language for over-complicated shell scripts".
>>
>> Citing pre-subrocess PEP 324, "Currently, Python has a large number of
>> different functions for process creation. This makes it hard for
>> developers to choose." Now there is one class with many methods and
>> many platform-specific comments and notices. To make thing worse
>> people on Unix use subprocess with fcntl and people on windows tend
>> not to use it at all, because it looks complicated and doesn't solve
>> the problem with asynchronous communication.
>>
>> That I suggest is to add either support for async crossplatfrom
>> read/write/probing of executed process or a comment at the top of
>> module documentation that will warn that subprocess works in blocking
>> mode. With async mode you can emulate blocking, the opposite is not
>> possible. This will save python users a lot of time.
>>
>> Thanks for reading my rant.
>>
>>
>> BTW, the proposed change is top 10 python recipe on ActiveState
>> http://code.activestate.com/recipes/langs/python/
>>
>> --
>> --anatoly t.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
>



-- 
--anatoly t.


More information about the Python-Dev mailing list