There is a related bit of functionality for subprocess that would allow for asynchronous handling of IO to/from the called subprocess. It has been implemented as a recipe [1], but requires the use of additional pywin32 functionality on Windows. As was the case for the original subprocess module, in order to get the proper functionality on Windows, we may need to include additional features from pywin32 into the _subprocess.c driver, or alternatively, convert all _subprocess.c bits into ctypes calls. If the inclusion of additional code into _subprocess.c or the use of ctypes is undesireable, this feature could require the *user* to install pywin32 on Windows, which would be unfortunate, but perfectly reasonable. With an asynchronous handler for the subprocess module, a user could ignore or process output from a subprocess as desired or necessary. - Josiah [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 Peter Astrand <astrand@lysator.liu.se> wrote:

On Tue, 13 Jun 2006, Martin Blais wrote:

Hi all. Now let's see if I remember something about my module...

...

In the subprocess module, by default the files handles in the child are inherited from the parent. To ignore a child's output, I can use the stdout or stderr options to send the output to a pipe::

p = Popen(command, stdout=PIPE, stderr=PIPE)

However, this is sensitive to the buffer deadlock problem, where for example the buffer for stderr might become full and a deadlock occurs because the child is blocked on writing to stderr and the parent is blocked on reading from stdout or waiting for the child to finish.

For example, using this command will cause deadlock::

call('cat /boot/vmlinuz'.split(), stdout=PIPE, stderr=PIPE)

Yes, the call convenience function is basically for the case when you are not interested in redirection.

...

Popen.communicate() implements a solution using either select() or multiple threads (under Windows) to read from the pipes, and returns the strings as a result. It works out like this::

p = Popen(command, stdout=PIPE, stderr=PIPE) output, errors = p.communicate() if p.returncode != 0: ?

Now, as a user of the subprocess module, sometimes I just want to call some child process and simply ignore its output, and to do so I am forced to use communicate() as above and wastefully capture and ignore the strings. This is actually quite a common use case. "Just run something, and check the return code".

Yes, this is a common case, and using communicate() is indeed overkill and wasteful.

...

Right now, in order to do this without polluting the parent's output, you cannot use the call() convenience (or is there another way?).

A workaround that works under UNIX is to do this::

FNULL = open('/dev/null', 'w') returncode = call(command, stdout=FNULL, stderr=FNULL)

Yes, this works. You can also do:

returncode = subprocess.call(command, stdout=open('/dev/null', 'w'), stderr=subprocess.STDOUT)

...

Some feedback requested, I'd like to know what you think:

1. Would it not be nice to add a IGNORE constant to subprocess.py that would do this automatically?, i.e. ::

returncode = call(command, stdout=IGNORE, stderr=IGNORE)

Rather than capture and accumulate the output, it would find an appropriate OS-specific way to ignore the output (the /dev/null file above works well under UNIX, how would you do this under Windows? I'm sure we can find something.)

I have a vague feeling of that this has been discussed before, but I cannot find a tracker for this. I guess an IGNORE constant would be nice. Using open('/dev/null', 'w') is only a few more characters to type, but as you say, it's not platform independent.

So, feel free to submit a patch or a Feature Request Tracker.

...

2. call() should be modified to not be sensitive to the deadlock problem, since its interface provides no way to return the contents of the output. The IGNORE value provides a possible solution for this.

How do you suggest the call() should be modified? I'm not really sure it can do more without being more complicated. Being simple is the main purpose of call().

...

3. With the /dev/null file solution, the following code actually works without deadlock, because stderr is never blocked on writing to /dev/null::

p = Popen(command, stdout=PIPE, stderr=IGNORE) text = p.stdout.read() retcode = p.wait()

Any idea how this idiom could be supported using a more portable solution (i.e. how would I make this idiom under Windows, is there some equivalent to /dev/null)?

Yes, as Terry Reedy points out, NUL: can be used.

Regards, /Peter Åstrand <astrand@lysator.liu.se>

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu

There is a related bit of functionality for subprocess that would allow for asynchronous handling of IO to/from the called subprocess. It has been implemented as a recipe [1], but requires the use of additional pywin32 functionality on Windows. As was the case for the original subprocess module, in order to get the proper functionality on Windows, we may need to include additional features from pywin32 into the _subprocess.c driver, or alternatively, convert all _subprocess.c bits into ctypes calls. If the inclusion of additional code into _subprocess.c or the use of ctypes is undesireable, this feature could require the *user* to install pywin32 on Windows, which would be unfortunate, but perfectly reasonable. With an asynchronous handler for the subprocess module, a user could ignore or process output from a subprocess as desired or necessary. - Josiah [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 Peter Astrand <astrand@lysator.liu.se> wrote:

On Tue, 13 Jun 2006, Martin Blais wrote:

Hi all. Now let's see if I remember something about my module...

...

In the subprocess module, by default the files handles in the child are inherited from the parent. To ignore a child's output, I can use the stdout or stderr options to send the output to a pipe::

p = Popen(command, stdout=PIPE, stderr=PIPE)

However, this is sensitive to the buffer deadlock problem, where for example the buffer for stderr might become full and a deadlock occurs because the child is blocked on writing to stderr and the parent is blocked on reading from stdout or waiting for the child to finish.

For example, using this command will cause deadlock::

call('cat /boot/vmlinuz'.split(), stdout=PIPE, stderr=PIPE)

Yes, the call convenience function is basically for the case when you are not interested in redirection.

...

Popen.communicate() implements a solution using either select() or multiple threads (under Windows) to read from the pipes, and returns the strings as a result. It works out like this::

p = Popen(command, stdout=PIPE, stderr=PIPE) output, errors = p.communicate() if p.returncode != 0: ?

Now, as a user of the subprocess module, sometimes I just want to call some child process and simply ignore its output, and to do so I am forced to use communicate() as above and wastefully capture and ignore the strings. This is actually quite a common use case. "Just run something, and check the return code".

Yes, this is a common case, and using communicate() is indeed overkill and wasteful.

...

Right now, in order to do this without polluting the parent's output, you cannot use the call() convenience (or is there another way?).

A workaround that works under UNIX is to do this::

FNULL = open('/dev/null', 'w') returncode = call(command, stdout=FNULL, stderr=FNULL)

Yes, this works. You can also do:

returncode = subprocess.call(command, stdout=open('/dev/null', 'w'), stderr=subprocess.STDOUT)

...

Some feedback requested, I'd like to know what you think:

1. Would it not be nice to add a IGNORE constant to subprocess.py that would do this automatically?, i.e. ::

returncode = call(command, stdout=IGNORE, stderr=IGNORE)

Rather than capture and accumulate the output, it would find an appropriate OS-specific way to ignore the output (the /dev/null file above works well under UNIX, how would you do this under Windows? I'm sure we can find something.)

I have a vague feeling of that this has been discussed before, but I cannot find a tracker for this. I guess an IGNORE constant would be nice. Using open('/dev/null', 'w') is only a few more characters to type, but as you say, it's not platform independent.

So, feel free to submit a patch or a Feature Request Tracker.

...

2. call() should be modified to not be sensitive to the deadlock problem, since its interface provides no way to return the contents of the output. The IGNORE value provides a possible solution for this.

How do you suggest the call() should be modified? I'm not really sure it can do more without being more complicated. Being simple is the main purpose of call().

...

3. With the /dev/null file solution, the following code actually works without deadlock, because stderr is never blocked on writing to /dev/null::

p = Popen(command, stdout=PIPE, stderr=IGNORE) text = p.stdout.read() retcode = p.wait()

Any idea how this idiom could be supported using a more portable solution (i.e. how would I make this idiom under Windows, is there some equivalent to /dev/null)?

Yes, as Terry Reedy points out, NUL: can be used.

Regards, /Peter Åstrand <astrand@lysator.liu.se>

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu