[New-bugs-announce] [issue17986] Alternative async subprocesses (pep 3145)

[Richard Oudkerk]

New submission from Richard Oudkerk:

In the attached file is an experimental implementation of an AsyncPopen
class.  It should work for Python 3.3, 3.4 on Unix and Windows.  Unlike

    http://code.google.com/p/subprocdev

(see #1191964) this does not depend on using time.sleep() and polling.
It also uses the stdlib's _winapi instead of ctypes.

It lets one do Unix-style multiplexing of stdin, stdout, stderr on
Windows (by using overlapped IO).

Differences from normal Popen:

* File objects created for stdin/stdout/stderr using "...=PIPE" are
  non-blocking.

* There are no options for buffering or for universal line endings
  (or unicode).

* There is an extra method

      select(file_list, timeout=None) -> ready_file_list

  which can be used to wait for stdin, stdout, stderr to be ready.
  file_list should be a sublist of [self.stdin, self.stdout, self.stderr]
  with no None elements.  Note that there is no separation between
  "readers" and "writers" the way there is for the normal select()
  function.  On Unix this is implemented using select() or poll().

* On Windows close() can fail with BlockingIOError.  To prevent this
  one must use select() to wait for self.stdin to be ready.

As an example, communicate() can be reimplemented using select() as follows:

    def communicate(p, input):
        buf = memoryview(input)
        collected = collections.defaultdict(list)
        registered = [f for f in (p.stdin, p.stdout, p.stderr)
                      if f is not None]

        while registered:
            for f in p.select(registered):
                if f is p.stdin:
                    if not buf:
                        f.close()
                        registered.remove(f)
                    else:
                        n = f.write(buf)
                        if n is not None:
                            buf = buf[n:]
                elif f in (p.stdout, p.stderr):
                    s = f.read(8192)
                    if s == b'':
                        f.close()
                        registered.remove(f)
                    elif s is not None:
                        collected[f].append(s)
                else:
                    raise RuntimeError('should not get here')

        return (b''.join(collected[p.stdout]) if p.stdout else None,
                b''.join(collected[p.stderr]) if p.stderr else None)

----------
files: asyncsubprocess.py
messages: 189304
nosy: sbt
priority: normal
severity: normal
status: open
title: Alternative async subprocesses (pep 3145)
versions: Python 3.4
Added file: http://bugs.python.org/file30269/asyncsubprocess.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17986>
_______________________________________