[Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

Pinging this thread 2 months later with a progress/status update.

To those that have reviewed, commented, helped, or otherwise pushed this
along, which includes (but is not limited to) Richard Oudkerk, eryksun,
Giampaolo Rodola, thank you.


The short version:
As far as I can tell, the patch is ready:
http://bugs.python.org/issue1191964

What is available:
There are docs, tests, and obviously the functionality. Some code was moved
from asyncio/windows_utils.py (which has a separate issue here:
https://code.google.com/p/tulip/issues/detail?id=170). The API was changed
slightly from what was proposed by Guido:

sent = Popen.write_nonblocking(input, timeout=0)
data = Popen.read_nonblocking(bufsize=4096)
data = Popen.read_stderr_nonblocking(bufsize=4096)

As a bonus feature, Windows communicate() calls no longer spawn worker
threads, and instead use overlapped IO.


I'm bringing this back up to python-dev to offer a slightly wider audience
for commentary/concerns, and hopefully to get a stamp of approval that it
is ready.

Thank you,
 - Josiah




On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson <josiah.carlson at gmail.com>
wrote:

> I've got a patch with partial tests and documentation that I'm holding off
> on upload because I believe there should be a brief discussion.
>
> Long story short, Windows needs a thread to handle writing in a
> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>
> If you'd like to continue following this issue and participate in the
> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>
>  - Josiah
>
>
>
> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson <josiah.carlson at gmail.com
> > wrote:
>
>>
>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
>>> josiah.carlson at gmail.com> wrote:
>>>
>>>>
>>>> If it makes you feel any better, I spent an hour this morning building
>>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and
>>>> not even using any part of asyncio (the Windows bits are in msvcrt and
>>>> _winapi). It works in Python 3.3+. You can see it here:
>>>> http://pastebin.com/0LpyQtU5
>>>>
>>>
>>> Seeing this makes *me* feel better. I think it's reasonable to add (some
>>> variant) of that to the subprocess module in Python 3.5. It also belongs in
>>> the Activestate cookbook. And no, the asyncio module hasn't made it
>>> obsolete.
>>>
>>
>> Cool.
>>
>>  Josiah, you sound upset about the whole thing -- to the point of
>>> writing unintelligible sentences and passive-aggressive digs at everyone
>>> reading this list. I'm sorry that something happened that led you feel that
>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote
>>> that code snippet -- it is completely clear what you want and why you want
>>> it, and also what should happen next (a few rounds of code review on the
>>> tracker).
>>>
>>
>> I'm not always a prat. Something about python-dev brings out parts of me
>> that I thought I had discarded from my personality years ago. Toss in a bit
>> of needing to re-explain ideas that I've been trying to explain for almost
>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>> pretty much why I won't be rejoining the party here regularly, you are all
>> better off without me commenting on 95% of threads like I used to.
>>
>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>> was when I spend time on this list. That's *my* issue, not yours. That I
>> spent any time redirecting my frustration towards you is BS, and if I could
>> take back the email I sent just before getting Guido's, I would.
>>
>> I would advise everyone to write it off as the ramblings of a
>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>> accurate. :)
>>
>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
>>> of example code. It doesn't specify the proposed interface, it just
>>> describes in way too many sentences how it should work, and contains a
>>> whole lot of references to various rants from which the reader is
>>> apparently meant to become enlightened. I don't know which of the three
>>> authors *really* wrote it, and I don't want to know -- I think the PEP is
>>> irrelevant to the proposed feature, which is of "put it in the bug tracker
>>> and work from there" category -- presumably the PEP was written based on
>>> the misunderstanding that having a PEP would make acceptance of the patch
>>> easier, or because during an earlier bikeshedding round someone said
>>> "please write a PEP" (someone always says that). I propose to scrap the PEP
>>> (set the status to Withdrawn) and just work on adding the methods to the
>>> subprocess module.
>>>
>>
>> I'm not going to argue. The first I read it was 2-3 days ago.
>>
>>  If it were me, I'd define three methods, with longer names to clarify
>>> what they do, e.g.
>>>
>>> proc.write_nonblocking(data)
>>> data = proc.read_nonblocking()
>>> data = proc.read_stderr_nonblocking()
>>>
>>
>> Easily doable.
>>
>> I.e. add _nonblocking to the method names to clarify that they may return
>>> '' when there's nothing available, and have a separate method for reading
>>> stderr instead of a flag. And I'd wonder if there should be an unambiguous
>>> way to detect EOF or whether the caller should just check for
>>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable
>>> when the other end is closed, and then the write() will fail. But maybe I
>>> forget.)
>>>
>>> But that's all bikeshedding and it can happen on the tracker or directly
>>> on the list just as easily; I don't see the need for a PEP.
>>>
>>
>>  Sounds good.
>>
>>  - Josiah
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/d6a1e8d9/attachment-0001.html>