[Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

And as I was writing the "thank you" to folks, I hit send too early. Also
thank you to Victor Stinner, Guido, Terry Reedy, and everyone else on this
thread :)

 - Josiah


On Thu, May 29, 2014 at 5:34 PM, Josiah Carlson <josiah.carlson at gmail.com>
wrote:

> Pinging this thread 2 months later with a progress/status update.
>
> To those that have reviewed, commented, helped, or otherwise pushed this
> along, which includes (but is not limited to) Richard Oudkerk, eryksun,
> Giampaolo Rodola, thank you.
>
>
> The short version:
> As far as I can tell, the patch is ready:
> http://bugs.python.org/issue1191964
>
> What is available:
> There are docs, tests, and obviously the functionality. Some code was
> moved from asyncio/windows_utils.py (which has a separate issue here:
> https://code.google.com/p/tulip/issues/detail?id=170). The API was
> changed slightly from what was proposed by Guido:
>
> sent = Popen.write_nonblocking(input, timeout=0)
> data = Popen.read_nonblocking(bufsize=4096)
> data = Popen.read_stderr_nonblocking(bufsize=4096)
>
> As a bonus feature, Windows communicate() calls no longer spawn worker
> threads, and instead use overlapped IO.
>
>
> I'm bringing this back up to python-dev to offer a slightly wider audience
> for commentary/concerns, and hopefully to get a stamp of approval that it
> is ready.
>
> Thank you,
>  - Josiah
>
>
>
>
> On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson <josiah.carlson at gmail.com
> > wrote:
>
>> I've got a patch with partial tests and documentation that I'm holding
>> off on upload because I believe there should be a brief discussion.
>>
>> Long story short, Windows needs a thread to handle writing in a
>> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>>
>> If you'd like to continue following this issue and participate in the
>> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>>
>>  - Josiah
>>
>>
>>
>> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson <
>> josiah.carlson at gmail.com> wrote:
>>
>>>
>>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
>>>> josiah.carlson at gmail.com> wrote:
>>>>
>>>>>
>>>>> If it makes you feel any better, I spent an hour this morning building
>>>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and
>>>>> not even using any part of asyncio (the Windows bits are in msvcrt and
>>>>> _winapi). It works in Python 3.3+. You can see it here:
>>>>> http://pastebin.com/0LpyQtU5
>>>>>
>>>>
>>>> Seeing this makes *me* feel better. I think it's reasonable to add
>>>> (some variant) of that to the subprocess module in Python 3.5. It also
>>>> belongs in the Activestate cookbook. And no, the asyncio module hasn't made
>>>> it obsolete.
>>>>
>>>
>>> Cool.
>>>
>>>  Josiah, you sound upset about the whole thing -- to the point of
>>>> writing unintelligible sentences and passive-aggressive digs at everyone
>>>> reading this list. I'm sorry that something happened that led you feel that
>>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote
>>>> that code snippet -- it is completely clear what you want and why you want
>>>> it, and also what should happen next (a few rounds of code review on the
>>>> tracker).
>>>>
>>>
>>> I'm not always a prat. Something about python-dev brings out parts of me
>>> that I thought I had discarded from my personality years ago. Toss in a bit
>>> of needing to re-explain ideas that I've been trying to explain for almost
>>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>>> pretty much why I won't be rejoining the party here regularly, you are all
>>> better off without me commenting on 95% of threads like I used to.
>>>
>>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>>> was when I spend time on this list. That's *my* issue, not yours. That I
>>> spent any time redirecting my frustration towards you is BS, and if I could
>>> take back the email I sent just before getting Guido's, I would.
>>>
>>> I would advise everyone to write it off as the ramblings of a
>>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>>> accurate. :)
>>>
>>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
>>>> of example code. It doesn't specify the proposed interface, it just
>>>> describes in way too many sentences how it should work, and contains a
>>>> whole lot of references to various rants from which the reader is
>>>> apparently meant to become enlightened. I don't know which of the three
>>>> authors *really* wrote it, and I don't want to know -- I think the PEP is
>>>> irrelevant to the proposed feature, which is of "put it in the bug tracker
>>>> and work from there" category -- presumably the PEP was written based on
>>>> the misunderstanding that having a PEP would make acceptance of the patch
>>>> easier, or because during an earlier bikeshedding round someone said
>>>> "please write a PEP" (someone always says that). I propose to scrap the PEP
>>>> (set the status to Withdrawn) and just work on adding the methods to the
>>>> subprocess module.
>>>>
>>>
>>> I'm not going to argue. The first I read it was 2-3 days ago.
>>>
>>>  If it were me, I'd define three methods, with longer names to clarify
>>>> what they do, e.g.
>>>>
>>>> proc.write_nonblocking(data)
>>>> data = proc.read_nonblocking()
>>>> data = proc.read_stderr_nonblocking()
>>>>
>>>
>>> Easily doable.
>>>
>>> I.e. add _nonblocking to the method names to clarify that they may
>>>> return '' when there's nothing available, and have a separate method for
>>>> reading stderr instead of a flag. And I'd wonder if there should be an
>>>> unambiguous way to detect EOF or whether the caller should just check for
>>>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable
>>>> when the other end is closed, and then the write() will fail. But maybe I
>>>> forget.)
>>>>
>>>> But that's all bikeshedding and it can happen on the tracker or
>>>> directly on the list just as easily; I don't see the need for a PEP.
>>>>
>>>
>>>  Sounds good.
>>>
>>>  - Josiah
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/aca9db64/attachment.html>