<div dir="ltr">By digging into the internals of a subprocess produced by Popen(), you can write in a blocking manner to the stdin pipe, and read in a blocking manner from the stdout/stderr pipe(s). For scripting most command-line operations, the lack of timeouts and the ability to *stop* trying to read is as important as being able to spawn an external process. It kind-of kills that side of the usefulness of Python as a tool for scripting.<div>
<br></div><div>The question is not whether or not a user of Python can dig into the internals, make some calls, then get it to be non-blocking - the existence of two different patches to do so (the most recent of which is from 4 1/2 years ago) shows that it *can* be done. The question is whether or not the desire for the functionality warrants having functions or methods to perform these operations in the standard library.</div>
<div><br></div><div>I and others have claimed that it should go into the standard library. Heck, there was enough of a push that Eric got paid to write his version of the functionality for a GSoC project in 2009. There has even been activity on the bug itself unrelated to deferring discussions as recently as May 2012 (after which activity seems to have paused for reasons I don't know). Some people have raised reasonable questions about the API and implementation, but no one is willing to offer an alternative API that they think would be better, so discussions about implementation of a non-existent API for inclusion are moot.</div>
<div><br></div><div><br></div><div>But honestly, I have approximately zero faith that what I say or do will lead to the inclusion of any changes to the subprocess module. Which is why I'm offering to write a short example that uses asyncio for inclusion in the docs. It's not what I've wanted for almost 9 years, but at least it has a chance of actually happening. I'll take a chance at updating the docs instead of a 3 to 9 month bikeshedding just to lead to rejection any day.</div>
<div><br></div><div><br></div><div>So yeah. Someone want to make a decision? Tell me to write the docs, I will. Tell me to go take a long walk off a short pier, I'll thank you for your time and leave you alone.</div><div>
<br></div><div> - Josiah</div><div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 27, 2014 at 7:18 PM, Terry Reedy <span dir="ltr"><<a href="mailto:tjreedy@udel.edu" target="_blank">tjreedy@udel.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 3/27/2014 9:16 PM, Josiah Carlson wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
You don't understand the point because you don't understand the feature<br>
request or PEP. That is probably my fault for not communicating the<br>
intent better in the past. The feature request and PEP were written to<br>
offer something like the below (or at least enough that the below could<br>
be built with minimal effort):<br>
<br>
def do_login(...):<br>
proc = subprocess.Popen(...)<br>
current = proc.recv(timeout=5)<br>
last_line = current.rstrip().rpartition('\<u></u>n')[-1]<br>
if last_line.endswith('login:'):<br>
proc.send(username)<br>
if proc.readline(timeout=5).<u></u>rstrip().endswith('password:')<u></u>:<br>
proc.send(password)<br>
if 'welcome' in proc.recv(timeout=5).lower():<br>
return proc<br>
proc.kill()<br>
<br>
The API above can be very awkward (as shown :P ), but that's okay. From<br>
those building blocks a (minimally) enterprising user would add<br>
functionality to suit their needs. The existing subprocess module only<br>
offers two methods for *any* amount of communication over pipes with the<br>
subprocess: check_output() and communicate(), only the latter of which<br>
supports sending data (once, limited by system-level pipe buffer<br>
lengths). Neither allow for nontrivial interactions from a single<br>
subprocess.Popen() invocation.<br>
</blockquote>
<br></div></div>
According to my reading of the doc, one should (in the absence of deadlocks, and without having timeouts) be able to use proc.stdin.write and proc.stdout.read. Do those not actually work?<div><div class="h5"><br>
<br>
<br>
The purpose was to be able to communicate<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
in a bidirectional manner with a subprocess without blocking, or<br>
practically speaking, blocking with a timeout. That's where the "async"<br>
term comes from. Again, there was never any intent to have the<br>
functionality be part of asyncore or any other asynchronous sockets<br>
framework, which is why there are no handle_*() methods, readable(),<br>
writable(), etc.<br>
<br>
Your next questions will be: But why bother at all? Why not just build<br>
the piece you need *inside* asyncio? Why does this need anything more?<br>
The answer to those questions are wants and needs. If I'm a user that<br>
needs interactive subprocess handling, I want to be able to do something<br>
like the code snippet above. The last thing I need is to have to rewrite<br>
the way my application/script/whatever handles *everything* just because<br>
a new asynchronous IO library has been included in the Python standard<br>
library - it's a bit like selling you a $300 bicycle when you need a $20<br>
wheel for your scooter.<br>
<br>
That there *now* exists the ability to have async subprocesses as part<br>
of asyncio is a fortunate happenstance, as the necessary underlying<br>
tools for building the above now exist in the standard library. It's a<br>
matter of properly embedding the asyncio-related bits inside a handful<br>
of functions to provide something like the above, which is what I was<br>
offering to write. But why not keep working on the subprocess module?<br>
Yep. Tried that. Coming up on 9 years since I created the feature<br>
request and original Activestate recipe. To go that route is going to be<br>
2-3 times as much work as has already been dedicated to get somewhere<br>
remotely acceptable for inclusion in Python 3.5, but more likely,<br>
subsequent rejection for similar reasons why it has been in limbo.<br>
<br>
But here's the thing: I can build enough using asyncio in 30-40 lines of<br>
Python to offer something like the above API. The problem is that it<br>
really has no natural home. It uses asyncio, so makes no sense to put in<br>
subprocess. It doesn't fit the typical asyncio behavior, so doesn't make<br>
sense to put in asyncio. The required functionality isn't big enough to<br>
warrant a submodule anywhere. Heck, it's even way too small to toss into<br>
an external PyPI module. But in the docs? It would show an atypical, but<br>
not wholly unreasonable use of asyncio (the existing example already<br>
shows what I would consider to be an atypical use of asyncio). It would<br>
provide a good starting point for someone who just wants/needs something<br>
like the snippet above. It is *yet another* use-case for asyncio. And it<br>
could spawn a larger library for offering a more fleshed-out<br>
subprocess-related API, though that is probably more wishful thinking on<br>
my part than anything.<br>
<br>
- Josiah<br>
<br>
<br>
<br>
On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner<br></div></div><div class="">
<<a href="mailto:victor.stinner@gmail.com" target="_blank">victor.stinner@gmail.com</a> <mailto:<a href="mailto:victor.stinner@gmail.com" target="_blank">victor.stinner@gmail.<u></u>com</a>>> wrote:<br>
<br>
2014-03-27 22:52 GMT+01:00 Josiah Carlson <<a href="mailto:josiah.carlson@gmail.com" target="_blank">josiah.carlson@gmail.com</a><br></div>
<mailto:<a href="mailto:josiah.carlson@gmail.com" target="_blank">josiah.carlson@gmail.<u></u>com</a>>>:<div class=""><br>
> * Because it is example docs, maybe a multi-week bikeshedding<br>
discussion<br>
> about API doesn't need to happen (as long as "read line", "read X<br>
bytes",<br>
> "read what is available", and "write this data" - all with<br>
timeouts - are<br>
> shown, people can build everything else they want/need)<br>
<br>
I don't understand this point. Using asyncio, you can read and write a<br>
single byte or a whole line. Using functions like asyncio.wait_for(),<br>
it's easy to add a timeout on such operation.<br>
<br>
Victor<br>
<br>
<br>
<br>
<br>
</div></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
-- <br>
Terry Jan Reedy</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/<u></u>mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com" target="_blank">https://mail.python.org/<u></u>mailman/options/python-dev/<u></u>josiah.carlson%40gmail.com</a><br>
</div></div></blockquote></div><br></div>