<div dir="ltr">*This* is the type of conversation that I wanted to avoid. But I'll answer your questions because I used to do exactly the same thing.<div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 28, 2014 at 3:20 AM, Victor Stinner <span dir="ltr"><<a href="mailto:victor.stinner@gmail.com" target="_blank">victor.stinner@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">2014-03-28 2:16 GMT+01:00 Josiah Carlson <<a href="mailto:josiah.carlson@gmail.com">josiah.carlson@gmail.com</a>>:<br>
<div class="">> def do_login(...):<br>
> proc = subprocess.Popen(...)<br>
> current = proc.recv(timeout=5)<br>
> last_line = current.rstrip().rpartition('\n')[-1]<br>
> if last_line.endswith('login:'):<br>
> proc.send(username)<br>
> if proc.readline(timeout=5).rstrip().endswith('password:'):<br>
> proc.send(password)<br>
> if 'welcome' in proc.recv(timeout=5).lower():<br>
> return proc<br>
> proc.kill()<br>
<br>
</div>I don't understand this example. How is it "asynchronous"? It looks<br>
like blocking calls. In my definition, asynchronous means that you can<br>
call this function twice on two processes, and they will run in<br>
parallel.<br></blockquote><div><br></div><div>In this context, async means not necessarily blocking. If you didn't provide a timeout, it would default to 0, which would return immediately with what was sent and/or received from the subprocess. If you don't believe me, that's fine, but it prevents meaningful discussion.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Using greenlet/eventlet, you can write code which looks blocking, but<br>
runs asynchronously. But I don't think that you are using greenlet or<br>
eventlet here.<br></blockquote><div><br></div><div>You are right. And you are talking about something that is completely out of scope.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I take a look at the implementation:<br>
<a href="http://code.google.com/p/subprocdev/source/browse/subprocess.py" target="_blank">http://code.google.com/p/subprocdev/source/browse/subprocess.py</a><br>
<br>
It doesn't look portable. On Windows, WriteFile() is used. This<br>
function is blocking, or I missed something huge :-) It's much better<br>
if a PEP is portable. Adding time.monotonic() only to Linux would make<br>
the PEP 418 much shorter (4 sentences instead of 10 pages? :-))!<br></blockquote><div><br></div><div>Of course it's not portable. Windows does things differently from other platforms. That's one of the reasons why early versions required pywin32. Before you reply to another message, I would encourage you to read the bug, the pep, and perhaps the recipe I just posted: <a href="http://pastebin.com/0LpyQtU5">http://pastebin.com/0LpyQtU5</a></div>
<div><br></div><div>Or you can try to believe that I have done all of those and believe what I say, especially when I say that I don't believe that spending a lot of time worrying about the original patch/recipe and the GSoC entry. They would all require a lot of work to make reasonably sane, which is why I wrote the minimal recipe above.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The implementation doesn't look reliable:<br>
<br>
def get_conn_maxsize(self, which, maxsize):<br>
# Not 100% certain if I get how this works yet.<br>
if maxsize is None:<br>
maxsize = 1024<br>
...<br>
<br>
This constant 1024 looks arbitrary. On UNIX, a write into a pipe may<br>
block with less bytes (512 bytes).<br></blockquote><div><br></div><div>Testing now I seem to be able to send non-reading subprocesses somewhat arbitrary amounts of data without leading to a block. But I can't test all Linux installations or verify that I'm correct. But whether or not this makes sense is moot, as I don't think it should be merged, and I don't believe anyone thinks it should be merged at this point.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">asyncio has a completly different design. On Windows, it uses<br>
overlapped operations with IOCP event loop. Such operation can be<br>
cancelled. Windows cares of the buffering. On UNIX, non-blocking mode<br>
is used with select() (or something faster like epoll) and asyncio<br>
retries to write more data when the pipe (or any file descriptor used<br>
for process stdin/stdoud/stderr) becomes ready (for reading/writing).<br>
<br>
asyncio design is more reliable and portable.<br></blockquote><div><br></div><div>More reliable, sure. More portable... only because all of the portability heavy lifting has been done and included in Python core. That's one other thing that you aren't understanding - the purpose of trying to have this in the standard library is so that people can use the functionality (async subprocesses) on multiple platforms without needing to write it themselves (poorly), ask on forums of one kind or another, copy and paste from some recipe posted to the internet, etc. It's a strict increase in the functionality and usefulness of the Python standard library and has literally zero backwards compatibility issues.</div>
<div><br></div><div>This is the absolute minimum functionality necessary to make people who need this functionality happy. No, really. Absolute minimum. Sort of what asyncore was - the minimum functionality necessary to have async sockets in Python. Was it dirty? Sure. Was it difficult to use? Some people had issues. Did it work? It worked well enough that people were making money building applications based on asyncore (myself included 10 years ago).</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I don't see how you can implement asynchronous communication with a<br>
subprocess without the complex machinery of an event loop.<br></blockquote><div><br></div><div>Words can have multiple meanings. The meaning of "async" in this context is different from what you believe it to mean, which is part of your confusion. I tried to address this in my last message, but either you didn't read that part, didn't understand that part, or don't believe what I wrote. So let me write it again:</div>
<div><br></div>In this context, "async subprocesses" means the ability to interactively interrogate a subprocess without necessarily blocking on input or output. Everyone posting questions about this on StackOverflow or other forums understands it this way. It *does not mean* that it needs to participate in an event loop, needs to be usable with asyncore, asyncio, Twisted, greenlets, gevent, or otherwise.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">If there is one thing that *I* need for you (and everyone else) to understand and believe in this conversation, it is the above. Do you? Yes? Okay. Now read everything that I've written again. No? Can you explain *why* you don't believe or understand me?</div>
<div class="gmail_quote"><br><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">> The API above can be very awkward (as shown :P ), but that's okay. From<br>
> those building blocks a (minimally) enterprising user would add<br>
> functionality to suit their needs. The existing subprocess module only<br>
> offers two methods for *any* amount of communication over pipes with the<br>
> subprocess: check_output() and communicate(), only the latter of which<br>
> supports sending data (once, limited by system-level pipe buffer lengths).<br>
<br>
</div>As I wrote, it's complex to handle non-blocking file descriptors. You<br>
have to catch EWOULDBLOCK and retries later when the file descriptor<br>
becomes ready. The main thread has to watch for such event on the file<br>
descriptor, or you need a dedicated thread. By the way,<br>
subprocess.communicate() is currently implemented using threads on<br>
Windows.<br></blockquote><div><br></div><div>I know what it takes, I've been writing async sockets for 12 years. I used to maintain asyncore/asynchat and related libraries. Actually, you can thank me for asyncore existing in Python 2.6+ (Giampaolo has done a great job and kept asyncore alive after I stopped participating daily python-dev about 5 years ago, and I can't thank him enough for that).</div>
<div><br></div><div>But to the point: stop bagging on the old patches. No one likes them. We all agree. The question is where do we go from here.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">> Neither allow for nontrivial interactions from a single subprocess.Popen()<br>
> invocation. The purpose was to be able to communicate in a bidirectional<br>
> manner with a subprocess without blocking, or practically speaking, blocking<br>
> with a timeout. That's where the "async" term comes from.<br>
<br>
</div>I call this "non-blocking functions", not "async functions".<br>
<br>
It's quite simple to check if a read will block on not on UNIX. It's<br>
more complex to implement it on Windows. And even more complex to<br>
handle to add a buffer to write().<br></blockquote><div><br></div><div>Okay, call it non-blocking subprocess reads and writes. Whatever you want to call it. And yes, I know what it takes to read and write on Windows... I've done it 3 times now (the original recipe, the original patch, now the above recipe).</div>
<div><br></div><div>But the other piece is that *this* doesn't necessarily need to be 100% robust - I'm not even advocating it to be in the Python standard library anywhere! I've given up on that. But a short example hanging out in the docs? Someone will use it. Someone will run into issues. They will add robustness. They will add functionality. And it will grow into something worth using before being posted to the cheeseshop.</div>
<div><br></div><div>The status quo is that people don't get answers anywhere in the Python docs or the Python stdlib. Python core is noticeably absent in a source of information about how someone would go about using the subprocess module in a completely reasonable and sane manner.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">> Your next questions will be: But why bother at all? Why not just build the<br>
> piece you need *inside* asyncio? Why does this need anything more? The<br>
> answer to those questions are wants and needs. If I'm a user that needs<br>
> interactive subprocess handling, I want to be able to do something like the<br>
> code snippet above. The last thing I need is to have to rewrite the way my<br>
> application/script/whatever handles *everything* just because a new<br>
> asynchronous IO library has been included in the Python standard library -<br>
> it's a bit like selling you a $300 bicycle when you need a $20 wheel for<br>
> your scooter.<br>
<br>
</div>You don't have to rewrite your whole application. If you only want to<br>
use asyncio event loop in a single function, you can use<br>
loop.run_until_complete(do_login) which blocks until the function<br>
completes. The "function" is an asynchronous coroutine in fact.<br></blockquote><div><br></div><div>The point of this conversation is that I was offering to write the handful of wrappers that would make interactions of the form that I showed earlier easy and possible with asyncio. So that a user didn't have to write them themselves.</div>
<div><br></div><div>[snip]</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Even if eval_python_async() is asynchronous, eval_python() function is<br>
blocking so you can write: print("1+1 = %r" % eval_python("1+1"))<br>
without callback nor "yield from".<br>
<br>
Running tasks in parallel is faster than running them in sequence<br>
(almost 5 times faster on my PC).<br></blockquote><div><br></div><div>This is completely unrelated to the conversation.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The syntax in eval_python_async() is close to the API you proposed,<br>
except that you have to add "yield from" in front of "blocking"<br>
functions like read() or drain() (it's the function to flush the stdin<br>
buffer, I'm not sure that it is needed in this example).<br>
<br>
The timeout is on the whole eval_python_async(), but you can as well<br>
using finer timeout on each read/write.<br>
<div class=""><br>
> But here's the thing: I can build enough using asyncio in 30-40 lines of<br>
> Python to offer something like the above API. The problem is that it really<br>
> has no natural home.<br>
<br>
</div>I agree that writing explicit asynchronous code is more complex than<br>
using eventlet. Asynchronous programming is hard.<br></blockquote><div><br></div><div>No, it's not hard. It just requires thinking in a different way. It's the thinking in a different way that's difficult. But I've been doing async sockets programming on and off for 13 years now, so I get it. What I'm offering is to help people *not* do that, because some people have difficulty thinking in that way.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="">> But in the docs? It would show an atypical, but not<br>
> wholly unreasonable use of asyncio (the existing example already shows what<br>
> I would consider to be an atypical use of asyncio).<br>
<br>
</div>The asyncio documentation is still a work-in-progress. I tried to<br>
document all APIs, but there are too few examples and the<br>
documentation is still focused on the API instead of being oriented to<br>
the user of the API.<br>
<br>
Don't hesitate to contribute to the documentation!<br></blockquote><div><br></div><div>So is this the "okay" that I've been waiting with baited breath for?</div><div> </div><div> - Josiah</div><div><br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
We can probably write a simple example showing how to interact with an<br>
interactive program like Python.<br>
<span class=""><font color="#888888"><br>
Victor<br>
</font></span></blockquote></div><br></div></div>