Importance of "async" keyword

Hi everybody, recently, I stumbled over the new 3.5 release and in doing so over PEP 0492. After careful consideration and after reading many blog posts of various coders, I first would like to thank Yury Selivanov and everybody else who brought PEP 0492 to its final state. I therefore considered usage within our projects, however, still find hazy items in PEP 0492. So, I would like to contribute my thoughts on this in order to either increase my understanding or even improve Python's async capability. In order to do this, I need a clarification regarding the rationale behind the async keyword. The PEP rationalizes its introduction with: "If useful() [...] would become a regular python function, [...] important() would be broken." What bothers me is, why should important() be broken in that case? Regards, Sven R. Kunze

Hi Sven, On 2015-06-24 2:14 PM, Sven R. Kunze wrote:
Let me first quote the PEP to make the question more obvious. The section at question is: [1]; it explains why "async" keyword is needed for function declarations. Here's a code example from the PEP: def useful(): ... await log(...) ... def important(): await useful() Sven, if we don't have 'async def', and instead say that "a function is a *coroutine function* when it has at least one 'await' expression", then when you refactor "useful()" by removing the "await" from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the "await useful()" call in the "important()" function will be broken. 'async def' guarantees that function always return a "coroutine"; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. This is also covered in the rationale section of the PEP [2] [1] https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword [2] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals Thanks, Yury P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives.

Thanks, Yury, for you quick response. On 24.06.2015 22:16, Yury Selivanov wrote:
I feared you would say that. Your reasoning assumes that *await* needs an *explicitly declared awaitable*. Let us assume for a moment, we had no async keyword. So, any awaitable needs to have at least 1 await in it. Why can we not have awaitables with 0 awaits in them?
Recently, I read Guido's blog about the history of Python and how he eliminated special cases from Python step by step. As I see it, the same could be done here. What is the difference of a function (no awaits) or an awaitable (> 1 awaits) from an end-user's perspective (i.e. the programmer)? My answer would be: none. When used the same way, they should behave in the same manner. As long as, we get our nice tracebacks when something went wrong, everything seems find to me. Please, correct me if I am wrong.
I can imagine that. As said I went through of some of them, but it could be that I missed some of them as well. Is there a way to search them through (by not using Google)? Regard, Sven

On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote:
I haven't been following the async discussion in detail, but I would expect that the answer is for the same reason that you cannot have a generator function with 0 yields in it.
The first is syncronous, the second is asyncronous.
How is that possible? def func(): # Simulate a time consuming calculation. time.sleep(10000) return 42 # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = func() I think that one of us is missing something here. As I said, I haven't followed the whole discussion, so it might be me. But on face value, I don't think what you say is reasonable.
You can download the mailing list archive for the relevant months, and use your mail client to search them. -- Steve

On 25.06.2015 04:16, Steven D'Aprano wrote:
Ah, wait. That is not what I intended and I completely agree with Guido when he says: "I want to be able to *syntactically* tell where the suspension points are in coroutines." http://code.activestate.com/lists/python-dev/135906/ So, it is either: # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = await func() So, from my perspective, no matter, how many (even zero) suspension points there are in func(), the first variant would still be blocking and the second one not. Many programmers (when adhering to classical programming) conceive the world as if they call a function and simply do not care about how it works and what it does. The main thing is, it does what it is supposed to do. Whether this function achieves the goal by working asynchronously or blocking, AND if I, as a programmer, allow the function to work asynchronously, is a completely different matter. That in turn is extremely well reflected by the 'await' keyword (IMHO at least). Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. One last thing regarding 'async def', that came to my mind recently, is that it compares to that necessity of Java to decorate functions with all possible exceptions that could be raised inside. Thankfully, we do not need to do that in Python. IIRC, it is considered bad practice as it ties modules together tighter as they need to be. If I add a single exception down in the traceback, I need to re-add it everywhere where that function is used. That is a mess regarding clean code. Somehow, 'async def' reminds me of that, too, but maybe, I am missing something here.
Thanks Steven. I already found one on the Web: http://code.activestate.com/lists/python-dev/ And as it seems, I already read through all relevant discussions. :-/

On 26 Jun 2015 05:15, "Andrew Svetlov" <andrew.svetlov@gmail.com> wrote:
Another issue that bothers me, is code reuse. Independent from whether
the that
Exactly, the sync/async split is inherent in the problem domain. Nobody really likes this consequence of explicitly asynchronous programming models, and this is a good article on why it's annoying: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ However, it's also one of those cases like binary floating point: there are good underlying reasons things are the way they are, but truly coming to grips with them can take a long time, so in the meantime time it's necessary to just accept it as "just one of those things" (like learning negative or complex numbers for the first time). For explicitly asynchronous programming, Glyph's post at https://glyph.twistedmatrix.com/2014/02/unyielding.html is a decent starting point on that journey. Cheers, Nick.

On 25.06.2015 21:11, Andrew Svetlov wrote:
My point is: why does everyone assume that it has to be like that? @Nick Thanks for these links; nice reads and reflect exactly what I think about these topics. Btw. complex numbers basically works the same way (same API) as integers. I would like to see that for functions and awaitables as well. Still, the general assumption is: "the sync/async split is inherent in the problem domain". Why? I do not see that inherent split. @Greg I do not like wrappers IF it is just because to have wrappers. We already have 2 types of wrappers: [normal function call] and [function call using await]. Both works like wrappers right in the place where you need them. @Steven
Where would the function suspend if there are zero suspension points?
It would not.
How can you tell what the suspension points *in* the coroutine are from "await func()"?
Let me answer this with a question: How can you tell what the suspension points *in* the coroutine are from "async def"?
Can you show a code snippet of your proposal? By analogy to PEP 0492:
def complex_calc(a): if a >= 0: return await open(unicode(a)).read() l = await complex_calc(a - 1) r = complex_calc(a - 2) return l + '; ' + r def business(): return complex_calc(5) def business_new() return await complex_calc(10) Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works. Regards, Sven

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:
Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works.
Sadly, I have basically no experience in this area -- perhaps that's why Sven's arguments make sense to me. ;) As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, "it's okay if other tasks run while I'm blocking on this call." Maybe we can get there someday. -- ~Ethan~

On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: def data_from_socket(): # Other tasks may run while we wait for data # The socket.read() function has yield points in it data = socket.read(1024, 1) return transmogrify(data) def respond_to_socket(): while True: data = data_from_socket() # We can pretend that socket writes happen instantly, # but if ever we can't write, it'll let other tasks wait while # we're blocked on it. socket.write("Got it, next please!") Do these functions really need to be aware that there are yield points in what they're calling? I come from a background of thinking with threads, so I'm accustomed to doing blocking I/O and assuming/expecting that other threads will carry on while we wait. If asynchronous I/O can be made that convenient, it'd be awesome. But since it hasn't already been made that easy in every other language, I expect there's some fundamental problem with this approach, something that intrinsically requires every step in the chain to know what's a (potential) block point. ChrisA

Hello, On Sat, 27 Jun 2015 00:31:01 +1000 Chris Angelico <rosuav@gmail.com> wrote:
Some say "convenient", others say "dangerous". Consider: buf = bytearray(MULTIMEGABYTE) In one function you do: socket.write(buf), in another function (coroutine), you keep mutating buf. So, currently in Python you know if you do: socket.write(buf) Then you know it will finish without interruptions for entire buffer. And if you write: await socket.write(buf) then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written. With threads, you always have to assume this last scenario, and do extra effort to protect against it (mutexes and stuff). Note that it's original thinking of Guido, but seeing how it all combines in asyncio (far from being nice consistent way), it's not surprising that some people will keep not understanding how asyncio works/how to use it, while other will grow disappointed by it. Here's my latest disappointment wrt to asyncio usage in MicroPython: http://bugs.python.org/issue24449 -- Best regards, Paul mailto:pmiscml@gmail.com

On Sat, Jun 27, 2015 at 12:51 AM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
Hmm. I'm not sure how this is a problem; whether you use 'await' or not, using a mutable object from two separate threads or coroutines is going to have that effect. The way I'm seeing it, coroutines are like cooperatively-switched threads; you don't have to worry about certain operations being interrupted (particularly low-level ones like refcount changes or list growth), but any time you hit an 'await', you have to assume a context switch. That's all very well, but I'm not sure it's that big a problem to accept that any call could context-switch; atomicity is already a concern in other cases, which is why we have principles like EAFP rather than LBYL. There's clearly some benefit to being able to assume that certain operations are uninterruptible (because there's no 'await' anywhere in them), but are they truly so? Signals can already interrupt something _anywhere_:
Any time you're using mutable globals, you have to assume that they can be mutated out from under you. Coroutines don't change this, so is there anything gained by knowing where you can't be interrupted by one specific possible context switch? ChrisA

On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico <rosuav@gmail.com> wrote:
Read Glyph's article, it explains why: https://glyph.twistedmatrix.com/2014/02/unyielding.html
Yes, and you could have an out of memory error anywhere in your program as well. (Don't do things in your signal handlers, set a flag.) But that doesn't change the stuff Glyph talks about (and Guido talks about) about *reasoning* about your code. I did my best to avoid using threads, and never invested the time and effort in Twisted. But I love programming with asyncio for highly concurrent applications. It fits in my brain :) --David

On Sat, Jun 27, 2015 at 2:07 AM, R. David Murray <rdmurray@bitdance.com> wrote:
Makes some fair points, but IMO it emphasizes what I'm saying about atomicity. If you really need it, establish it by something other than just having code that naively progresses through. That's what databasing is for, after all. Threading and signals force you to think about concurrency; other models *may* allow you to pretend it doesn't exist. Anyway, that answers my question about why the explicit "this can let other things run" marker. Thanks all. ChrisA

On Fri Jun 26 16:51:13 CEST 2015, Paul Sokolovsky wrote:
So, currently in Python you know if you do:
socket.write(buf)
Then you know it will finish without interruptions for entire buffer.
How do you know that? Are you assuming that socket.write is a builtin, rather than a python method? (Not even a python wrapper around a builtin?) Even if that were true, it would only mean that the call itself is processed within a single bytecode ... there is no guarantee that the write method won't release the GIL or call back into python (and thereby allow a thread switch) as part of its own logic.
And if you write:
await socket.write(buf)
then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written.
I would consider that external mutation to be bad form ... at least as bad as violating the expectation of an atomic socket.write() up above. So either way, nothing bad SHOULD happen, but it might anyhow. I'm not seeing what the async-coloring actually bought you... -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

On 2015-06-26 10:31 AM, Chris Angelico wrote:
Chris, Sven, if you want this behavior, gevent & Stackless Python are your friends. This approach, however, won't ever be merged in CPython (this was discussed plenty of times both on python-ideas and python-dev). There is also a great essay about explicit vs implicit suspension points: https://glyph.twistedmatrix.com/2014/02/unyielding.html Yury

On 27 June 2015 at 00:31, Chris Angelico <rosuav@gmail.com> wrote:
Folks, it's worth reading through both https://glyph.twistedmatrix.com/2014/02/unyielding.html and http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... It *is* possible to make a language where *everything* is run as a coroutine, and get rid of the subroutine/coroutine distinction that way - a subroutine would *literally* be executed as a coroutine that never waited for anything. The "doesn't suspend" case could then be handled as an implicit optimisation, rather than as a distinct type, and the *only* way to do IO would be asynchronously through an event loop, rather than through blocking API calls. But that language wouldn't be Python. Python's core execution model is a synchronous procedural dynamically typed one, with everything beyond that, whether object oriented programming, functional programming, asynchronous programming, gradual typing, etc, being a way of managing the kinds of complexity that arise when managing more state, or more complicated algorithms, or more interactivity, or a larger development team.
As Glyph explains, implicitly switched coroutines are just shared memory threading under another name - when any function call can cause you to lose control of the flow of execution, you have to consider your locking model in much the same way as you do for preemptive threading. For Python 2, this "lightweight threads" model is available as https://pypi.python.org/pypi/gevent, and there also appears to have been some good recent progress on Python 3 compatibility: https://github.com/gevent/gevent/issues/38 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06/26/2015 10:31 AM, Chris Angelico wrote:
I think "yield points" is a concept that needs to be spelled out a bit clearer in the PEP 492. It seems that those points are defined by other means outside of a function defined with "async def". From the PEP... * It is a SyntaxError to have yield or yield from expressions in an async function. So somewhere in an async function, it needs to "await something" with a yield in it that isn't an async function. This seems to be a bit counter intuitive to me. Or am I missing something? Regards, Ron

On 27 June 2015 at 04:06, Ron Adam <ron3200@gmail.com> wrote:
This isn't the case - it can be async functions and C level coroutines all the way down. Using a generator or other iterable instead requires adaptation to the awaitable protocol (which makes it possible to tap into all the work that has been done for the generator-based coroutines used previously, rather than having to rewrite it to use native coroutines). This isn't very clear in the currently released beta as we made some decisions to simplify the original implementation that we thought would also be OK from an API design perspective, but turned out to pose significant problems once folks actually started trying to integrate native coroutines with other async systems beyond asyncio. Yury fixed those underlying object model limitations as part of addressing Ben Darnell's report of problems attempting to integrate native coroutine support into Tornado (https://hg.python.org/cpython/rev/7a0a1a4ac639). Yury had already updated the PEP to account for those changes, but I've now also added a specific note regarding the API design change in response to beta feedback: https://hg.python.org/peps/rev/0c963fa25db8 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:
Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works.
Assuming "async def business_new" (to avoid the syntax error), there's no difference between those functions or the one they're calling: * "complex_calc" returns an awaitable object that, after you've awaited it, will result in an int. * "business" returns the return value of "complex_calc", which is an awaitable object that, after you've awaited it, will result in an int. * "business_new" returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). The value of the await syntax is when you're doing something interesting, a.k.a. your function is more than delegating directly to another function: async def business_async(): tasks = [complex_calc_async(i) for i in range(10)] results = [await t for t in tasks] await write_to_disk_async(filename, results) return sum(results) Now it's actually useful to be able to await when we choose. Each call to complex_calc_async() could be starting a thread and then suspending until the thread is complete, so we actually start all 10 threads running here before blocking, and then collect the results in the order we started them (and not the order they complete, though I think asyncio has a function to do that). Without the explicit await this would be impossible. The async def also lets us create coroutines consistently even if they don't await anything: if has_version: async def get_version_async(): return VERSION else: async def get_version_async(): return (await get_major_version_async(), await get_minor_version_async()) async def show_version(): print(await get_version_async()) If, like generators, regular functions became coroutines only in the presence of an await, we'd have to do convoluted code to produce the fast get_version_async(), or else make the caller worry about whether they can skip the await. (Also consider calls that cache their result - a coroutine MUST be awaited, but it doesn't have to await anything if it already has the result). (Aside: the "_async" suffix is based on the convention used in C#, and it's certainly one that I'll be using throughout my async Python code and encouraging in any code that I review. It's the most obvious way to let callers know whether they need to await the function result or not.) Cheers, Steve

On 2015-06-26 1:40 PM, Ethan Furman wrote:
"business_new" should be defined with an 'async' keyword, that's where all the confusion came from: async def business_new(): return await complex_calc(10) Now, "business_new()" returns a coroutine (which will resolve to the result of "complex_calc" awaitable), "await business_new()" will return an int. Yury

Hi, I have never said I wanted implicit asyncio. Explicit is the Python way after all, it served me well and I stick to that. I like the 'await' syntax to mark suspension points. But the 'async' coloring makes no sense to me. It is an implementation details of asyncio (IMHO). From what I can gather, there are 4 distinct cases I would need to take care of. However, I do not wish the team to go through of all of them. Additionally, they would even need to think of whether a function returns an 'awaitable that returns an int', an 'int' or both depending on the input parameters (or due to an implementation error). Maybe, somebody even makes a crazy error like returning an 'awaitable that returns an awaitable that returns an int'. If somebody likes to create/handle this kind of stuff (for whatever reason), using the asyncio library and creating crazy wrapper objects would be the way to go; not the default syntax. Regards, Sven

My understanding of coloring is "needs special treatment". Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. Sensible would be something similar to: call function: business as usual await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion await awaitable: suspension point and could be suspended internally as well On 02.07.2015 00:02, Greg Ewing wrote:

On 3 July 2015 at 06:55, Sven R. Kunze <srkunze@mail.de> wrote:
I'm afraid you're going to be disappointed in that regard, as wishing that event driven programs behaved more like synchronous programs is like wishing that complex numbers behaved like real numbers. There's an extra level of complexity that is being deliberately introduced in order to improve Python's ability to express certain kinds of algorithms, and it isn't effective to just try to wish that complexity away. The payoff is that code that otherwise needs to be written as a long series of disconnected callback chains (as has long been possible with Twisted) can instead be written to look more like comparable synchronous code (and this approach should bring with it much improved introspection support, at least in 3.5+ now that gi_yieldfrom and cr_await are being exposed at the Python level).
These both fail, and deliberately so: we don't know what they're supposed to mean, and we refuse the temptation to guess. They're also quite likely to indicate a bug (e.g. forgetting to call a native coroutine function to get the coroutine out of it, forgetting to wait for an awaitable) rather than something folks have done deliberately. It's possible shorthand adapters may emerge over time, like: # Call awaitable from synchronous code def wait_for_result(awaitable): """Usage: result = asyncio.wait_for_result(awaitable)""" return asyncio.get_event_loop().run_until_complete(awaitable.__await__()) # Call blocking operation from asynchronous code def blocking_call(f, *args, **kwds): """Usage: result = await asyncio.blocking_call(f, *args, **kwds))""" cb = functools.partial(f, *args, **kwds) return asyncio.get_event_loop().run_in_executor(cb) However, those can be written already as utility functions, so we can wait and see how strong the demand is for them as adapters. (They may also be potentially useful to have as recipes in the documentation) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote:
1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage.
That is great. So, let's do the next step.
Where do we guess here? It is a definition.
Until a solution, we return to watching and have one reason less to switch to Python 3. :-/

On 6 July 2015 at 10:27, Chris Angelico <rosuav@gmail.com> wrote:
Exactly. While complex numbers are a superset of the real numbers from a value perspective, from a behavioural perspective, there are things you can do when working only with real numbers that you can't do when you have to account for the fact that here might be complex numbers are in the mix. In a Python context, the essential operations specific to real numbers are listed at https://docs.python.org/3/library/numbers.html#numbers.Real There's also a difference between the scope of the math and cmath modules: >>> import math, cmath >>> set(dir(math)) - set(dir(cmath)) {'floor', 'pow', 'erf', 'trunc', 'copysign', 'expm1', 'ldexp', 'fsum', 'erfc', 'lgamma', 'frexp', 'gamma', 'factorial', 'log2', 'fabs', 'log1p', 'atan2', 'hypot', 'modf', 'radians', 'degrees', 'fmod', 'ceil'} It's a general principle of design that expanding the kinds of values you accept (e.g. from real numbers to complex numbers) means you reduce the kinds of operations you can assume will work (e.g. to the behaviours of 2D complex numbers, rather than the 1D real number line). Similarly, as you step further down the numeric tower from real numbers to rationals and integers, you reduce the set of supported values, but you also increase the number of defined behaviours. When it comes to coroutines and subroutines, coroutines are the superset - a subroutine is just a coroutine that never suspends before producing a result. You can thus make certain simplifying assumptions when executing subroutines that you can't make when executing a coroutine. It also turns out that "never suspends" is actually problematic, which is why nominally subroutine based code these days tends to instead have *implicit* suspension points based on either operating system level pre-emptive multithreading where suspension may occur at any point or greenlet style state switching where suspension may occur as part of any function call (whether an explicit call or as part of a syntactic protocol). These approaches provide parallel execution at the expense of the ability to reason locally about code correctness, which then causes all sorts of *other* problems. That said, I think there's definitely value in providing a very simple answer to the "how do I make a blocking call from a coroutine?" question, so I filed an RFE to add asyncio.blocking_call: http://bugs.python.org/issue24571 I'm less convinced of the value of "asyncio.wait_for_result()", so I haven't filed an RFE for that one. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06.07.2015 03:41, Nick Coghlan wrote:
Nice step forward, Nick. Thanks a lot.
I'm less convinced of the value of "asyncio.wait_for_result()", so I haven't filed an RFE for that one.
I have done that for you, because I feel people need to have a convenient tool to **bridge both worlds** in either direction: http://bugs.python.org/issue24578 That is even another issue that came to my mind once in a while but I forgot to post it here: How are mature projects are supposed to make the transition to asyncio when they see performance opportunities in using it? We have several millions lines of code. I actually imagined we could simply drop an 'await' once in a while in order to gain from asyncio's power. As it seems, we need to inconveniently write all sort of wrappers (unnecessarily from my perspective) to retain existing functionality and leverage asyncio's strength at the same time in order not to break anything. That is, in fact, the main reason why I conduct this discussion. I feel this transition is mostly impossible, very costly or only possible for new code (where is remains to be seen whether it fits in the existing codebase). I also feel I am missing something of the bigger picture and I am not sure if something like this is planned for the future. But from my perspective in order to leverage asyncio's power, you need at least two coroutines running at the same time, right? So, in order to get things running, I still need some sort of get_event_loop into which I can put my top-level coroutines. Assume my venerable business functionality: def business_old(): content1 = open('big.file').read() # blocks until finished content2 = open('huge.file').read() # blocks until finished return content1 + content2 I would like to rewrite/amend it to work asynchronously with minimal effort such as: def business_new(): content1 = fork open('big.file').read() # wraps up the calls into awaitables content2 = fork open('huge.file').read() # and put them into the event loop return content1 + content2 # variables are used => await evaluation I might have missed something but I think you get my point. Correct me if I am wrong, but inferring from the given example of PEP 492, currently, we would need to do the following: def business_new_2(): content1 = open('big.file').read() # get us two awaitables/futures content2 = open('huge.file').read() # ... # somehow the loop magic loop = asyncio.get_event_loop() loop.run_until_complete(content1) loop.run_until_complete(content2) try: loop.run_forever() # might be something different finally: loop.close() return content1.result() + content2.result() I certainly do not want to put that into our codebase. Especially when this kind of code block could occur at any level many times in various functions. Regards, Sven

On 7 July 2015 at 06:08, Sven R. Kunze <srkunze@mail.de> wrote:
No, you haven't missed anything, but I think the convenience APIs we're discussing in this thread are what you need, rather than async/await. Specifically, your main application would remain synchronous, but the event loop could be used to run selected operations in a background thread: def business_new(): future1 = asyncio.call_async(open('big.file').read) future2 = asyncio.call_async(open('huge.file').read) content1 = asyncio.wait_for_result(future1) content2 = asyncio.wait_for_result(future2) return content1 + content2 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jul 05, 2015 at 11:50:00PM +0200, Sven R. Kunze wrote:
I don't think that this is a complelling analogy for calling regular functions and awaitable functions. Doing arithmetic is a bit more complicated than just the four elementary operators you mention. Contrast: 10 < 20 10+20j < 20+10j It is not just that complex numbers can do more than real numbers. They can also do *less*. But aside from that, your analogy looks to me like this: (1) I can do arithmetic on reals; (2) I can do arithmetic on complex numbers; (3) and I can do arithmetic on mixed real + complex expressions; (A) I can travel in a car to the shops down the road; (B) I can travel in a space ship to the moon; (C) so why can't I travel in a car to the moon? Or use a space ship to fly to the shops down the road? Because people think we cannot have the best things of both worlds! It seems to me that if "people think we cannot have the best things of both worlds" (your words), it is because there are solid reasons for that belief. Could somebody build a car that can fly to the moon? Probably, but it would cost *more* than the combination of separate space ship plus car, it would require as much maintenance and support as a space ship, and the fuel economy when driving to work would be terrible. Not to mention all the complaints about the noise and the pollution. I think that the distinction between regular and concurrent routines is practical and necessary, *not* just because of people's closed minds, but because of decades of collective experience with them. To convince me differently, you will need more than some rather dubious analogies or arguments from theoretical purity that sequential syncronous code is just a special case of concurrent asyncronous code. An actual working implementation speaks most loudly of all, but at the very least you will need to stick to concrete arguments, not analogies. Are you aware of any other languages which have eliminated the distinction between regular and concurrent functions? If it has already been done, that would make a good argument in favour of your idea. -- Steven

On 6 July 2015 at 10:49, Steven D'Aprano <steve@pearwood.info> wrote:
I actually really like the analogy, as the "Why can't complex numbers be just like real numbers?" reaction is pretty common when folks are first learning to use them for things like signal analysis. I know it didn't really sink in for me at university - it was only a couple of years into doing digital signal processing full time that the concepts involved in switching back and forth between linear real number based time domain analysis and cyclical complex number based frequency domain analysis really started to make sense to me. There's a wonderful page at http://betterexplained.com/articles/a-visual-intuitive-guide-to-imaginary-nu... which not only does a great job of providing a relatively intuitive explanation of the behaviour of complex numbers as an answer to the question "What is the square root of negative 1?", it also compares them to the original reactions to the "absurd" notion of negative numbers as an answer to the question "What is the result of subtracting a larger number from a smaller one?". "Why can't coroutines be just like subroutines?" strikes me as being a similar case where the answer is "because not everything can be appropriately modelled as a subroutine", but that answer isn't going to make intuitive sense if you've never personally encountered the limits of subroutine based algorithm design. I also think the analogy helps provide good design guidance, as folks are already familiar with the notion of "use real numbers if you can, complex numbers if you need to", and extending that to an attitude of "use subroutines if you can, coroutines if you need to" would be a *very* good thing in terms of encouraging maintainable designs. "Are complex numbers better than real numbers?" is hopefully a self-evidently nonsensical question - some things are better modelled as complex numbers, others as real numbers, so the only reasonable answer is to ask "What are you trying to model?". "Are coroutines better than subroutines?" is the same kind of question, just applied to algorithm design rather than numeric modelling. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

"A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why?" Function != awaitable - the answer is right there in the terminology. Different names, different things. Given A, B and the fact that an awaitable is just an object, here's another important point: * I can call a function and might get an awaitable. If the function was decorated with async, the "might" becomes "will", but it's still just a function returning a value. Because an awaitable is indistinguishable from any other value, the compiler doesn't know whether to await or not, so it has to be specified by the developer. If the language specifies it, then it's impossible to handle awaitable objects (for example, to put them in a list and await them later). C# is another language that implemented this exact model a few years ago and it works well. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Sven R. Kunze<mailto:srkunze@mail.de> Sent: 7/5/2015 14:50 To: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Importance of "async" keyword Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote:
1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage.
That is great. So, let's do the next step.
Where do we guess here? It is a definition.
Until a solution, we return to watching and have one reason less to switch to Python 3. :-/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

Ethan Furman wrote:
I assumed "async def business_new()", rather than some imaginary "await in a non-async function will block because I love to create deadlocks in my code" feature. Note that blocking prevents *all* coroutines from making progress, unlike threading. When you await all the way to an event loop, it defers the rest of the coroutine until a signal (via a callback) is raised and continues running other coroutines. Cheers, Steve

On 26 June 2015 at 23:48, Sven R. Kunze <srkunze@mail.de> wrote:
Not using complex numbers in Python - coming to grips with what they mean physically.
I would like to see that for functions and awaitables as well.
This is akin to asking for unification of classes and modules - they're both namespaces so they have a lot of similarities, but the ways in which they're different are by design. Similarly, subroutines and coroutines are not the same thing: https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines They're *related*, but they're still not the same thing. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sven R. Kunze wrote:
Using the word "blocking" this way is potentially confusing. The calling task is *always* blocked until the operation completes. The difference lies in whether *other* tasks are blocked or not.
You understand correctly. If the function is declared async, you must call it with await; if not, you must not. That's not ideal, but it's an unavoidable consequence of the way async/await are implemented.
So, we would have to implement things twice for the asyncio world and the classic world.
Not exactly; it's possible to create a wrapper that takes an async function and runs it to completion, allowing it to be called from sync code. I can't remember offhand, but there's likely something like this already in asyncio.
It's a similar situation, but nowhere near as severe. There are only two possibilities, async or not. Once you've converted a function to async, you won't need to change it again on that score. -- Greg

On 26 Jun 2015 10:46, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Sven R. Kunze wrote:
So, we would have to implement things twice for the asyncio world and
the classic world.
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLo... It's also possible to use a more comprehensive synchronous-to-asynchronous adapter like gevent to call asynchronous code from synchronous code. Going in the other direction (calling sync code from async) uses a thread or process pool: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLo... Cheers, Nick.

On Thu, Jun 25, 2015 at 05:55:53PM +0200, Sven R. Kunze wrote:
On 25.06.2015 04:16, Steven D'Aprano wrote:
On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote:
[...]
I think so. If you call a syncronous function when you want something asyncronous, or vice versa, you're going to be disappointed or confused by the result. [...]
Where would the function suspend if there are zero suspension points? How can you tell what the suspension points *in* the coroutine are from "await func()"?
Concurrency is not classical single-processing programming.
It might help if you give a concrete example. Can you show a code snippet of your proposal? A single function with zero suspension points being called both asyncronously and syncronously, where does it suspend? It doesn't need to be a big complex function, something simple will do.
I expect that it will be simple to write a wrapper that converts an ayncronous function to a syncronous one. It's just a matter of waiting until it's done. -- Steve

Hi Sven, On 2015-06-24 2:14 PM, Sven R. Kunze wrote:
Let me first quote the PEP to make the question more obvious. The section at question is: [1]; it explains why "async" keyword is needed for function declarations. Here's a code example from the PEP: def useful(): ... await log(...) ... def important(): await useful() Sven, if we don't have 'async def', and instead say that "a function is a *coroutine function* when it has at least one 'await' expression", then when you refactor "useful()" by removing the "await" from it, it stops being a *coroutine function*, which means that it won't return an *awaitable* anymore. Hence the "await useful()" call in the "important()" function will be broken. 'async def' guarantees that function always return a "coroutine"; it eliminates the need of using @asyncio.coroutine decorator (or similar), which besides making code easier to read, also improves the performance. Not to mention new 'async for' and 'async with' statements. This is also covered in the rationale section of the PEP [2] [1] https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword [2] https://www.python.org/dev/peps/pep-0492/#rationale-and-goals Thanks, Yury P.S. This and many other things were discussed at length on the mailing lists, I suggest you to browse through the archives.

Thanks, Yury, for you quick response. On 24.06.2015 22:16, Yury Selivanov wrote:
I feared you would say that. Your reasoning assumes that *await* needs an *explicitly declared awaitable*. Let us assume for a moment, we had no async keyword. So, any awaitable needs to have at least 1 await in it. Why can we not have awaitables with 0 awaits in them?
Recently, I read Guido's blog about the history of Python and how he eliminated special cases from Python step by step. As I see it, the same could be done here. What is the difference of a function (no awaits) or an awaitable (> 1 awaits) from an end-user's perspective (i.e. the programmer)? My answer would be: none. When used the same way, they should behave in the same manner. As long as, we get our nice tracebacks when something went wrong, everything seems find to me. Please, correct me if I am wrong.
I can imagine that. As said I went through of some of them, but it could be that I missed some of them as well. Is there a way to search them through (by not using Google)? Regard, Sven

On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote:
I haven't been following the async discussion in detail, but I would expect that the answer is for the same reason that you cannot have a generator function with 0 yields in it.
The first is syncronous, the second is asyncronous.
How is that possible? def func(): # Simulate a time consuming calculation. time.sleep(10000) return 42 # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = func() I think that one of us is missing something here. As I said, I haven't followed the whole discussion, so it might be me. But on face value, I don't think what you say is reasonable.
You can download the mailing list archive for the relevant months, and use your mail client to search them. -- Steve

On 25.06.2015 04:16, Steven D'Aprano wrote:
Ah, wait. That is not what I intended and I completely agree with Guido when he says: "I want to be able to *syntactically* tell where the suspension points are in coroutines." http://code.activestate.com/lists/python-dev/135906/ So, it is either: # Call func syncronously, blocking until the calculation is done: x = func() # Call func asyncronously, without blocking: y = await func() So, from my perspective, no matter, how many (even zero) suspension points there are in func(), the first variant would still be blocking and the second one not. Many programmers (when adhering to classical programming) conceive the world as if they call a function and simply do not care about how it works and what it does. The main thing is, it does what it is supposed to do. Whether this function achieves the goal by working asynchronously or blocking, AND if I, as a programmer, allow the function to work asynchronously, is a completely different matter. That in turn is extremely well reflected by the 'await' keyword (IMHO at least). Another issue that bothers me, is code reuse. Independent from whether the 'async def' makes sense or not, it would not allow us to reuse asyncio functions as if they were normal functions and vice versa (if I understood that correctly). So, we would have to implement things twice for the asyncio world and the classic world. To me, it would be important to use one function in either world where it suits me better. I am uncertain if that makes sense but right now it does to me. One last thing regarding 'async def', that came to my mind recently, is that it compares to that necessity of Java to decorate functions with all possible exceptions that could be raised inside. Thankfully, we do not need to do that in Python. IIRC, it is considered bad practice as it ties modules together tighter as they need to be. If I add a single exception down in the traceback, I need to re-add it everywhere where that function is used. That is a mess regarding clean code. Somehow, 'async def' reminds me of that, too, but maybe, I am missing something here.
Thanks Steven. I already found one on the Web: http://code.activestate.com/lists/python-dev/ And as it seems, I already read through all relevant discussions. :-/

On 26 Jun 2015 05:15, "Andrew Svetlov" <andrew.svetlov@gmail.com> wrote:
Another issue that bothers me, is code reuse. Independent from whether
the that
Exactly, the sync/async split is inherent in the problem domain. Nobody really likes this consequence of explicitly asynchronous programming models, and this is a good article on why it's annoying: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ However, it's also one of those cases like binary floating point: there are good underlying reasons things are the way they are, but truly coming to grips with them can take a long time, so in the meantime time it's necessary to just accept it as "just one of those things" (like learning negative or complex numbers for the first time). For explicitly asynchronous programming, Glyph's post at https://glyph.twistedmatrix.com/2014/02/unyielding.html is a decent starting point on that journey. Cheers, Nick.

On 25.06.2015 21:11, Andrew Svetlov wrote:
My point is: why does everyone assume that it has to be like that? @Nick Thanks for these links; nice reads and reflect exactly what I think about these topics. Btw. complex numbers basically works the same way (same API) as integers. I would like to see that for functions and awaitables as well. Still, the general assumption is: "the sync/async split is inherent in the problem domain". Why? I do not see that inherent split. @Greg I do not like wrappers IF it is just because to have wrappers. We already have 2 types of wrappers: [normal function call] and [function call using await]. Both works like wrappers right in the place where you need them. @Steven
Where would the function suspend if there are zero suspension points?
It would not.
How can you tell what the suspension points *in* the coroutine are from "await func()"?
Let me answer this with a question: How can you tell what the suspension points *in* the coroutine are from "async def"?
Can you show a code snippet of your proposal? By analogy to PEP 0492:
def complex_calc(a): if a >= 0: return await open(unicode(a)).read() l = await complex_calc(a - 1) r = complex_calc(a - 2) return l + '; ' + r def business(): return complex_calc(5) def business_new() return await complex_calc(10) Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works. Regards, Sven

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:
Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works.
Sadly, I have basically no experience in this area -- perhaps that's why Sven's arguments make sense to me. ;) As Nick said earlier: the caller always blocks; by extension (to my mind, at least) putting an `await` in front of something is saying, "it's okay if other tasks run while I'm blocking on this call." Maybe we can get there someday. -- ~Ethan~

On Sat, Jun 27, 2015 at 12:20 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Apologies if this is a really REALLY dumb question, but... How hard would it be to then dispense with the await keyword, and simply _always_ behave that way? Something like: def data_from_socket(): # Other tasks may run while we wait for data # The socket.read() function has yield points in it data = socket.read(1024, 1) return transmogrify(data) def respond_to_socket(): while True: data = data_from_socket() # We can pretend that socket writes happen instantly, # but if ever we can't write, it'll let other tasks wait while # we're blocked on it. socket.write("Got it, next please!") Do these functions really need to be aware that there are yield points in what they're calling? I come from a background of thinking with threads, so I'm accustomed to doing blocking I/O and assuming/expecting that other threads will carry on while we wait. If asynchronous I/O can be made that convenient, it'd be awesome. But since it hasn't already been made that easy in every other language, I expect there's some fundamental problem with this approach, something that intrinsically requires every step in the chain to know what's a (potential) block point. ChrisA

Hello, On Sat, 27 Jun 2015 00:31:01 +1000 Chris Angelico <rosuav@gmail.com> wrote:
Some say "convenient", others say "dangerous". Consider: buf = bytearray(MULTIMEGABYTE) In one function you do: socket.write(buf), in another function (coroutine), you keep mutating buf. So, currently in Python you know if you do: socket.write(buf) Then you know it will finish without interruptions for entire buffer. And if you write: await socket.write(buf) then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written. With threads, you always have to assume this last scenario, and do extra effort to protect against it (mutexes and stuff). Note that it's original thinking of Guido, but seeing how it all combines in asyncio (far from being nice consistent way), it's not surprising that some people will keep not understanding how asyncio works/how to use it, while other will grow disappointed by it. Here's my latest disappointment wrt to asyncio usage in MicroPython: http://bugs.python.org/issue24449 -- Best regards, Paul mailto:pmiscml@gmail.com

On Sat, Jun 27, 2015 at 12:51 AM, Paul Sokolovsky <pmiscml@gmail.com> wrote:
Hmm. I'm not sure how this is a problem; whether you use 'await' or not, using a mutable object from two separate threads or coroutines is going to have that effect. The way I'm seeing it, coroutines are like cooperatively-switched threads; you don't have to worry about certain operations being interrupted (particularly low-level ones like refcount changes or list growth), but any time you hit an 'await', you have to assume a context switch. That's all very well, but I'm not sure it's that big a problem to accept that any call could context-switch; atomicity is already a concern in other cases, which is why we have principles like EAFP rather than LBYL. There's clearly some benefit to being able to assume that certain operations are uninterruptible (because there's no 'await' anywhere in them), but are they truly so? Signals can already interrupt something _anywhere_:
Any time you're using mutable globals, you have to assume that they can be mutated out from under you. Coroutines don't change this, so is there anything gained by knowing where you can't be interrupted by one specific possible context switch? ChrisA

On Sat, 27 Jun 2015 01:10:33 +1000, Chris Angelico <rosuav@gmail.com> wrote:
Read Glyph's article, it explains why: https://glyph.twistedmatrix.com/2014/02/unyielding.html
Yes, and you could have an out of memory error anywhere in your program as well. (Don't do things in your signal handlers, set a flag.) But that doesn't change the stuff Glyph talks about (and Guido talks about) about *reasoning* about your code. I did my best to avoid using threads, and never invested the time and effort in Twisted. But I love programming with asyncio for highly concurrent applications. It fits in my brain :) --David

On Sat, Jun 27, 2015 at 2:07 AM, R. David Murray <rdmurray@bitdance.com> wrote:
Makes some fair points, but IMO it emphasizes what I'm saying about atomicity. If you really need it, establish it by something other than just having code that naively progresses through. That's what databasing is for, after all. Threading and signals force you to think about concurrency; other models *may* allow you to pretend it doesn't exist. Anyway, that answers my question about why the explicit "this can let other things run" marker. Thanks all. ChrisA

On Fri Jun 26 16:51:13 CEST 2015, Paul Sokolovsky wrote:
So, currently in Python you know if you do:
socket.write(buf)
Then you know it will finish without interruptions for entire buffer.
How do you know that? Are you assuming that socket.write is a builtin, rather than a python method? (Not even a python wrapper around a builtin?) Even if that were true, it would only mean that the call itself is processed within a single bytecode ... there is no guarantee that the write method won't release the GIL or call back into python (and thereby allow a thread switch) as part of its own logic.
And if you write:
await socket.write(buf)
then you know there may be interruption points inside socket.write(), in particular something else may mutate it while it's being written.
I would consider that external mutation to be bad form ... at least as bad as violating the expectation of an atomic socket.write() up above. So either way, nothing bad SHOULD happen, but it might anyhow. I'm not seeing what the async-coloring actually bought you... -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

On 2015-06-26 10:31 AM, Chris Angelico wrote:
Chris, Sven, if you want this behavior, gevent & Stackless Python are your friends. This approach, however, won't ever be merged in CPython (this was discussed plenty of times both on python-ideas and python-dev). There is also a great essay about explicit vs implicit suspension points: https://glyph.twistedmatrix.com/2014/02/unyielding.html Yury

On 27 June 2015 at 00:31, Chris Angelico <rosuav@gmail.com> wrote:
Folks, it's worth reading through both https://glyph.twistedmatrix.com/2014/02/unyielding.html and http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... It *is* possible to make a language where *everything* is run as a coroutine, and get rid of the subroutine/coroutine distinction that way - a subroutine would *literally* be executed as a coroutine that never waited for anything. The "doesn't suspend" case could then be handled as an implicit optimisation, rather than as a distinct type, and the *only* way to do IO would be asynchronously through an event loop, rather than through blocking API calls. But that language wouldn't be Python. Python's core execution model is a synchronous procedural dynamically typed one, with everything beyond that, whether object oriented programming, functional programming, asynchronous programming, gradual typing, etc, being a way of managing the kinds of complexity that arise when managing more state, or more complicated algorithms, or more interactivity, or a larger development team.
As Glyph explains, implicitly switched coroutines are just shared memory threading under another name - when any function call can cause you to lose control of the flow of execution, you have to consider your locking model in much the same way as you do for preemptive threading. For Python 2, this "lightweight threads" model is available as https://pypi.python.org/pypi/gevent, and there also appears to have been some good recent progress on Python 3 compatibility: https://github.com/gevent/gevent/issues/38 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06/26/2015 10:31 AM, Chris Angelico wrote:
I think "yield points" is a concept that needs to be spelled out a bit clearer in the PEP 492. It seems that those points are defined by other means outside of a function defined with "async def". From the PEP... * It is a SyntaxError to have yield or yield from expressions in an async function. So somewhere in an async function, it needs to "await something" with a yield in it that isn't an async function. This seems to be a bit counter intuitive to me. Or am I missing something? Regards, Ron

On 27 June 2015 at 04:06, Ron Adam <ron3200@gmail.com> wrote:
This isn't the case - it can be async functions and C level coroutines all the way down. Using a generator or other iterable instead requires adaptation to the awaitable protocol (which makes it possible to tap into all the work that has been done for the generator-based coroutines used previously, rather than having to rewrite it to use native coroutines). This isn't very clear in the currently released beta as we made some decisions to simplify the original implementation that we thought would also be OK from an API design perspective, but turned out to pose significant problems once folks actually started trying to integrate native coroutines with other async systems beyond asyncio. Yury fixed those underlying object model limitations as part of addressing Ben Darnell's report of problems attempting to integrate native coroutine support into Tornado (https://hg.python.org/cpython/rev/7a0a1a4ac639). Yury had already updated the PEP to account for those changes, but I've now also added a specific note regarding the API design change in response to beta feedback: https://hg.python.org/peps/rev/0c963fa25db8 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06/26/2015 06:48 AM, Sven R. Kunze wrote:
Maybe, I completely missed the point of the proposal, but this is the way I would expect it to work. Putting in an 'await' whenever I see fit and it just works.
Assuming "async def business_new" (to avoid the syntax error), there's no difference between those functions or the one they're calling: * "complex_calc" returns an awaitable object that, after you've awaited it, will result in an int. * "business" returns the return value of "complex_calc", which is an awaitable object that, after you've awaited it, will result in an int. * "business_new" returns an awaitable object that, after you've awaited it, will result in an int. In all three of these cases, the result is the same. The fact that the awaitable object returned from any of them is implemented by a coroutine isn't important (in the same way that an iterable object may be implemented by a generator, but it's really irrelevant). The value of the await syntax is when you're doing something interesting, a.k.a. your function is more than delegating directly to another function: async def business_async(): tasks = [complex_calc_async(i) for i in range(10)] results = [await t for t in tasks] await write_to_disk_async(filename, results) return sum(results) Now it's actually useful to be able to await when we choose. Each call to complex_calc_async() could be starting a thread and then suspending until the thread is complete, so we actually start all 10 threads running here before blocking, and then collect the results in the order we started them (and not the order they complete, though I think asyncio has a function to do that). Without the explicit await this would be impossible. The async def also lets us create coroutines consistently even if they don't await anything: if has_version: async def get_version_async(): return VERSION else: async def get_version_async(): return (await get_major_version_async(), await get_minor_version_async()) async def show_version(): print(await get_version_async()) If, like generators, regular functions became coroutines only in the presence of an await, we'd have to do convoluted code to produce the fast get_version_async(), or else make the caller worry about whether they can skip the await. (Also consider calls that cache their result - a coroutine MUST be awaited, but it doesn't have to await anything if it already has the result). (Aside: the "_async" suffix is based on the convention used in C#, and it's certainly one that I'll be using throughout my async Python code and encouraging in any code that I review. It's the most obvious way to let callers know whether they need to await the function result or not.) Cheers, Steve

On 2015-06-26 1:40 PM, Ethan Furman wrote:
"business_new" should be defined with an 'async' keyword, that's where all the confusion came from: async def business_new(): return await complex_calc(10) Now, "business_new()" returns a coroutine (which will resolve to the result of "complex_calc" awaitable), "await business_new()" will return an int. Yury

Hi, I have never said I wanted implicit asyncio. Explicit is the Python way after all, it served me well and I stick to that. I like the 'await' syntax to mark suspension points. But the 'async' coloring makes no sense to me. It is an implementation details of asyncio (IMHO). From what I can gather, there are 4 distinct cases I would need to take care of. However, I do not wish the team to go through of all of them. Additionally, they would even need to think of whether a function returns an 'awaitable that returns an int', an 'int' or both depending on the input parameters (or due to an implementation error). Maybe, somebody even makes a crazy error like returning an 'awaitable that returns an awaitable that returns an int'. If somebody likes to create/handle this kind of stuff (for whatever reason), using the asyncio library and creating crazy wrapper objects would be the way to go; not the default syntax. Regards, Sven

My understanding of coloring is "needs special treatment". Being special or not (containing an 'await' or not), as long as I don't need to care, I can call them either way (with 'await' or not) and each case works sensibly that's fine with me. Sensible would be something similar to: call function: business as usual await function: suspension point and runs the function until completion call awaitable: runs the awaitable until completion await awaitable: suspension point and could be suspended internally as well On 02.07.2015 00:02, Greg Ewing wrote:

On 3 July 2015 at 06:55, Sven R. Kunze <srkunze@mail.de> wrote:
I'm afraid you're going to be disappointed in that regard, as wishing that event driven programs behaved more like synchronous programs is like wishing that complex numbers behaved like real numbers. There's an extra level of complexity that is being deliberately introduced in order to improve Python's ability to express certain kinds of algorithms, and it isn't effective to just try to wish that complexity away. The payoff is that code that otherwise needs to be written as a long series of disconnected callback chains (as has long been possible with Twisted) can instead be written to look more like comparable synchronous code (and this approach should bring with it much improved introspection support, at least in 3.5+ now that gi_yieldfrom and cr_await are being exposed at the Python level).
These both fail, and deliberately so: we don't know what they're supposed to mean, and we refuse the temptation to guess. They're also quite likely to indicate a bug (e.g. forgetting to call a native coroutine function to get the coroutine out of it, forgetting to wait for an awaitable) rather than something folks have done deliberately. It's possible shorthand adapters may emerge over time, like: # Call awaitable from synchronous code def wait_for_result(awaitable): """Usage: result = asyncio.wait_for_result(awaitable)""" return asyncio.get_event_loop().run_until_complete(awaitable.__await__()) # Call blocking operation from asynchronous code def blocking_call(f, *args, **kwds): """Usage: result = await asyncio.blocking_call(f, *args, **kwds))""" cb = functools.partial(f, *args, **kwds) return asyncio.get_event_loop().run_in_executor(cb) However, those can be written already as utility functions, so we can wait and see how strong the demand is for them as adapters. (They may also be potentially useful to have as recipes in the documentation) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote:
1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage.
That is great. So, let's do the next step.
Where do we guess here? It is a definition.
Until a solution, we return to watching and have one reason less to switch to Python 3. :-/

On 6 July 2015 at 10:27, Chris Angelico <rosuav@gmail.com> wrote:
Exactly. While complex numbers are a superset of the real numbers from a value perspective, from a behavioural perspective, there are things you can do when working only with real numbers that you can't do when you have to account for the fact that here might be complex numbers are in the mix. In a Python context, the essential operations specific to real numbers are listed at https://docs.python.org/3/library/numbers.html#numbers.Real There's also a difference between the scope of the math and cmath modules: >>> import math, cmath >>> set(dir(math)) - set(dir(cmath)) {'floor', 'pow', 'erf', 'trunc', 'copysign', 'expm1', 'ldexp', 'fsum', 'erfc', 'lgamma', 'frexp', 'gamma', 'factorial', 'log2', 'fabs', 'log1p', 'atan2', 'hypot', 'modf', 'radians', 'degrees', 'fmod', 'ceil'} It's a general principle of design that expanding the kinds of values you accept (e.g. from real numbers to complex numbers) means you reduce the kinds of operations you can assume will work (e.g. to the behaviours of 2D complex numbers, rather than the 1D real number line). Similarly, as you step further down the numeric tower from real numbers to rationals and integers, you reduce the set of supported values, but you also increase the number of defined behaviours. When it comes to coroutines and subroutines, coroutines are the superset - a subroutine is just a coroutine that never suspends before producing a result. You can thus make certain simplifying assumptions when executing subroutines that you can't make when executing a coroutine. It also turns out that "never suspends" is actually problematic, which is why nominally subroutine based code these days tends to instead have *implicit* suspension points based on either operating system level pre-emptive multithreading where suspension may occur at any point or greenlet style state switching where suspension may occur as part of any function call (whether an explicit call or as part of a syntactic protocol). These approaches provide parallel execution at the expense of the ability to reason locally about code correctness, which then causes all sorts of *other* problems. That said, I think there's definitely value in providing a very simple answer to the "how do I make a blocking call from a coroutine?" question, so I filed an RFE to add asyncio.blocking_call: http://bugs.python.org/issue24571 I'm less convinced of the value of "asyncio.wait_for_result()", so I haven't filed an RFE for that one. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 06.07.2015 03:41, Nick Coghlan wrote:
Nice step forward, Nick. Thanks a lot.
I'm less convinced of the value of "asyncio.wait_for_result()", so I haven't filed an RFE for that one.
I have done that for you, because I feel people need to have a convenient tool to **bridge both worlds** in either direction: http://bugs.python.org/issue24578 That is even another issue that came to my mind once in a while but I forgot to post it here: How are mature projects are supposed to make the transition to asyncio when they see performance opportunities in using it? We have several millions lines of code. I actually imagined we could simply drop an 'await' once in a while in order to gain from asyncio's power. As it seems, we need to inconveniently write all sort of wrappers (unnecessarily from my perspective) to retain existing functionality and leverage asyncio's strength at the same time in order not to break anything. That is, in fact, the main reason why I conduct this discussion. I feel this transition is mostly impossible, very costly or only possible for new code (where is remains to be seen whether it fits in the existing codebase). I also feel I am missing something of the bigger picture and I am not sure if something like this is planned for the future. But from my perspective in order to leverage asyncio's power, you need at least two coroutines running at the same time, right? So, in order to get things running, I still need some sort of get_event_loop into which I can put my top-level coroutines. Assume my venerable business functionality: def business_old(): content1 = open('big.file').read() # blocks until finished content2 = open('huge.file').read() # blocks until finished return content1 + content2 I would like to rewrite/amend it to work asynchronously with minimal effort such as: def business_new(): content1 = fork open('big.file').read() # wraps up the calls into awaitables content2 = fork open('huge.file').read() # and put them into the event loop return content1 + content2 # variables are used => await evaluation I might have missed something but I think you get my point. Correct me if I am wrong, but inferring from the given example of PEP 492, currently, we would need to do the following: def business_new_2(): content1 = open('big.file').read() # get us two awaitables/futures content2 = open('huge.file').read() # ... # somehow the loop magic loop = asyncio.get_event_loop() loop.run_until_complete(content1) loop.run_until_complete(content2) try: loop.run_forever() # might be something different finally: loop.close() return content1.result() + content2.result() I certainly do not want to put that into our codebase. Especially when this kind of code block could occur at any level many times in various functions. Regards, Sven

On 7 July 2015 at 06:08, Sven R. Kunze <srkunze@mail.de> wrote:
No, you haven't missed anything, but I think the convenience APIs we're discussing in this thread are what you need, rather than async/await. Specifically, your main application would remain synchronous, but the event loop could be used to run selected operations in a background thread: def business_new(): future1 = asyncio.call_async(open('big.file').read) future2 = asyncio.call_async(open('huge.file').read) content1 = asyncio.wait_for_result(future1) content2 = asyncio.wait_for_result(future2) return content1 + content2 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jul 05, 2015 at 11:50:00PM +0200, Sven R. Kunze wrote:
I don't think that this is a complelling analogy for calling regular functions and awaitable functions. Doing arithmetic is a bit more complicated than just the four elementary operators you mention. Contrast: 10 < 20 10+20j < 20+10j It is not just that complex numbers can do more than real numbers. They can also do *less*. But aside from that, your analogy looks to me like this: (1) I can do arithmetic on reals; (2) I can do arithmetic on complex numbers; (3) and I can do arithmetic on mixed real + complex expressions; (A) I can travel in a car to the shops down the road; (B) I can travel in a space ship to the moon; (C) so why can't I travel in a car to the moon? Or use a space ship to fly to the shops down the road? Because people think we cannot have the best things of both worlds! It seems to me that if "people think we cannot have the best things of both worlds" (your words), it is because there are solid reasons for that belief. Could somebody build a car that can fly to the moon? Probably, but it would cost *more* than the combination of separate space ship plus car, it would require as much maintenance and support as a space ship, and the fuel economy when driving to work would be terrible. Not to mention all the complaints about the noise and the pollution. I think that the distinction between regular and concurrent routines is practical and necessary, *not* just because of people's closed minds, but because of decades of collective experience with them. To convince me differently, you will need more than some rather dubious analogies or arguments from theoretical purity that sequential syncronous code is just a special case of concurrent asyncronous code. An actual working implementation speaks most loudly of all, but at the very least you will need to stick to concrete arguments, not analogies. Are you aware of any other languages which have eliminated the distinction between regular and concurrent functions? If it has already been done, that would make a good argument in favour of your idea. -- Steven

On 6 July 2015 at 10:49, Steven D'Aprano <steve@pearwood.info> wrote:
I actually really like the analogy, as the "Why can't complex numbers be just like real numbers?" reaction is pretty common when folks are first learning to use them for things like signal analysis. I know it didn't really sink in for me at university - it was only a couple of years into doing digital signal processing full time that the concepts involved in switching back and forth between linear real number based time domain analysis and cyclical complex number based frequency domain analysis really started to make sense to me. There's a wonderful page at http://betterexplained.com/articles/a-visual-intuitive-guide-to-imaginary-nu... which not only does a great job of providing a relatively intuitive explanation of the behaviour of complex numbers as an answer to the question "What is the square root of negative 1?", it also compares them to the original reactions to the "absurd" notion of negative numbers as an answer to the question "What is the result of subtracting a larger number from a smaller one?". "Why can't coroutines be just like subroutines?" strikes me as being a similar case where the answer is "because not everything can be appropriately modelled as a subroutine", but that answer isn't going to make intuitive sense if you've never personally encountered the limits of subroutine based algorithm design. I also think the analogy helps provide good design guidance, as folks are already familiar with the notion of "use real numbers if you can, complex numbers if you need to", and extending that to an attitude of "use subroutines if you can, coroutines if you need to" would be a *very* good thing in terms of encouraging maintainable designs. "Are complex numbers better than real numbers?" is hopefully a self-evidently nonsensical question - some things are better modelled as complex numbers, others as real numbers, so the only reasonable answer is to ask "What are you trying to model?". "Are coroutines better than subroutines?" is the same kind of question, just applied to algorithm design rather than numeric modelling. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

"A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why?" Function != awaitable - the answer is right there in the terminology. Different names, different things. Given A, B and the fact that an awaitable is just an object, here's another important point: * I can call a function and might get an awaitable. If the function was decorated with async, the "might" becomes "will", but it's still just a function returning a value. Because an awaitable is indistinguishable from any other value, the compiler doesn't know whether to await or not, so it has to be specified by the developer. If the language specifies it, then it's impossible to handle awaitable objects (for example, to put them in a list and await them later). C# is another language that implemented this exact model a few years ago and it works well. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Sven R. Kunze<mailto:srkunze@mail.de> Sent: 7/5/2015 14:50 To: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Importance of "async" keyword Thanks, Nick, for you reasoned response. On 03.07.2015 11:40, Nick Coghlan wrote:
1) I can add, subtract, multiply and divide real numbers. 2) I can add, subtract, multiply and divide complex numbers. 3) I can even add, subtract, multiply and divide complex numbers AND real numbers altogether in a single expression. 4) Granted, complex numbers can do more but that basically follows from their definition and does not jeopardize ease of usage. A) I can call a function and might get a return value. B) I can await an awaitable and might get a return value. C) I cannot use them interchangeably. Why? Because people think we cannot have the best things of both worlds. D) Granted, awaitables can do more but that basically follows from their definition and does not need to jeopardize ease of usage.
That is great. So, let's do the next step.
Where do we guess here? It is a definition.
Until a solution, we return to watching and have one reason less to switch to Python 3. :-/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

Ethan Furman wrote:
I assumed "async def business_new()", rather than some imaginary "await in a non-async function will block because I love to create deadlocks in my code" feature. Note that blocking prevents *all* coroutines from making progress, unlike threading. When you await all the way to an event loop, it defers the rest of the coroutine until a signal (via a callback) is raised and continues running other coroutines. Cheers, Steve

On 26 June 2015 at 23:48, Sven R. Kunze <srkunze@mail.de> wrote:
Not using complex numbers in Python - coming to grips with what they mean physically.
I would like to see that for functions and awaitables as well.
This is akin to asking for unification of classes and modules - they're both namespaces so they have a lot of similarities, but the ways in which they're different are by design. Similarly, subroutines and coroutines are not the same thing: https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines They're *related*, but they're still not the same thing. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sven R. Kunze wrote:
Using the word "blocking" this way is potentially confusing. The calling task is *always* blocked until the operation completes. The difference lies in whether *other* tasks are blocked or not.
You understand correctly. If the function is declared async, you must call it with await; if not, you must not. That's not ideal, but it's an unavoidable consequence of the way async/await are implemented.
So, we would have to implement things twice for the asyncio world and the classic world.
Not exactly; it's possible to create a wrapper that takes an async function and runs it to completion, allowing it to be called from sync code. I can't remember offhand, but there's likely something like this already in asyncio.
It's a similar situation, but nowhere near as severe. There are only two possibilities, async or not. Once you've converted a function to async, you won't need to change it again on that score. -- Greg

On 26 Jun 2015 10:46, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Sven R. Kunze wrote:
So, we would have to implement things twice for the asyncio world and
the classic world.
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLo... It's also possible to use a more comprehensive synchronous-to-asynchronous adapter like gevent to call asynchronous code from synchronous code. Going in the other direction (calling sync code from async) uses a thread or process pool: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLo... Cheers, Nick.

On Thu, Jun 25, 2015 at 05:55:53PM +0200, Sven R. Kunze wrote:
On 25.06.2015 04:16, Steven D'Aprano wrote:
On Wed, Jun 24, 2015 at 11:21:54PM +0200, Sven R. Kunze wrote:
[...]
I think so. If you call a syncronous function when you want something asyncronous, or vice versa, you're going to be disappointed or confused by the result. [...]
Where would the function suspend if there are zero suspension points? How can you tell what the suspension points *in* the coroutine are from "await func()"?
Concurrency is not classical single-processing programming.
It might help if you give a concrete example. Can you show a code snippet of your proposal? A single function with zero suspension points being called both asyncronously and syncronously, where does it suspend? It doesn't need to be a big complex function, something simple will do.
I expect that it will be simple to write a wrapper that converts an ayncronous function to a syncronous one. It's just a matter of waiting until it's done. -- Steve
participants (13)
-
Andrew Svetlov
-
Chris Angelico
-
Ethan Furman
-
Greg Ewing
-
Jim J. Jewett
-
Nick Coghlan
-
Paul Sokolovsky
-
R. David Murray
-
Ron Adam
-
Steve Dower
-
Steven D'Aprano
-
Sven R. Kunze
-
Yury Selivanov