PEP 492 quibble and request

From the PEP:
Why not a __future__ import
__future__ imports are inconvenient and easy to forget to add.
That is a horrible rationale for not using an import. By that logic we should have everything in built-ins. ;)
Working example ...
The working example only uses async def and await, not async with nor async for nor __aenter__, etc., etc. Could you put in a more complete example -- maybe a basic chat room with both server and client -- that demonstrated more of the new possibilities? Having gone through the PEP again, I am still no closer to understanding what happens here: data = await reader.read(8192) What does the flow of control look like at the interpreter level? -- ~Ethan~

Hi Ethan, On 2015-04-29 8:21 PM, Ethan Furman wrote:
Andrew Svetlov has implemented some new features in his aiomysql driver: https://github.com/aio-libs/aiomysql/blob/await/tests/test_async_iter.py I don't want to cite it in the PEP because it's not complete yet, and some idioms (like 'async with') aren't used to their full potential.
'await' is semantically equivalent to 'yield from' in this line. To really understand all implementation details of this line you need to read PEP 3156 and experiment with asyncio. There is no easier way, unfortunately. I can't add a super detailed explanation how event loops can be implemented in PEP 492, that's not in its scope. The good news is that to use asyncio on a daily basis you don't need to know all details, as you don't need to know how 'ceval.c' works and how 'setTimeout' is implemented in JavaScript. Yury

Hello, On Wed, 29 Apr 2015 20:33:09 -0400 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
+1 But if you really want, you can. The likely reason for that though would be desire to develop "yield from" for an alternative Python implementation. You can take inspiration from a diagram I drew while I implemented "yield from" for MicroPython: https://dl.dropboxusercontent.com/u/44884329/yield-from.pdf -- Best regards, Paul mailto:pmiscml@gmail.com

On 30 April 2015 at 10:33, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
From my own perspective, I've spent a reasonable amount of time attempting to explain to folks the "modal" nature of generators, in
This request isn't about understanding the implementation details, it's about understanding what Python *users* will gain from the PEP without *needing* to understand the implementation details. For that, I'd like to see some not-completely-trivial example code in (or at least linked from) the PEP written using: * trollius (no "yield from", Python 2 compatible, akin to Twisted's inlineDeferred's) * asyncio/tulip ("yield from", Python 3.3+ compatible) * PEP 492 (async/await, Python 3.5+ only) The *intent* of PEP 492, like PEP 380 and PEP 342 before it, is "make asynchronous programming in Python easier". I think it will actually succeed in that goal, but I don't think it currently does an especially job of explaining that to folks that aren't both already deeply invested in the explicitly asynchronous programming model *and* thoroughly aware of the fact that most of us need asynchronous programming to look as much like synchronous programming as possible in order for it to fit our brains. Some folks can fit ravioli code with callbacks going everywhere in their brains, but I can't, and it's my experience that most other folks can't either. This lack means the PEP that gets confused objections from folks that wish explicitly asynchronous programming models would just go away entirely (they won't), as well as from folks that already understand it and don't see why we can't continue treating it as a special case of other features that folks have to learn how to use from first principles, rather than saving them that up front learning cost by doing the pattern extraction to make event driven explicitly asynchronous programming its own first class concept with dedicated syntax (a hint on that front: with statements are just particular patterns for using try/except/finally, decorators are just particular patterns in using higher order functions, for statements are just particular patterns in using while statements and builtins, and even imports and classes just represent particular patterns in combining dictionaries, code execution and the metaclass machinery - the pattern extraction and dedicated syntax associated with all of them makes it possible to learn to *use* these concepts without first having to learn how to *implement* them) that you can use them both as pure iterable data sources *and* as coroutines (as per PEP 342). The problem I've found is that our current approach ends up failing the "conceptually different things should also look superficially different" test: outside certain data pipeline processing problems, generators-as-iterators and generators-as-coroutines mostly end up being fundamentally *different* ways of approaching a programming problem, but the current shared syntax encourages users to attempt to place them in the same mental bucket. Those users will remain eternally confused until they learn to place them in two different buckets despite the shared syntax (the 5 or so different meanings of "static" in C++ come to mind at this point...). With explicitly asynchronous development*, this problem of needing to learn to segment the world into two pieces is particularly important, and this wonderful rant on red & blue functions published a couple of months ago helps explain why: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ The dedicated async/await syntax proposed in PEP 492 lets the end user mostly *not care* about all the messiness that needs to happen under the covers to make explicitly asynchronous programming as close to normal synchronous programming as possible. The fact that under the hood in CPython normal functions, coroutines, and generators are implemented largely as variant behaviours of the same type (producing respectively the requested result, an object expecting to be driven by an event loop to produce the requested result, and an object expecting to be driven be an iterative loop to produce a succession of requested values rather than a single result) can (and at least arguably should) be irrelevant to the mental model formed by future Python programmers (see http://uxoslo.com/2014/01/14/ux-hints-why-mental-models-matter/ for more on the difference between the representational models we present directly to end users and the underlying implementation models we use as programmers to actually make things work) Regards, Nick. P.S. *While it's not a substitute for explicitly asynchronous development, implicitly asynchronous code still has an important role to play as one of the 3 models (together with thread pools and blocking on the event loop while running a coroutine to completion) that lets synchronous code play nice with asynchronous code: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick, On 2015-04-29 9:52 PM, Nick Coghlan wrote:
I'll see what I can do with aiomysql library to showcase async for and async with. I have a few outstanding tasks with reference implementation to test/add though. I'm not sure that trollius will add anything to it. It's just like asyncio, but uses 'yield' instead of 'yield from' and was envisioned by Victor Stinner as a transitional-framework to ease the porting of openstack to python 3.
Agree. This confusion of trying to fit two fundamentally different programming models in one syntax is what led me to start thinking about PEP 492 ideas several years ago. And the absence of 'async with' and 'async for' statements forced me to use greenlets, which is another reason for the PEP.
I'll see how I can incorporate your thoughts into Abstract and Rationale sections. Thanks, Yury

On 30 April 2015 at 02:52, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree 100% on this. As things stand, it feels as though asyncio feels frighteningly complex for anyone who isn't deeply involved with it (it certainly does to me). The current PEP feels to an outsider as if it is solving problems that are specialist and to the average (non-asyncio) programmer add language complexity with no real benefit. It's not specific to PEP 492, but what I would like to see is: 1. More tutorial level examples of how to use asyncio. Specifically *not* examples of how to write web services, or how to do async web requests in your existing async program. Instead, how to integrate asyncio into generally non-async code. For example, looking at pip, I see a few places where I can anticipate asyncio might be useful - the link-chasing code, the package download code, and the code to run setup.py in a subprocess, seem like places where we could do stuff in an async manner (it's not a big enough deal that we've ever used threads, so I doubt we'd want to use asyncio either in practice, but they are certainly the *types* of code I see as benefitting from async). 2. Following on from this, how do I isolate async code from the rest of my program (i.e. I don't want to have to rewrite my whole program around an event loop just to run a bunch of background programs in parallel)? 3. Clarification on the roles of async/await vs yield from/generator.send. Are they both useful, and if so in what contexts (ignoring "if you want to support Python 3.4" compatibility cases)? How should a programmer choose which is appropriate? 4. A much better explanation of *when* any of the async constructs are appropriate at all. The name "asyncio" implies IO, and all of the examples further tend to imply "sockets". So the immediate impression is that only socket programmers and people writing network protocols should care. Of course, if asyncio and the PEP *are* only really relevant to network protocols, then my impressions are actually correct and I should drop out of the discussion. But if that's the case, it seems like a lot of language change for a relatively specialist use case.

On Thu, 30 Apr 2015 10:02:17 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
That's my impression too. There's nothing remotely horrible about "yield from". Although we should have done the right thing from the start, this is a lot of language churn to introduce *now*, not to mention annoyingly similar but incompatible constructs that make the language more difficult to understand. So I'm rather -0.5 on the idea. Regards Antoine.

On 30 April 2015 at 10:21, Ethan Furman <ethan@stoneleaf.us> wrote:
It is also makes things more painful than they need to be for syntax highlighters. 'as' went through the "not really a keyword" path, and it's a recipe for complexity in the code generation toolchain and general quirkiness as things behave in unexpected ways. We have a defined process for introducing new keywords (i.e. __future__ imports) and the PEP doesn't adequately make the case for why we shouldn't use it here. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Apr 29, 2015 at 5:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This response is silly. The point is not against import but against __future__. A __future__ import definitely is inconvenient -- few people I know could even recite the correct constraints on their placement.
It is also makes things more painful than they need to be for syntax highlighters.
Does it? Do highlighters even understand __future__ imports? I wouldn't mind if a highlighter always highlighted 'async' and 'await' as keywords even where they aren't yet -- since they will be in 3.7.
I don't recall that -- but it was a really long time ago so I may misremember (did we even have __future__ at the time?).
That's fair. But because of the difficulty in introducing new keywords, many proposals have been shot down or discouraged (or changed to use punctuation characters or abuse existing keywords) -- we should give Yury some credit for figuring out a way around this. Honestly I'm personally on the fence. -- --Guido van Rossum (python.org/~guido)

On 30 April 2015 at 11:12, Guido van Rossum <guido@python.org> wrote:
Yeah, that's a good point.
I don't actually know, I only know about its former pseudo-keyword status because we made it a real keyword as part of "from __future__ import with_statement". I think I was conflating it with the hassles we encountered at various points due to None, True, and False not being real keywords in Python 2, but I don't believe the problems we had with those apply here (given that we won't be using 'await' and 'async' as values in any context that the bytecode generation chain cares about).
Yeah, I'm coming around to the idea. For the async pseudo-keyword, I can see that the proposal only allows its use in cases that were previously entirely illegal, but I'm not yet clear on how the PEP proposes to avoid changing the meaning of the following code: x = await(this_is_a_function_call) Unless I'm misreading the proposed grammar in the PEP (which is entirely possible), I believe PEP 492 would reinterpret that as: x = await this_is_not_a_function_call_any_more Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, but here's the other clever bit: it's only interpreted this way *inside* a function declared with 'async def'. Outside such functions, 'await' is not a keyword, so that grammar rule doesn't trigger. (Kind of similar to the way that the print_function __future__ disables the keyword-ness of 'print', except here it's toggled on or off depending on whether the nearest surrounding scope is 'async def' or not. The PEP could probably be clearer about this; it's all hidden in the Transition Plan section.) -- --Guido van Rossum (python.org/~guido)

On 30 April 2015 at 12:31, Guido van Rossum <guido@python.org> wrote:
Ah, nice, although even reading the Transition Plan section didn't clue me in to that particular aspect of the idea :) Given that clarification, I think the rationale for "no __future__ statement needed" can be strengthened by focusing on the fact that such a statement would largely be *redundant*, given that: * "async def", "async with", and "async for" are all currently syntax errors, and hence adding them is backwards compatible if "async" is otherwise treated as a normal variable name * "await <expr>" only gains its new interpretation when used inside an "async def" statement, so "async def" fills the role that a module level compiler declaration like "from __future__ import async_functions" would otherwise fill That said, it may be worth having the future statement *anyway* as: 1. It gives us the runtime accessible record of the feature transition in the __future__ module 2. It lets folks opt-in to the full keyword implementation from day 1 if they prefer, addressing the tokenisation limitation noted in the PEP: https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2015-04-29 11:01 PM, Nick Coghlan wrote:
Thanks, Nick. I've fixed the Transition Plan section, and rewrote the "why not __future__" one too. https://hg.python.org/peps/rev/552773d7e085 https://hg.python.org/peps/rev/5db3ad3d540b Yury

On Wed, Apr 29, 2015 at 07:31:22PM -0700, Guido van Rossum wrote:
You mean we could write code like this? def await(x): ... if condition: async def spam(): await (eggs or cheese) else: def spam(): await(eggs or cheese) I must admit that's kind of cool, but I'm sure I'd regret it. -- Steve

On 05/01/2015 07:54 AM, Steven D'Aprano wrote:
Actually in the above... def await(x): return x Then in any scope where async is used, the keyword will mask the await function. Are the following correct? Another useful async function might be... async def yielding(): pass In a routine is taking very long time, just inserting "await yielding()" in the long calculation would let other awaitables run. If the async loop only has one coroutine (awaitable) in it, then it will be just like calling a regular function. No waiting would occur. -Ron

On Fri, May 1, 2015 at 12:49 PM, Ron Adam <ron3200@gmail.com> wrote:
provided by the event loop or scheduler framework you're using.
If the async loop only has one coroutine (awaitable) in it, then it will be just like calling a regular function. No waiting would occur.
-- --Guido van Rossum (python.org/~guido)

On 1 May 2015 at 20:59, Guido van Rossum <guido@python.org> wrote:
Really? I was under the impression that 'await yielding()' as defined above would actually not suspend the coroutine at all, therefore not giving any opportunity for the scheduler to resume another coroutine, and I thought I understood the PEP well enough. Does this mean that somehow "await x" guarantees that the coroutine will suspend at least once? To me the async def above was the equivalent of the following in the 'yield from' world: def yielding(): return yield # Just to make it a generator Then "yield from yielding()" will not yield at all - which makes its name rather misleading! -- Arnaud

On 05/02/2015 04:18 PM, Arnaud Delobelle wrote:
It was my understanding that yield from also suspends the current thread, allowing others to run. Of course if it's the only thread, it would not. But maybe I'm misremembering earlier discussions. If it doesn't suspend the current thread, then you need to put scheduler.sleep() calls throughout your co-routines. I think Guido is saying it could be either. Cheers, Ron

On Sat, May 2, 2015 at 1:18 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
You're correct. That's why I said it should be left up to the framework -- ultimately what you *do* have to put in such a function has to be understood by the framenwork. And that's why in other messages I've used await asyncio.sleep(0) as an example. Look up its definition.
-- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
No. First, it's possible for x to finish without yielding. But even if x yields, there is no guarantee that the scheduler will run something else -- it might just resume the same task, even if there is another one that could run. It's up to the scheduler whether it implements any kind of "fair" scheduling policy. -- Greg

On 05/03/2015 03:03 PM, Arnaud Delobelle wrote:
Guido is correct of course. In examples I've used before with trampolines, a co-routine would be yielded back to the event loop, and if there was any other co-routines in the event loop they would execute first. I'm not sure if async and await can be used with a trampoline type scheduler. A scheduler might use a timer or priority based system system to schedule events. So yes, it's up to the scheduler and the pep492 is intended to be flexible as to what scheduler is used. Cheers, Ron

On Wed, Apr 29, 2015 at 06:12:37PM -0700, Guido van Rossum wrote:
Are you talking about actual Python programmers, or people who dabble with the odd Python script now and again? I'm kinda shocked if it's the first. It's not a complex rule: the __future__ import must be the first line of actual executable code in the file, so it can come after any encoding cookie, module docstring, comments and blank lines, but before any other code. The only part I didn't remember was that you can have multiple __future__ imports, I thought they all had to be on one line. (Nice to learn something new!) [...]
I have a memory of much rejoicing when "as" was made a keyword, and an emphatic "we're never going to do that again!" about semi-keywords. I've tried searching for the relevant post(s), but cannot find anything. Maybe I imagined it? But I do have Python 2.4 available, when we could write lovely code like this: py> import math as as py> as <module 'math' from '/usr/lib/python2.4/lib-dynload/mathmodule.so'> I'm definitely not looking forward to anything like that again. -- Steve

The high-level answer to this is, like yield, the function temporarily returns all the way up the stack to the caller who intends to advance the iterator/async function. This is typically the event loop, and the main confusion here comes when the loop is implicit. If you explicitly define an event loop it's roughly equivalent to the for loop that is calling next on a generator. For pip, I expect that's what you'd have - a blocking function ("do_work()"?, "execute_plan()"?) that creates a loop and starts it's tasks running within it. Each task inside that call with be asynchronous with respect to each other (think about passing generators to zip() ) but the loop will block the rest of your code until they're all finished. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: 4/30/2015 2:07 To: Greg Ewing<mailto:greg.ewing@canterbury.ac.nz> Cc: Python Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] PEP 492 quibble and request On 30 April 2015 at 09:58, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Well, if I don't know, I get confused as to where I invoke the event loop, how my non-async code runs alongside this etc. Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

Hi Ethan, On 2015-04-29 8:21 PM, Ethan Furman wrote:
Andrew Svetlov has implemented some new features in his aiomysql driver: https://github.com/aio-libs/aiomysql/blob/await/tests/test_async_iter.py I don't want to cite it in the PEP because it's not complete yet, and some idioms (like 'async with') aren't used to their full potential.
'await' is semantically equivalent to 'yield from' in this line. To really understand all implementation details of this line you need to read PEP 3156 and experiment with asyncio. There is no easier way, unfortunately. I can't add a super detailed explanation how event loops can be implemented in PEP 492, that's not in its scope. The good news is that to use asyncio on a daily basis you don't need to know all details, as you don't need to know how 'ceval.c' works and how 'setTimeout' is implemented in JavaScript. Yury

Hello, On Wed, 29 Apr 2015 20:33:09 -0400 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
+1 But if you really want, you can. The likely reason for that though would be desire to develop "yield from" for an alternative Python implementation. You can take inspiration from a diagram I drew while I implemented "yield from" for MicroPython: https://dl.dropboxusercontent.com/u/44884329/yield-from.pdf -- Best regards, Paul mailto:pmiscml@gmail.com

On 30 April 2015 at 10:33, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
From my own perspective, I've spent a reasonable amount of time attempting to explain to folks the "modal" nature of generators, in
This request isn't about understanding the implementation details, it's about understanding what Python *users* will gain from the PEP without *needing* to understand the implementation details. For that, I'd like to see some not-completely-trivial example code in (or at least linked from) the PEP written using: * trollius (no "yield from", Python 2 compatible, akin to Twisted's inlineDeferred's) * asyncio/tulip ("yield from", Python 3.3+ compatible) * PEP 492 (async/await, Python 3.5+ only) The *intent* of PEP 492, like PEP 380 and PEP 342 before it, is "make asynchronous programming in Python easier". I think it will actually succeed in that goal, but I don't think it currently does an especially job of explaining that to folks that aren't both already deeply invested in the explicitly asynchronous programming model *and* thoroughly aware of the fact that most of us need asynchronous programming to look as much like synchronous programming as possible in order for it to fit our brains. Some folks can fit ravioli code with callbacks going everywhere in their brains, but I can't, and it's my experience that most other folks can't either. This lack means the PEP that gets confused objections from folks that wish explicitly asynchronous programming models would just go away entirely (they won't), as well as from folks that already understand it and don't see why we can't continue treating it as a special case of other features that folks have to learn how to use from first principles, rather than saving them that up front learning cost by doing the pattern extraction to make event driven explicitly asynchronous programming its own first class concept with dedicated syntax (a hint on that front: with statements are just particular patterns for using try/except/finally, decorators are just particular patterns in using higher order functions, for statements are just particular patterns in using while statements and builtins, and even imports and classes just represent particular patterns in combining dictionaries, code execution and the metaclass machinery - the pattern extraction and dedicated syntax associated with all of them makes it possible to learn to *use* these concepts without first having to learn how to *implement* them) that you can use them both as pure iterable data sources *and* as coroutines (as per PEP 342). The problem I've found is that our current approach ends up failing the "conceptually different things should also look superficially different" test: outside certain data pipeline processing problems, generators-as-iterators and generators-as-coroutines mostly end up being fundamentally *different* ways of approaching a programming problem, but the current shared syntax encourages users to attempt to place them in the same mental bucket. Those users will remain eternally confused until they learn to place them in two different buckets despite the shared syntax (the 5 or so different meanings of "static" in C++ come to mind at this point...). With explicitly asynchronous development*, this problem of needing to learn to segment the world into two pieces is particularly important, and this wonderful rant on red & blue functions published a couple of months ago helps explain why: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ The dedicated async/await syntax proposed in PEP 492 lets the end user mostly *not care* about all the messiness that needs to happen under the covers to make explicitly asynchronous programming as close to normal synchronous programming as possible. The fact that under the hood in CPython normal functions, coroutines, and generators are implemented largely as variant behaviours of the same type (producing respectively the requested result, an object expecting to be driven by an event loop to produce the requested result, and an object expecting to be driven be an iterative loop to produce a succession of requested values rather than a single result) can (and at least arguably should) be irrelevant to the mental model formed by future Python programmers (see http://uxoslo.com/2014/01/14/ux-hints-why-mental-models-matter/ for more on the difference between the representational models we present directly to end users and the underlying implementation models we use as programmers to actually make things work) Regards, Nick. P.S. *While it's not a substitute for explicitly asynchronous development, implicitly asynchronous code still has an important role to play as one of the 3 models (together with thread pools and blocking on the event loop while running a coroutine to completion) that lets synchronous code play nice with asynchronous code: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick, On 2015-04-29 9:52 PM, Nick Coghlan wrote:
I'll see what I can do with aiomysql library to showcase async for and async with. I have a few outstanding tasks with reference implementation to test/add though. I'm not sure that trollius will add anything to it. It's just like asyncio, but uses 'yield' instead of 'yield from' and was envisioned by Victor Stinner as a transitional-framework to ease the porting of openstack to python 3.
Agree. This confusion of trying to fit two fundamentally different programming models in one syntax is what led me to start thinking about PEP 492 ideas several years ago. And the absence of 'async with' and 'async for' statements forced me to use greenlets, which is another reason for the PEP.
I'll see how I can incorporate your thoughts into Abstract and Rationale sections. Thanks, Yury

On 30 April 2015 at 02:52, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree 100% on this. As things stand, it feels as though asyncio feels frighteningly complex for anyone who isn't deeply involved with it (it certainly does to me). The current PEP feels to an outsider as if it is solving problems that are specialist and to the average (non-asyncio) programmer add language complexity with no real benefit. It's not specific to PEP 492, but what I would like to see is: 1. More tutorial level examples of how to use asyncio. Specifically *not* examples of how to write web services, or how to do async web requests in your existing async program. Instead, how to integrate asyncio into generally non-async code. For example, looking at pip, I see a few places where I can anticipate asyncio might be useful - the link-chasing code, the package download code, and the code to run setup.py in a subprocess, seem like places where we could do stuff in an async manner (it's not a big enough deal that we've ever used threads, so I doubt we'd want to use asyncio either in practice, but they are certainly the *types* of code I see as benefitting from async). 2. Following on from this, how do I isolate async code from the rest of my program (i.e. I don't want to have to rewrite my whole program around an event loop just to run a bunch of background programs in parallel)? 3. Clarification on the roles of async/await vs yield from/generator.send. Are they both useful, and if so in what contexts (ignoring "if you want to support Python 3.4" compatibility cases)? How should a programmer choose which is appropriate? 4. A much better explanation of *when* any of the async constructs are appropriate at all. The name "asyncio" implies IO, and all of the examples further tend to imply "sockets". So the immediate impression is that only socket programmers and people writing network protocols should care. Of course, if asyncio and the PEP *are* only really relevant to network protocols, then my impressions are actually correct and I should drop out of the discussion. But if that's the case, it seems like a lot of language change for a relatively specialist use case.

On Thu, 30 Apr 2015 10:02:17 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
That's my impression too. There's nothing remotely horrible about "yield from". Although we should have done the right thing from the start, this is a lot of language churn to introduce *now*, not to mention annoyingly similar but incompatible constructs that make the language more difficult to understand. So I'm rather -0.5 on the idea. Regards Antoine.

On 30 April 2015 at 10:21, Ethan Furman <ethan@stoneleaf.us> wrote:
It is also makes things more painful than they need to be for syntax highlighters. 'as' went through the "not really a keyword" path, and it's a recipe for complexity in the code generation toolchain and general quirkiness as things behave in unexpected ways. We have a defined process for introducing new keywords (i.e. __future__ imports) and the PEP doesn't adequately make the case for why we shouldn't use it here. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Apr 29, 2015 at 5:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This response is silly. The point is not against import but against __future__. A __future__ import definitely is inconvenient -- few people I know could even recite the correct constraints on their placement.
It is also makes things more painful than they need to be for syntax highlighters.
Does it? Do highlighters even understand __future__ imports? I wouldn't mind if a highlighter always highlighted 'async' and 'await' as keywords even where they aren't yet -- since they will be in 3.7.
I don't recall that -- but it was a really long time ago so I may misremember (did we even have __future__ at the time?).
That's fair. But because of the difficulty in introducing new keywords, many proposals have been shot down or discouraged (or changed to use punctuation characters or abuse existing keywords) -- we should give Yury some credit for figuring out a way around this. Honestly I'm personally on the fence. -- --Guido van Rossum (python.org/~guido)

On 30 April 2015 at 11:12, Guido van Rossum <guido@python.org> wrote:
Yeah, that's a good point.
I don't actually know, I only know about its former pseudo-keyword status because we made it a real keyword as part of "from __future__ import with_statement". I think I was conflating it with the hassles we encountered at various points due to None, True, and False not being real keywords in Python 2, but I don't believe the problems we had with those apply here (given that we won't be using 'await' and 'async' as values in any context that the bytecode generation chain cares about).
Yeah, I'm coming around to the idea. For the async pseudo-keyword, I can see that the proposal only allows its use in cases that were previously entirely illegal, but I'm not yet clear on how the PEP proposes to avoid changing the meaning of the following code: x = await(this_is_a_function_call) Unless I'm misreading the proposed grammar in the PEP (which is entirely possible), I believe PEP 492 would reinterpret that as: x = await this_is_not_a_function_call_any_more Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Apr 29, 2015 at 7:07 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, but here's the other clever bit: it's only interpreted this way *inside* a function declared with 'async def'. Outside such functions, 'await' is not a keyword, so that grammar rule doesn't trigger. (Kind of similar to the way that the print_function __future__ disables the keyword-ness of 'print', except here it's toggled on or off depending on whether the nearest surrounding scope is 'async def' or not. The PEP could probably be clearer about this; it's all hidden in the Transition Plan section.) -- --Guido van Rossum (python.org/~guido)

On 30 April 2015 at 12:31, Guido van Rossum <guido@python.org> wrote:
Ah, nice, although even reading the Transition Plan section didn't clue me in to that particular aspect of the idea :) Given that clarification, I think the rationale for "no __future__ statement needed" can be strengthened by focusing on the fact that such a statement would largely be *redundant*, given that: * "async def", "async with", and "async for" are all currently syntax errors, and hence adding them is backwards compatible if "async" is otherwise treated as a normal variable name * "await <expr>" only gains its new interpretation when used inside an "async def" statement, so "async def" fills the role that a module level compiler declaration like "from __future__ import async_functions" would otherwise fill That said, it may be worth having the future statement *anyway* as: 1. It gives us the runtime accessible record of the feature transition in the __future__ module 2. It lets folks opt-in to the full keyword implementation from day 1 if they prefer, addressing the tokenisation limitation noted in the PEP: https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2015-04-29 11:01 PM, Nick Coghlan wrote:
Thanks, Nick. I've fixed the Transition Plan section, and rewrote the "why not __future__" one too. https://hg.python.org/peps/rev/552773d7e085 https://hg.python.org/peps/rev/5db3ad3d540b Yury

On Wed, Apr 29, 2015 at 07:31:22PM -0700, Guido van Rossum wrote:
You mean we could write code like this? def await(x): ... if condition: async def spam(): await (eggs or cheese) else: def spam(): await(eggs or cheese) I must admit that's kind of cool, but I'm sure I'd regret it. -- Steve

On 05/01/2015 07:54 AM, Steven D'Aprano wrote:
Actually in the above... def await(x): return x Then in any scope where async is used, the keyword will mask the await function. Are the following correct? Another useful async function might be... async def yielding(): pass In a routine is taking very long time, just inserting "await yielding()" in the long calculation would let other awaitables run. If the async loop only has one coroutine (awaitable) in it, then it will be just like calling a regular function. No waiting would occur. -Ron

On Fri, May 1, 2015 at 12:49 PM, Ron Adam <ron3200@gmail.com> wrote:
provided by the event loop or scheduler framework you're using.
If the async loop only has one coroutine (awaitable) in it, then it will be just like calling a regular function. No waiting would occur.
-- --Guido van Rossum (python.org/~guido)

On 1 May 2015 at 20:59, Guido van Rossum <guido@python.org> wrote:
Really? I was under the impression that 'await yielding()' as defined above would actually not suspend the coroutine at all, therefore not giving any opportunity for the scheduler to resume another coroutine, and I thought I understood the PEP well enough. Does this mean that somehow "await x" guarantees that the coroutine will suspend at least once? To me the async def above was the equivalent of the following in the 'yield from' world: def yielding(): return yield # Just to make it a generator Then "yield from yielding()" will not yield at all - which makes its name rather misleading! -- Arnaud

On 05/02/2015 04:18 PM, Arnaud Delobelle wrote:
It was my understanding that yield from also suspends the current thread, allowing others to run. Of course if it's the only thread, it would not. But maybe I'm misremembering earlier discussions. If it doesn't suspend the current thread, then you need to put scheduler.sleep() calls throughout your co-routines. I think Guido is saying it could be either. Cheers, Ron

On Sat, May 2, 2015 at 1:18 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
You're correct. That's why I said it should be left up to the framework -- ultimately what you *do* have to put in such a function has to be understood by the framenwork. And that's why in other messages I've used await asyncio.sleep(0) as an example. Look up its definition.
-- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
No. First, it's possible for x to finish without yielding. But even if x yields, there is no guarantee that the scheduler will run something else -- it might just resume the same task, even if there is another one that could run. It's up to the scheduler whether it implements any kind of "fair" scheduling policy. -- Greg

On 05/03/2015 03:03 PM, Arnaud Delobelle wrote:
Guido is correct of course. In examples I've used before with trampolines, a co-routine would be yielded back to the event loop, and if there was any other co-routines in the event loop they would execute first. I'm not sure if async and await can be used with a trampoline type scheduler. A scheduler might use a timer or priority based system system to schedule events. So yes, it's up to the scheduler and the pep492 is intended to be flexible as to what scheduler is used. Cheers, Ron

On Wed, Apr 29, 2015 at 06:12:37PM -0700, Guido van Rossum wrote:
Are you talking about actual Python programmers, or people who dabble with the odd Python script now and again? I'm kinda shocked if it's the first. It's not a complex rule: the __future__ import must be the first line of actual executable code in the file, so it can come after any encoding cookie, module docstring, comments and blank lines, but before any other code. The only part I didn't remember was that you can have multiple __future__ imports, I thought they all had to be on one line. (Nice to learn something new!) [...]
I have a memory of much rejoicing when "as" was made a keyword, and an emphatic "we're never going to do that again!" about semi-keywords. I've tried searching for the relevant post(s), but cannot find anything. Maybe I imagined it? But I do have Python 2.4 available, when we could write lovely code like this: py> import math as as py> as <module 'math' from '/usr/lib/python2.4/lib-dynload/mathmodule.so'> I'm definitely not looking forward to anything like that again. -- Steve

The high-level answer to this is, like yield, the function temporarily returns all the way up the stack to the caller who intends to advance the iterator/async function. This is typically the event loop, and the main confusion here comes when the loop is implicit. If you explicitly define an event loop it's roughly equivalent to the for loop that is calling next on a generator. For pip, I expect that's what you'd have - a blocking function ("do_work()"?, "execute_plan()"?) that creates a loop and starts it's tasks running within it. Each task inside that call with be asynchronous with respect to each other (think about passing generators to zip() ) but the loop will block the rest of your code until they're all finished. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: 4/30/2015 2:07 To: Greg Ewing<mailto:greg.ewing@canterbury.ac.nz> Cc: Python Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] PEP 492 quibble and request On 30 April 2015 at 09:58, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Well, if I don't know, I get confused as to where I invoke the event loop, how my non-async code runs alongside this etc. Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
participants (12)
-
Antoine Pitrou
-
Arnaud Delobelle
-
Ethan Furman
-
Greg Ewing
-
Guido van Rossum
-
Nick Coghlan
-
Paul Moore
-
Paul Sokolovsky
-
Ron Adam
-
Steve Dower
-
Steven D'Aprano
-
Yury Selivanov