PEP 492 vs. PEP 3152, new round

I've tried to catch up with the previous threads. A summary of issues brought up: 1. precise syntax of `async def` (or do we need it at all) 2. do we need `async for` and `async with` (and how to spell them) 3. syntactic priority of `await` 4. `cocall` vs. `await` 5. do we really need `__aiter__` and friends 6. StopAsyncException 7. compatibility with asyncio and existing users of it (I've added a few myself.) I'll try to take them one by one. *1. precise syntax of `async def`* Of all the places to put `async` I still like *before* the `def` the best. I often do "imprecise search" for e.g. /def foo/ and would be unhappy if this didn't find async defs. Putting it towards the end (`def foo async()` or `def foo() async`) makes it easier to miss. A decorator makes it hard to make the syntactic distinctions required to reject `await` outside an async function. So I still prefer *`async def`*. *2. do we need `async for` and `async with`* Yes we do. Most of you are too young to remember, but once upon a time you couldn't loop over the lines of a file with a `for` loop like you do now. The amount of code that was devoted to efficiently iterate over files was tremendous. We're back in that stone age with the asyncio `StreamReader` class -- it supports `read()`, `readline()` and so on, but you can't use it with `for`, so you have to write a `while True` loop. `asyncio for` makes it possible to add a simple `__anext__` to the `StreamReader` class, as follows: ``` async def __anext__(self): line = await self.readline() if not line: raise StopAsyncIteration return line ``` A similar argument can be made for `async with`; the transaction commit is pretty convincing, but it also helps to be able to wait e.g. for a transport to drain upon closing a write stream. As for how to spell these, I think having `async` at the front makes it most clear that this is a special form. (Though maybe we should consider `await for` and `await with`? That would have the advantage of making it easy to scan for all suspension points by searching for /await/. But being a verb it doesn't read very well.) *3. syntactic priority of `await`* Yury, could you tweak the syntax for `await` so that we can write the most common usages without parentheses? In particular I'd like to be able to write ``` return await foo() with await foo() as bar: ... foo(await bar(), await bletch()) ``` (I don't care about `await foo() + await bar()` but it would be okay.) ``` I think this is reasonable with some tweaks of the grammar (similar to what Greg did for cocall, but without requiring call syntax at the end). *4. `cocall` vs. `await`* Python evolves. We couldn't have PEP 380 (`yield from`) without prior experience with using generators as coroutines (PEP 342), which in turn required basic generators (PEP 255), and those were a natural evolution of Python's earlier `for` loop. We couldn't PEP 3156 (asyncio) without PEP 380 and all that came before. The asyncio library is getting plenty of adoption and it has the concept of separating the *getting* of a future[1] from *waiting* for it. IIUC this is also how `await` works in C# (it just requires something with an async type). This has enabled a variety of operations that take futures and produce more futures. [1] I write `future` with a lowercase 'f' to include concepts like coroutine generator objects. *I just can't get used to this aspect of PEP 3152, so I'm rejecting it.* Sorry Greg, but that's the end. We must see `await` as a refinement of `yield from`, not as an alternative. (Yury: PEP 492 is not accepted yet, but you're getting closer.) One more thing: this separation is "Pythonic" in the sense that it's similar to the way *getting* a callable object is a separate act from *calling* it. While this is a cause for newbie bugs (forgetting to call an argument-less function) it has also enabled the concept of "callable" as more general and more powerful in Python: any time you need to pass a callable, you can pass e.g. a bound method or a class or something you got from `functools.partial`, and that's a useful thing (other languages require you to introduce something like a lambda in such cases, which can be painful if the thing you wrap has a complex signature -- or they don't support function parameters at all, like Java). I know that Greg defends it by explaining that `cocal f(args)` is not a `cocall` operator applied to `f(args)`, it is the *single* operator `cocall ...(args)` applied to `f`. But this is too subtle, and it just doesn't jive with the long tradition of using `yield from f` where f is some previously obtained future. *5. do we really need `__aiter__` and friends* There's a lot of added complexity, but I think it's worth it. I don't think we need to make the names longer, the 'a' prefix is fine for these methods. I think it's all in the protocols: regular `with` uses `__enter__` and `__exit__`; `async with` uses `__aenter__` and `__aexit__` (which must return futures). Ditto for `__aiter__` and `__anext__`. I guess this means that the async equivalent to obtaining an iterator through `it = iter(xs)` followed by `for x over it` will have to look like `ait = await aiter(xs)` followed by `for x over ait`, where an iterator is required to have an `__aiter__` method that's an async function and returns self immediately. But what if you left out the `await` from the first call? I.e. can this work? ``` ait = aiter(xs) async for x in ait: print(x) ``` The question here is whether the object returned by aiter(xs) has an `__aiter__` method. Since it was intended to be the target of `await`, it has an `__await__` method. But that itself is mostly an alias for `__iter__`, not `__aiter__`. I guess it can be made to work, the object just has to implement a bunch of different protocols. *6. StopAsyncException* I'm not sure about this. The motivation given in the PEP seems to focus on the need for `__anext__` to be async. But is this really the right pattern? What if we required `ait.__anext__()` to return a future, which can either raise good old `StopIteration` or return the next value from the iteration when awaited? I'm wondering if there are a few alternatives to be explored around the async iterator protocol still. *7. compatibility with asyncio and existing users of it* This is just something I want to stress. On the one hand it should be really simple to take code written for pre-3.5 asyncio and rewrite it to PEP 492 -- simple change `@asyncio.coroutine` to `async def` and change `yield from` to `await`. (Everything else is optional; existing patterns for loops and context managers should continue to work unchanged, even if in some cases you may be able to simplify the code by using `async for` and `async with`.) But it's also important that *even if you don't switch* (i.e. if you want to keep your code compatible with pre-3.5 asyncio) you can still use the PEP 492 version of asyncio -- i.e. the asyncio library that comes with 3.5 must seamlessly support mixing code that uses `await` and code that uses `yield from`. And this should go both ways -- if you have some code that uses PEP 492 and some code that uses pre-3.5 asyncio, they should be able to pass their coroutines to each other and wait for each other's coroutines. That's all I have for now. Enjoy! -- --Guido van Rossum (python.org/~guido)

Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote:
I don't remember the reason why yield requires parentheses in expressions, hopefully it's not something fundamental. This has always annoyed me, so let's try to fix that for await. I'll experiment.
With the current semantics that PEP 492 proposes, "await" for "aiter()" is mandatory. You have to write ait = await aiter(xs) async for x in ait: print(c) We can add some logic that will check that the iterator passed to 'async for' is not an unresolved awaitable and resolve it (instead of immediately checking if it has __anext__ method), but that will complicate the implementation. It will also introduce more than one way of doing things. I think that users will recognize "async builtins" (when we add them) by the first letter "a" and use them in "await" expressions consistently.
Correct. And yes, we address this all by having iteration protocols clearly separated.
__anext__ should return an awaitable (following the terminology of the PEP), which can be a coroutine-object. I'm not sure that with semantics of PEP 479 it can actually raise StopIteration (without some hacks in genobject). I'm also trying to think forward about how we can add generator-coroutines (the ones that combine 'await' and some form of 'yield') to make writing asynchronous iterators easier. I think that reusing StopIteration on that level will be a very hard thing to understand and implement. I'll experiment with reference implementation and update the PEP. Thank you, Yury

Hi, 2015-04-24 19:03 GMT+02:00 Guido van Rossum <guido@python.org>:
1. precise syntax of `async def`
Of all the places to put `async` I still like *before* the `def` the best.
So do I.
2. do we need `async for` and `async with`
Yes we do.
I agree.
IMO another point must be discussed, corner cases in the grammar and parser of the current PEP & implementation: https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings And the fact the Python 3.7 will make async & await keywords without proving a way to prepare the code for this major change in the Python syntax. According to the PEP, patching the parser to detect async & await as keywords is almost a hack, and there are corner cases where it doesn't work "as expected". That's why I suggest to reconsider the idea of supporting an *optional* "from __future__ import async" to get async and await as keywords in the current file. This import would allow all crazy syntax. The parser might suggest to use the import when it fails to parse an async or await keyword :-) If you don't like the compromise of a parser with corner cases and an optional __future__ to "workaround these cases", I'm also ok to reproduce what we did with the introduction of the with keyword. I mean not supporting async nor await by default, and maing __future__ mandatory to get the new feature. => 0 risk of backward compatibility issue => no more crazy hacks in the parser
My rationale in my other email was similar (ability to get a function without calling it, as you wrote, like bounded methods), so obviously I agree with it :-) I accept the compromise of creating a coroutine object without wait for it (obvious and common bug when learning asyncio). Hopefully, we keep the coroutine wrapper feature (ok, maybe I suggested this idea to Yury because I suffered so much when I learnt how to use asyncio ;-)), so it will still be easy to emit a warning in debug mode. On Stackoverflow, when someone posts a code with bug, I'm now replying "please rerun your code with asyncio debug mode enabled" ;-) I also mentionned it at the *beginning* of the asyncio doc (it's documented at the end of the asyncio doc!).
5. do we really need `__aiter__` and friends
There's a lot of added complexity, but I think it's worth it.
I agree. I don't see how to keep the PEP consistent without having new dedicated protocols.
6. StopAsyncException
(Sorry, I have no opinion on this point.)
7. compatibility with asyncio and existing users of it
The current state of the PEP makes types.coroutine() mandatory. If a generator-based coroutine is not modified with types.coroutine, await cannot be used on it. To be more concrete: asyncio coroutines not declared with @asyncio.coroutine cannot be used with await. Would it be crazy to allow waiting on a generator-based coroutine (current asyncio coroutines) without having to call types.coroutine() on it? Maybe I just missed the purpose of disallow this. It's also possible to modify asyncio to detect at runtime when an asyncio coroutine is not decorated by @asyncio.coroutine (emit a warning or even raise an exception). Victor

Victor, On 2015-04-24 5:32 PM, Victor Stinner wrote:
I'd be big -1 on that. The current PEP design is all about strictly prohibiting users from calling regular generators with 'await' expression. And if a generator isn't decorated with @coroutine - then it's a regular generator for us.
I'd be +1 to add a warning to Task and other places where we accept generator-based coroutines. Thanks! Yury

Victor Stinner wrote:
To me, these new features *obviously* should require a __future__ import. Anything else would be crazy.
I'm disappointed that there will *still* be no direct and reliable way to detect and clearly report this kind of error, and that what there is will only be active in a special debug mode. -- Greg

Hi Greg, 2015-04-25 7:02 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
It looks like our BDFL, Guido, made a choice. This point was discussed enough and most of us now agree on this point. There is no need to repeat yourself multiple time, we are well aware that you strongly disagree with this compromise ;-) It's probably the major difference between the PEP 492 and PEP 3152, and Guido decided to reject the PEP 3152. It's now time to focus our good energy on discussing remaining questions on the PEP 492 to make it the best PEP ever! Guido did a great great to summarize these questions. Thank you, Victor

Me too. Also, we would use a similar vocabulary to existing users of the feature. This is exactly how Hack does it: http://docs.hhvm.com/manual/en/hack.async.php <http://docs.hhvm.com/manual/en/hack.async.php>, how ECMAScript 7 proposes it: http://wiki.ecmascript.org/doku.php?id=strawman:async_functions <http://wiki.ecmascript.org/doku.php?id=strawman:async_functions> and similarly to how C# does it (async comes after the public/private modifier but before the return type): https://msdn.microsoft.com/en-us/library/hh156513.aspx <https://msdn.microsoft.com/en-us/library/hh156513.aspx>
2. do we need `async for` and `async with`
Yes we do.
+1
(Though maybe we should consider `await for` and `await with`? That would have the advantage of making it easy to scan for all suspension points by searching for /await/. But being a verb it doesn't read very well.)
I’m on the fence here. OT1H, I think “await for something in a_container” and “await with a_context_manager():” also read pretty well. It’s also more consistent with being the one way of finding “yield points”. OTOH, “await with a_context_manager():” suggests the entire statement is awaited on, which is not true. “async with” is more opaque in this way, it simply states “there are going to be implementation-specific awaits inside”. So it’s more consistent with “async def foo()”. All in all I think I’m leaning towards “async for” and “async with”. More importantly though, I’m wondering how obvious will the failure mode be when somebody uses a bare “for” instead of an “async for”. Ditto for “with” vs. “async with”. How much debugging will be necessary to find that it’s only a missing “async” before the loop? Side note: to add to the confusion about syntax, Hack’s equivalent for “async for” - which doesn’t really translate well to Python - uses “await” and ties it to the iterable: foreach ($list await as $input) { … } The equivalent in Python would be: for input in list await: Notably, this is ugly and would be confused with `for input in await list:` which means something different. Also, this particular construct represents less than 0.01% of all “await” occurences in Facebook code, suggesting it’s not performance critical.
3. syntactic priority of `await`
Yury, could you tweak the syntax for `await` so that we can write the most common usages without parentheses?
+1 Yury points out there was likely a reason this wasn’t the case for `yield` in the first place. It would be good to revisit that. Maybe for yield itself, too?
4. `cocall` vs. `await`
I just can't get used to this aspect of PEP 3152, so I'm rejecting it.
+1
(Yury: PEP 492 is not accepted yet, but you're getting closer.)
May I suggest using the bat-signal to summon Glyph to confirm this is going to be helpful/usable with Twisted as well?
5. do we really need `__aiter__` and friends
There's a lot of added complexity, but I think it's worth it. I don't think we need to make the names longer, the 'a' prefix is fine for these methods.
+1
6. StopAsyncException
I'm not sure about this. The motivation given in the PEP seems to focus on the need for `__anext__` to be async. But is this really the right pattern? What if we required `ait.__anext__()` to return a future, which can either raise good old `StopIteration` or return the next value from the iteration when awaited? I'm wondering if there are a few alternatives to be explored around the async iterator protocol still.
So are you suggesting to pass the returned value in a future? In this case the future would need to be passed to __anext__, so the Cursor example from the PEP would look like this: class Cursor: def __init__(self): self.buffer = collections.deque() def _prefetch(self): ... async def __aiter__(self): return self async def __anext__(self, fut): if not self.buffer: self.buffer = await self._prefetch() if self.buffer: fut.set_result(self.buffer.popleft()) else: fut.set_exception(StopIteration) While this is elegant, my concern is that one-future-per-iteration-step might be bad for performance. Maybe consider the following. The `async def` syntax decouples the concept of a coroutine from the implementation. While it’s still based on generators under the hood, the user no longer considers his “async function” to be a generator or conforming to the generator protocol. From the user’s perpective, it’s obvious that the return below means something different than the exception: async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopIteration return self.buffer.popleft() So, the same way we added wrapping in RuntimeErrors for generators in PEP 479, we might add transparent wrapping in a _StopAsyncIteration for CO_COROUTINE.
7. compatibility with asyncio and existing users of it
+1, this is really important. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

Lukasz, On 2015-04-24 5:37 PM, Łukasz Langa wrote:
This. I also think of 'await with' as I'm awaiting on the whole statement. And I don't even know how to interpret that.
FWIW I have to experiment more with the reference implementation, but at the moment I'm big -1 on touching StopIteration for coroutines. It's used for too many things. Thanks! Yury

On 25 April 2015 at 07:37, Łukasz Langa <lukasz@langa.pl> wrote:
yield requires parentheses in most cases for readability purposes. For example, does this pass one argument or two to "f"?: f(yield a, b) Answer: neither, it's a syntax error, as you have to supply the parentheses to say whether you mean "f((yield a, b))" or "f((yield a), b)". Requiring parentheses except in a few pre-approved cases (specifically, as a standalone statement and as the RHS of assignment statements) eliminated all that potential ambiguity. Allowing "return yield a, b" and "return yield from a, b" by analogy with assignment statements would be fine, but getting more permissive than that with yield expressions doesn't make sense due to the ambiguity between yielding a tuple and other uses of commas as separators in various parts of the grammar. PEP 492's "await" is a different case, as asynchronously waiting for a tuple doesn't make any sense. However, I'm not sure the grammar will let you reasonably express "anything except a tuple literal" as the subexpression after the await keyword, so it will likely still need special case handling in the affected statements in order to avoid "await a, b" being interpreted as waiting for a tuple. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On top of my previous response, one more thing to consider is that this idea brings a builtin Future back to the proposal, which has already been rejected in the "No implicit wrapping in Futures” section of the PEP. PEP 492 manages to solve all issues without introducing a built-in Future. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

Guido van Rossum wrote:
But how is an awaitable supposed to raise StopIteration if it's implemented by a generator or async def[*] function? Those things use StopIteration to wrap return values. I like the idea of allowing StopIteration to be raised in an async def function and wrapping it somehow. I'd add that it could also be unwrapped automatically when it emerges from 'await', so that code manually invoking __anext__ can catch StopIteration as usual. I don't think this could conflict with any existing uses of StopIteration, since raising it inside generators is currently forbidden. [*] I'm still struggling with what to call those things. Calling them just "coroutines" seems far too ambiguous. (There should be a Zen item something along the lines of "If you can't think of a concise and unambiguous name for it, it's probably a bad idea".) -- Greg

2015-04-25 8:23 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Hum, wrap and then unwrap is not cheap. Exception handling in CPython is known to be slow (it's one of the favorite topic for microbenchmarks ;-)). I would prefer to not make wrapping/unwrapping a feature and instead use different exceptions. Again, I don't understand the problem, so I'm just sure that my remark makes sense. But please be careful of performances. I already saw microbenchmarks on function call vs consuming a generator to justify that asyncio is way too slow and must not be used... See for example the "2. AsyncIO uses appealing, but relatively inefficient Python paradigms" section of the following article: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ The microbenchmark shows that yield from is 6x as slow as function calls (and function calls are also known to be slow in CPython). (I don't think that it makes sense to compare function calls versus consuming a generator to benchmark asyncio, but other developers have a different opinion of that.) Victor

Guido van Rossum wrote:
Making 'await' a prefix operator with the same precedence as unary minus would allow most reasonable usages, I think. The only reason "yield from" has such a constrained syntax is that it starts with "yield", which is similarly constrained. Since 'await' is a brand new keyword isn't bound by those constraints. -- Greg

Hi Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote:
I've done some experiments with grammar, and it looks like we indeed can parse await quite differently from yield. Three different options: Option #1. Parse 'await' exactly like we parse 'yield'. This how we parse 'await' in the latest version of reference implementation. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-1-patch Repo to play with: https://github.com/1st1/cpython/tree/await Syntax: await a() res = (await a()) + (await b()) res = (await (await a())) if (await a()): pass return (await a()) print((await a())) func(arg=(await a())) await a() * b() Option #2. Keep 'await_expr' in 'atom' terminal, but don't require parentheses. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-2-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens Syntax: await a() res = (await a()) + await b() # w/o parens (a() + await b()) res = await await a() if await a(): pass return await a() print(await a()) func(arg=await a()) await a() * b() Option #3. Create a new terminal for await expression between 'atom' and 'power'. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-3-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens2 Syntax: await a() res = await a() + await b() res = await (await a()) # invalid syntax w/o parens if await a(): pass return await a() print(await a()) func(arg=await a()) await (a() * b()) # w/o parens '(await a() * b()) I think that Option #3 is a clear winner. Thanks, Yury

On 26 April 2015 at 06:18, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Very nice! How would the following statements parse with this option? res = await a(), b() res = [await a(), b()] with await a(), b: pass f(await a(), b) I think it's OK if these end up requiring parentheses in order to do the right thing (as that will be helpful for humans regardless), but the PEP should be clear as to whether or not they do: res = (await a()), b() res = [(await a()), b()] with (await a()), b: pass f((await a()), b) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yury Selivanov wrote:
You don't seem to have tried what I suggested, which is to make 'await' a unary operator with the same precedence as '-', i.e. replace factor: ('+'|'-'|'~') factor | power with factor: ('+'|'-'|'~'|'await') factor | power That would allow await a() res = await a() + await b() res = await await a() if await a(): pass return await a() print(await a()) func(arg=await a()) await a() * b() -- Greg

Looking at the grammar -- the only downside of the current approach is that you can't do 'await await fut'. I still think that it reads better with parens. If we put 'await' to 'factor' terminal we would allow await -fut # await (-fut) I think I something like power: atom_expr ['**' factor] atom_expr: [AWAIT] atom_expr | atom_trailer atom_trailer: atom trailer* will fix 'await await' situation, but does it really need to be fixed? Yury On 2015-04-27 9:44 AM, Yury Selivanov wrote:

On 04/29, Yury Selivanov wrote:
On 2015-04-29 5:12 AM, Greg Ewing wrote:
Yury Selivanov wrote:
That's like saying we should always put parens around the number being raised in n ** x because -2**4 != (-2)**4 Please do not overuse parens. Python is not lisp, and await is not a function, so parens should not be needed in the common case. -- ~Ethan~

Ethan, On 2015-04-29 11:29 AM, Ethan Furman wrote:
Python is not lisp, and await is not a function, so parens should not be needed in the common case.
Which common case you mean? Please see this table https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression Yury

Also please see this: https://hg.python.org/peps/rev/d048458307b7 FWIW, 'await await fut' isn't something that you likely to see in your life; 'await -fut' is 99.999% just a bug. I'm not sure why Greg is pushing his Grammar idea so aggressively. Yury On 2015-04-29 11:33 AM, Yury Selivanov wrote:

— Sent from Mailbox On Wednesday Apr 29, 2015 at 7:14 PM, Yury Selivanov <yselivanov.ml@gmail.com>, wrote:Also please see this: https://hg.python.org/peps/rev/d048458307b7 FWIW, 'await await fut' isn't something that you likely to see in your life; 'await -fut' is 99.999% just a bug. Agree. I'm not sure why Greg is pushing his Grammar idea so aggressively. Yury On 2015-04-29 11:33 AM, Yury Selivanov wrote:
Ethan,
On 2015-04-29 11:29 AM, Ethan Furman wrote:
Python is not lisp, and await is not a
function, so parens should not be needed in the common case.
Which common case you mean?
Please see this table
https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression
Yury
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.co...

Yury Selivanov wrote:
I'm not sure why Greg is pushing his Grammar idea so aggressively.
Because I believe that any extra complexity in the grammar needs a very strong justification. It's complexity in the core language, like a new keyword, so it puts a burden on everyone's brain. Saying "I don't think anyone would ever need to write this, therefore we should disallow it" is not enough, given that there is a substantial cost to disallowing it. If you don't think there's a cost, consider that we *both* seem to be having trouble predicting the consequences of your proposed syntax, and you're the one who invented it. That's not a good sign! -- Greg

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I have a slightly different view. A bunch of things *must* work, e.g. f(await g(), await h()) or with await f(): (there's a longer list in the PEP). Other things may be ambiguous to most readers, e.g. what does await f() + g() mean, or can we say await await f(), and the solution is to recommend adding parentheses that make things clear to the parser *and* humans. Yury's proposal satisfies my requirements, and if we really find some unpleasant edge case we can fix it during the 3.5 release (the PEP will be provisional). -- --Guido van Rossum (python.org/~guido)

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
FWIW, now that I've seen the precedence table in the updated PEP, it seems really natural to me: https://www.python.org/dev/peps/pep-0492/#updated-operator-precedence-table According to that, "await" is just a prefix operator that binds more tightly than any arithmetic operation, but less tightly than indexing/funcall/attribute lookup, which seems about right. However, if what I just wrote were true, then that would mean that "await -foo" and "await await foo" would be syntactically legal (though useless). The fact that they apparently are *not* legal means that in fact there is still some weird thing going on in the syntax that I don't understand. And the PEP gives no further details, it just suggests I go read the parser generator source. My preference would be that the PEP be updated so that my one-sentence summary above became correct. But like Guido, I don't necessarily care about the exact details all that much. What I do feel strongly about is that whatever syntax we end up with, there should be *some* accurate human-readable description of *what it is*. AFAICT the PEP currently doesn't have that. -n -- Nathaniel J. Smith -- http://vorpus.org

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
In a PEP you should probably give grammar that is not the actual grammar from the implementation, but matches the grammar used in the reference manual on docs.python.org. -- --Guido van Rossum (python.org/~guido)

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hah, good question :-). Of course we all learned how to parse arithmetic in school, so perhaps it's a bit cheating to refer to that knowledge. Except of course basically all our users *do* have that knowledge (or else are forced to figure it out anyway). So I would be happy with a description of "await" that just says "it's like unary minus but higher precedence". Even if we put aside our trained intuitions about arithmetic, I think it's correct to say that the way unary minus is parsed is: everything to the right of it that has a tighter precedence gets collected up and parsed as an expression, and then it takes that expression as its argument. Still pretty simple. -- Nathaniel J. Smith -- http://vorpus.org

On Wed, Apr 29, 2015 at 5:05 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
So you're saying that "await -fut" and "await await fut" are actually legal syntax after all, contra what the PEP says? Because "- -fut" is totally legal syntax, so if await and unary minus work the same... (Again I don't care about those examples in their own right, I just find it frustrating that I can't answer these questions without asking you each time.) -n -- Nathaniel J. Smith -- http://vorpus.org

Nathaniel Smith wrote:
Tighter or equal, actually: '--a' is allowed. This explains why Yury's syntax disallows 'await -f'. The 'await' operator requires something after it, but there's *nothing* between it and the following '-', which binds less tightly. So it's understandable, but you have to think a bit harder. Why do we have to think harder? I suspect it's because the notion of precedence is normally introduced to resolve ambiguities. Knowing that infix '*' has higher precedence than infix '+' tells us that 'a + b * c' is parsed as 'a + (b * c)' and not '(a + b) * c'. Similarly, knowing that infix '.' has higher precedence than prefix '-' tells us that '-a.b' is parsed as '-(a.b)' rather than '(-a).b'. However, giving prefix 'await' higher precedence than prefix '-' doesn't serve to resolve any ambiguity. '- await f' is parsed as '-(await f)' either way, and 'await f + g' is parsed as '(await f) + g' either way. So when we see 'await -f', we think we already know what it means. There is only one possible order for the operations, so it doesn't look as though precedence comes into it at all, and we don't consider it when judging whether it's a valid expression. What's the conclusion from all this? I think it's that using precedence purely to disallow certain constructs, rather than to resolve ambiguities, leads to a grammar with less-than-intuitive characteristics. -- Greg

On Apr 30, 2015 1:57 AM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
The other reason this threw me is that I've recently been spending time with a shunting yard parser, and in shunting yard parsers unary prefix operators just work in the expected way (their precedence only affects their interaction with later binary operators; a chain of unaries is always allowed). It's just a limitation of the parser generator tech that python uses that it can't handle unary operators in the natural fashion. (OTOH it can handle lots of cases that shunting yard parsers can't -- I'm not criticizing python's choice of parser.) Once I read the new "documentation grammar" this became much clearer.
The actual effect of making "await" a different precedence is to resolve the ambiguity in await x ** 2 If await acted like -, then this would be await (x ** 2) But with the proposed grammar, it's instead (await x) ** 2 Which is probably correct, and produces the IMHO rather nice invariant that "await" binds more tightly than arithmetic in general (instead of having to say that it binds more tightly than arithmetic *except* in this one corner case...). But then given the limitations of Python's parser plus the desire to disambiguate the expression above in the given way, it becomes an arguably regrettable, yet inevitable, consequence that await -fut await +fut await ~fut become parse errors. AFAICT these and the ** case are the only expressions where there's any difference between Yury's proposed grammar and your proposal of treating await like unary minus. -n

On Apr 30, 2015 8:40 PM, "Guido van Rossum" <guido@python.org> wrote:
On Thu, Apr 30, 2015 at 8:30 PM, Nathaniel Smith <njs@pobox.com> wrote:
The actual effect of making "await" a different precedence is to resolve
the ambiguity in that "await" binds more tightly than arithmetic in general (instead of having to say that it binds more tightly than arithmetic *except* in this one corner case...)
Correct.
AFAICT these and the ** case are the only expressions where there's any
difference between Yury's proposed grammar and your proposal of treating await like unary minus. But then given the limitations of Python's parser plus the desire to disambiguate the expression above in the given way, it becomes an arguably regrettable, yet inevitable, consequence that
I didn't say it was regrettable, I said it was arguably regrettable. For proof, see the last week of python-dev ;-). (I guess all else being equal it would be nice if unary operators could stack arbitrarily, since that really is the more natural parse rule IMO and also if things had worked that way then I would have spent this thread less confused. But this is a pure argument from elegance. In practice there's obviously no good reason to write "await -fut" or "-not x", so meh, whatever.) -n

On 04/29, Yury Selivanov wrote:
The common case of not wanting to write parenthesis. ;) Seriously, why are the parens necessary? await (-coro()) from the many examples above that just work, it's a mystery why await -coro() cannot also just work and be the same as the parenthesized version. -- ~Ethan~

On 04/29, Yury Selivanov wrote:
I obviously don't understand your position that "it does not make any sense" -- perhaps you could explain a bit? What I see is a suspension point that is waiting for the results of coro(), which will be negated (and returned/assigned/whatever). What part of that doesn't make sense? -- ~Ethan~

On 29 April 2015 at 19:32, Ethan Furman <ethan@stoneleaf.us> wrote:
Would that not be "-await coro()"? What "await -coro()" would mean is to call coro() (which would return something you can wait on), then apply - to that (then waiting on the result of that negation). But what does negating an awaitable object mean? Obviously you can *define* it to mean something, but you probably didn't. Paul

Hi Ethan, On 2015-04-29 2:32 PM, Ethan Furman wrote:
Because you want operators to be resolved in the order you see them, generally. You want '(await -fut)' to: 1. Suspend on fut; 2. Get the result; 3. Negate it. This is a non-obvious thing. I would myself interpret it as: 1. Get fut.__neg__(); 2. await on it. So I want to make this syntactically incorrect: 'await -fut' would throw a SyntaxError. To do what you want, write a pythonic '- await fut'. Yury

On Apr 29, 2015 11:49 AM, "Yury Selivanov" <yselivanov.ml@gmail.com> wrote:
As a bystander, I don't really care either way about whether await -fut is syntactically valid (since like you say it's semantically nonsense regardless and no one will ever write it). But I would rather like to actually know what the syntax actually is, not just have a list of examples (which kinda gives me perl flashbacks). Is there any simple way to state what the rules for parsing await are? Or do I just have to read the parser code if I want to know that? (I suspect this may also be the impetus behind Greg's request that it just be treated the same as unary minus. IMHO it matters much more that the rules be predictable and teachable than that they allow or disallow every weird edge case in exactly the right way.) -n

Hi Nathaniel, BTW, I'm going to reply to you in the other thread about context-local objects soon. I have some thoughts on the topic. On 2015-04-29 3:14 PM, Nathaniel Smith wrote:
There is a summary of grammar changes in the PEP: https://www.python.org/dev/peps/pep-0492/#grammar-updates You can also see the actual grammar file from the reference implementation: https://github.com/1st1/cpython/blob/await/Grammar/Grammar Thanks, Yury

On 04/29, Yury Selivanov wrote:
Both you and Paul are correct on this, thank you. The proper resolution of await -coro() is indeed to get the result of coro(), call it's __neg__ method, and then await on that. And that is perfectly reasonable, and should not be a SyntaxError; what it might be is an AttributeError (no __neg__ method) or an AsyncError (__neg__ returned non-awaitable object), or might even just work [1]... but it definitely should /not/ be a SyntaxError. -- ~Ethan~ [1] http://stackoverflow.com/q/7719018/208880

On Thu, Apr 30, 2015 at 9:15 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Why not? Unlike some other languages, Python does not have uniform priorities for unary operators, so it's reasonable for some unary operations to have a different priority than others, and certain things will be SyntaxErrors because of that. E.g. you can write "not -x" but you can't write "- not x". This is because they have different priorities: 'not' has a very low priority so it binds less tight than comparisons (which bind less tight than arithmetic), but '-' has a high priority. (There are other quirks, e.g. -2**2 means -(2**2) and you can write 2**-2; but you can't write 2**not x.) -- --Guido van Rossum (python.org/~guido)

On 04/30, Guido van Rossum wrote:
For one, Yury's answer is "- await x" which looks just as nonsensical as "- not x". For another, an error of some type will be raised if either __neg__ doesn't exist or it doesn't return an awaitable, so a SyntaxError is unnecessary. For a third, by making it a SyntaxError you are forcing the use of parens to get what should be the behavior anyway. In other words, a SyntaxError is nat any clearer than "AttributeError: obj has no __neg__ method" and it's not any clearer than "AwaitError: __neg__ returned not-awaitable". Those last two errors tell you exactly what you did wrong. -- ~Ethan~

On 2015-04-30 1:56 PM, Ethan Furman wrote:
"- await x" is a perfectly valid code: result = - await compute_in_db() (same as "result = - (yield from do_something())")
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere. FWIW, I'll fix the 'await (await x)' expression to be parsed without parens.
This is debatable. "obj has no __neg__ method" isn't obvious to everyone (especially to those people who aren't using operator overloading). [1] https://www.python.org/dev/peps/pep-0492/#examples-of-await-expressions Yury

On 04/30, Yury Selivanov wrote:
On 2015-04-30 1:56 PM, Ethan Furman wrote:
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere.
--> await -coro() SyntaxError --> await (-coro()) # not a SyntaxError, therefore parens are # forced
Good news! The error there is actually --> -object() TypeError: bad operand type for unary -: 'object' Which is definitely clear, even for those who don't do operator overloading. -- ~Ethan~

On 1/05/2015 6:04 a.m., Yury Selivanov wrote:
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere.
It's not about requiring or not requiring parens. It's about making the simplest possible change to the grammar necessary to achieve the desired goals. Keeping the grammar simple makes it easy for humans to reason about. The question is whether syntactically disallowing certain constructs that are unlikely to be needed is a desirable enough goal to be worth complicating the grammar. You think it is, some others of us think it's not.
FWIW, I'll fix the 'await (await x)' expression to be parsed without parens.
I don't particularly care whether 'await -x' or 'await await x' can be written without parens or not. The point is that the simplest grammar change necessary to be able to write the things we *do* want also happens to allow those. I don't see that as a problem worth worrying about. -- Greg

On Thu, Apr 30, 2015 at 6:13 PM, Greg <greg.ewing@canterbury.ac.nz> wrote:
+1. It seems weird to add a whole new precedence level when an existing one works fine. Accidentally negating a future/deferred is not a significant source of errors, so I don't get why that would be a justifying example. -- Devin

On Thu, Apr 30, 2015 at 6:56 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
You can call me weird, but I *like* fine-tuning operator binding rules to suit my intuition for an operator. 'await' is not arithmetic, so I don't see why it should be lumped in with '-'. It's not like the proposed grammar change introducing 'await' is earth-shattering in complexity. -- --Guido van Rossum (python.org/~guido)

On 1/05/2015 5:38 a.m., Guido van Rossum wrote:
you can write "not -x" but you can't write "- not x".
That seems just as arbitrary and unintuitive, though. There are some other unintuitive consequences as well, e.g. you can write not a + b but it's not immediately obvious that this is parsed as 'not (a + b)' rather than '(not a) + b'. The presence of one arbitrary and unintuitive thing in the grammar is not by itself a justification for adding another one. -- Greg

Yury Selivanov wrote:
I don't want this: "await a() * b()" to be parsed, it's not meaningful.
Why not? If a() is a coroutine that returns a number, why shouldn't I be able to multiply it by something? I don't think your currently proposed grammar prevents that anyway. We can have <expr> --> <factor> '*' <factor> --> <power> '*' <power> --> <atom_expr> '*' <atom_expr> --> 'await' <atom> <trailer>* '*' <atom> <trailer>* --> 'await' 'a' '(' ')' '*' 'b' '(' ')' It does, on the other hand, seem to prevent x = - await a() which looks perfectly sensible to me. I don't like the idea of introducing another level of precedence. Python already has too many of those to keep in my brain. Being able to tell people "it's just like unary minus" makes it easy to explain (and therefore possibly a good idea!). -- Greg

Greg, On 2015-04-29 1:40 AM, Greg Ewing wrote:
Sorry, I thought you meant parsing "await a()*b()" as "await (a() * b())".
This works just fine: https://github.com/1st1/cpython/commit/33b3cd052243cd71c064eb385c7a557eec3ce... Current grammar prevents this: "await -fut", and this: "await fut ** 2" being parsed as "await (fut ** 2)
It's just like unary minus ;) Yury

Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote:
I don't remember the reason why yield requires parentheses in expressions, hopefully it's not something fundamental. This has always annoyed me, so let's try to fix that for await. I'll experiment.
With the current semantics that PEP 492 proposes, "await" for "aiter()" is mandatory. You have to write ait = await aiter(xs) async for x in ait: print(c) We can add some logic that will check that the iterator passed to 'async for' is not an unresolved awaitable and resolve it (instead of immediately checking if it has __anext__ method), but that will complicate the implementation. It will also introduce more than one way of doing things. I think that users will recognize "async builtins" (when we add them) by the first letter "a" and use them in "await" expressions consistently.
Correct. And yes, we address this all by having iteration protocols clearly separated.
__anext__ should return an awaitable (following the terminology of the PEP), which can be a coroutine-object. I'm not sure that with semantics of PEP 479 it can actually raise StopIteration (without some hacks in genobject). I'm also trying to think forward about how we can add generator-coroutines (the ones that combine 'await' and some form of 'yield') to make writing asynchronous iterators easier. I think that reusing StopIteration on that level will be a very hard thing to understand and implement. I'll experiment with reference implementation and update the PEP. Thank you, Yury

Hi, 2015-04-24 19:03 GMT+02:00 Guido van Rossum <guido@python.org>:
1. precise syntax of `async def`
Of all the places to put `async` I still like *before* the `def` the best.
So do I.
2. do we need `async for` and `async with`
Yes we do.
I agree.
IMO another point must be discussed, corner cases in the grammar and parser of the current PEP & implementation: https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings And the fact the Python 3.7 will make async & await keywords without proving a way to prepare the code for this major change in the Python syntax. According to the PEP, patching the parser to detect async & await as keywords is almost a hack, and there are corner cases where it doesn't work "as expected". That's why I suggest to reconsider the idea of supporting an *optional* "from __future__ import async" to get async and await as keywords in the current file. This import would allow all crazy syntax. The parser might suggest to use the import when it fails to parse an async or await keyword :-) If you don't like the compromise of a parser with corner cases and an optional __future__ to "workaround these cases", I'm also ok to reproduce what we did with the introduction of the with keyword. I mean not supporting async nor await by default, and maing __future__ mandatory to get the new feature. => 0 risk of backward compatibility issue => no more crazy hacks in the parser
My rationale in my other email was similar (ability to get a function without calling it, as you wrote, like bounded methods), so obviously I agree with it :-) I accept the compromise of creating a coroutine object without wait for it (obvious and common bug when learning asyncio). Hopefully, we keep the coroutine wrapper feature (ok, maybe I suggested this idea to Yury because I suffered so much when I learnt how to use asyncio ;-)), so it will still be easy to emit a warning in debug mode. On Stackoverflow, when someone posts a code with bug, I'm now replying "please rerun your code with asyncio debug mode enabled" ;-) I also mentionned it at the *beginning* of the asyncio doc (it's documented at the end of the asyncio doc!).
5. do we really need `__aiter__` and friends
There's a lot of added complexity, but I think it's worth it.
I agree. I don't see how to keep the PEP consistent without having new dedicated protocols.
6. StopAsyncException
(Sorry, I have no opinion on this point.)
7. compatibility with asyncio and existing users of it
The current state of the PEP makes types.coroutine() mandatory. If a generator-based coroutine is not modified with types.coroutine, await cannot be used on it. To be more concrete: asyncio coroutines not declared with @asyncio.coroutine cannot be used with await. Would it be crazy to allow waiting on a generator-based coroutine (current asyncio coroutines) without having to call types.coroutine() on it? Maybe I just missed the purpose of disallow this. It's also possible to modify asyncio to detect at runtime when an asyncio coroutine is not decorated by @asyncio.coroutine (emit a warning or even raise an exception). Victor

Victor, On 2015-04-24 5:32 PM, Victor Stinner wrote:
I'd be big -1 on that. The current PEP design is all about strictly prohibiting users from calling regular generators with 'await' expression. And if a generator isn't decorated with @coroutine - then it's a regular generator for us.
I'd be +1 to add a warning to Task and other places where we accept generator-based coroutines. Thanks! Yury

Victor Stinner wrote:
To me, these new features *obviously* should require a __future__ import. Anything else would be crazy.
I'm disappointed that there will *still* be no direct and reliable way to detect and clearly report this kind of error, and that what there is will only be active in a special debug mode. -- Greg

Hi Greg, 2015-04-25 7:02 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
It looks like our BDFL, Guido, made a choice. This point was discussed enough and most of us now agree on this point. There is no need to repeat yourself multiple time, we are well aware that you strongly disagree with this compromise ;-) It's probably the major difference between the PEP 492 and PEP 3152, and Guido decided to reject the PEP 3152. It's now time to focus our good energy on discussing remaining questions on the PEP 492 to make it the best PEP ever! Guido did a great great to summarize these questions. Thank you, Victor

Me too. Also, we would use a similar vocabulary to existing users of the feature. This is exactly how Hack does it: http://docs.hhvm.com/manual/en/hack.async.php <http://docs.hhvm.com/manual/en/hack.async.php>, how ECMAScript 7 proposes it: http://wiki.ecmascript.org/doku.php?id=strawman:async_functions <http://wiki.ecmascript.org/doku.php?id=strawman:async_functions> and similarly to how C# does it (async comes after the public/private modifier but before the return type): https://msdn.microsoft.com/en-us/library/hh156513.aspx <https://msdn.microsoft.com/en-us/library/hh156513.aspx>
2. do we need `async for` and `async with`
Yes we do.
+1
(Though maybe we should consider `await for` and `await with`? That would have the advantage of making it easy to scan for all suspension points by searching for /await/. But being a verb it doesn't read very well.)
I’m on the fence here. OT1H, I think “await for something in a_container” and “await with a_context_manager():” also read pretty well. It’s also more consistent with being the one way of finding “yield points”. OTOH, “await with a_context_manager():” suggests the entire statement is awaited on, which is not true. “async with” is more opaque in this way, it simply states “there are going to be implementation-specific awaits inside”. So it’s more consistent with “async def foo()”. All in all I think I’m leaning towards “async for” and “async with”. More importantly though, I’m wondering how obvious will the failure mode be when somebody uses a bare “for” instead of an “async for”. Ditto for “with” vs. “async with”. How much debugging will be necessary to find that it’s only a missing “async” before the loop? Side note: to add to the confusion about syntax, Hack’s equivalent for “async for” - which doesn’t really translate well to Python - uses “await” and ties it to the iterable: foreach ($list await as $input) { … } The equivalent in Python would be: for input in list await: Notably, this is ugly and would be confused with `for input in await list:` which means something different. Also, this particular construct represents less than 0.01% of all “await” occurences in Facebook code, suggesting it’s not performance critical.
3. syntactic priority of `await`
Yury, could you tweak the syntax for `await` so that we can write the most common usages without parentheses?
+1 Yury points out there was likely a reason this wasn’t the case for `yield` in the first place. It would be good to revisit that. Maybe for yield itself, too?
4. `cocall` vs. `await`
I just can't get used to this aspect of PEP 3152, so I'm rejecting it.
+1
(Yury: PEP 492 is not accepted yet, but you're getting closer.)
May I suggest using the bat-signal to summon Glyph to confirm this is going to be helpful/usable with Twisted as well?
5. do we really need `__aiter__` and friends
There's a lot of added complexity, but I think it's worth it. I don't think we need to make the names longer, the 'a' prefix is fine for these methods.
+1
6. StopAsyncException
I'm not sure about this. The motivation given in the PEP seems to focus on the need for `__anext__` to be async. But is this really the right pattern? What if we required `ait.__anext__()` to return a future, which can either raise good old `StopIteration` or return the next value from the iteration when awaited? I'm wondering if there are a few alternatives to be explored around the async iterator protocol still.
So are you suggesting to pass the returned value in a future? In this case the future would need to be passed to __anext__, so the Cursor example from the PEP would look like this: class Cursor: def __init__(self): self.buffer = collections.deque() def _prefetch(self): ... async def __aiter__(self): return self async def __anext__(self, fut): if not self.buffer: self.buffer = await self._prefetch() if self.buffer: fut.set_result(self.buffer.popleft()) else: fut.set_exception(StopIteration) While this is elegant, my concern is that one-future-per-iteration-step might be bad for performance. Maybe consider the following. The `async def` syntax decouples the concept of a coroutine from the implementation. While it’s still based on generators under the hood, the user no longer considers his “async function” to be a generator or conforming to the generator protocol. From the user’s perpective, it’s obvious that the return below means something different than the exception: async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopIteration return self.buffer.popleft() So, the same way we added wrapping in RuntimeErrors for generators in PEP 479, we might add transparent wrapping in a _StopAsyncIteration for CO_COROUTINE.
7. compatibility with asyncio and existing users of it
+1, this is really important. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

Lukasz, On 2015-04-24 5:37 PM, Łukasz Langa wrote:
This. I also think of 'await with' as I'm awaiting on the whole statement. And I don't even know how to interpret that.
FWIW I have to experiment more with the reference implementation, but at the moment I'm big -1 on touching StopIteration for coroutines. It's used for too many things. Thanks! Yury

On 25 April 2015 at 07:37, Łukasz Langa <lukasz@langa.pl> wrote:
yield requires parentheses in most cases for readability purposes. For example, does this pass one argument or two to "f"?: f(yield a, b) Answer: neither, it's a syntax error, as you have to supply the parentheses to say whether you mean "f((yield a, b))" or "f((yield a), b)". Requiring parentheses except in a few pre-approved cases (specifically, as a standalone statement and as the RHS of assignment statements) eliminated all that potential ambiguity. Allowing "return yield a, b" and "return yield from a, b" by analogy with assignment statements would be fine, but getting more permissive than that with yield expressions doesn't make sense due to the ambiguity between yielding a tuple and other uses of commas as separators in various parts of the grammar. PEP 492's "await" is a different case, as asynchronously waiting for a tuple doesn't make any sense. However, I'm not sure the grammar will let you reasonably express "anything except a tuple literal" as the subexpression after the await keyword, so it will likely still need special case handling in the affected statements in order to avoid "await a, b" being interpreted as waiting for a tuple. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On top of my previous response, one more thing to consider is that this idea brings a builtin Future back to the proposal, which has already been rejected in the "No implicit wrapping in Futures” section of the PEP. PEP 492 manages to solve all issues without introducing a built-in Future. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

Guido van Rossum wrote:
But how is an awaitable supposed to raise StopIteration if it's implemented by a generator or async def[*] function? Those things use StopIteration to wrap return values. I like the idea of allowing StopIteration to be raised in an async def function and wrapping it somehow. I'd add that it could also be unwrapped automatically when it emerges from 'await', so that code manually invoking __anext__ can catch StopIteration as usual. I don't think this could conflict with any existing uses of StopIteration, since raising it inside generators is currently forbidden. [*] I'm still struggling with what to call those things. Calling them just "coroutines" seems far too ambiguous. (There should be a Zen item something along the lines of "If you can't think of a concise and unambiguous name for it, it's probably a bad idea".) -- Greg

2015-04-25 8:23 GMT+02:00 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Hum, wrap and then unwrap is not cheap. Exception handling in CPython is known to be slow (it's one of the favorite topic for microbenchmarks ;-)). I would prefer to not make wrapping/unwrapping a feature and instead use different exceptions. Again, I don't understand the problem, so I'm just sure that my remark makes sense. But please be careful of performances. I already saw microbenchmarks on function call vs consuming a generator to justify that asyncio is way too slow and must not be used... See for example the "2. AsyncIO uses appealing, but relatively inefficient Python paradigms" section of the following article: http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/ The microbenchmark shows that yield from is 6x as slow as function calls (and function calls are also known to be slow in CPython). (I don't think that it makes sense to compare function calls versus consuming a generator to benchmark asyncio, but other developers have a different opinion of that.) Victor

Guido van Rossum wrote:
Making 'await' a prefix operator with the same precedence as unary minus would allow most reasonable usages, I think. The only reason "yield from" has such a constrained syntax is that it starts with "yield", which is similarly constrained. Since 'await' is a brand new keyword isn't bound by those constraints. -- Greg

Hi Guido, On 2015-04-24 1:03 PM, Guido van Rossum wrote:
I've done some experiments with grammar, and it looks like we indeed can parse await quite differently from yield. Three different options: Option #1. Parse 'await' exactly like we parse 'yield'. This how we parse 'await' in the latest version of reference implementation. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-1-patch Repo to play with: https://github.com/1st1/cpython/tree/await Syntax: await a() res = (await a()) + (await b()) res = (await (await a())) if (await a()): pass return (await a()) print((await a())) func(arg=(await a())) await a() * b() Option #2. Keep 'await_expr' in 'atom' terminal, but don't require parentheses. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-2-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens Syntax: await a() res = (await a()) + await b() # w/o parens (a() + await b()) res = await await a() if await a(): pass return await a() print(await a()) func(arg=await a()) await a() * b() Option #3. Create a new terminal for await expression between 'atom' and 'power'. Required grammar changes: https://gist.github.com/1st1/cb0bd257b04adb87e167#file-option-3-patch Repo to play with (parser module is broken atm): https://github.com/1st1/cpython/tree/await_noparens2 Syntax: await a() res = await a() + await b() res = await (await a()) # invalid syntax w/o parens if await a(): pass return await a() print(await a()) func(arg=await a()) await (a() * b()) # w/o parens '(await a() * b()) I think that Option #3 is a clear winner. Thanks, Yury

On 26 April 2015 at 06:18, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Very nice! How would the following statements parse with this option? res = await a(), b() res = [await a(), b()] with await a(), b: pass f(await a(), b) I think it's OK if these end up requiring parentheses in order to do the right thing (as that will be helpful for humans regardless), but the PEP should be clear as to whether or not they do: res = (await a()), b() res = [(await a()), b()] with (await a()), b: pass f((await a()), b) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Yury Selivanov wrote:
You don't seem to have tried what I suggested, which is to make 'await' a unary operator with the same precedence as '-', i.e. replace factor: ('+'|'-'|'~') factor | power with factor: ('+'|'-'|'~'|'await') factor | power That would allow await a() res = await a() + await b() res = await await a() if await a(): pass return await a() print(await a()) func(arg=await a()) await a() * b() -- Greg

Looking at the grammar -- the only downside of the current approach is that you can't do 'await await fut'. I still think that it reads better with parens. If we put 'await' to 'factor' terminal we would allow await -fut # await (-fut) I think I something like power: atom_expr ['**' factor] atom_expr: [AWAIT] atom_expr | atom_trailer atom_trailer: atom trailer* will fix 'await await' situation, but does it really need to be fixed? Yury On 2015-04-27 9:44 AM, Yury Selivanov wrote:

On 04/29, Yury Selivanov wrote:
On 2015-04-29 5:12 AM, Greg Ewing wrote:
Yury Selivanov wrote:
That's like saying we should always put parens around the number being raised in n ** x because -2**4 != (-2)**4 Please do not overuse parens. Python is not lisp, and await is not a function, so parens should not be needed in the common case. -- ~Ethan~

Ethan, On 2015-04-29 11:29 AM, Ethan Furman wrote:
Python is not lisp, and await is not a function, so parens should not be needed in the common case.
Which common case you mean? Please see this table https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression Yury

Also please see this: https://hg.python.org/peps/rev/d048458307b7 FWIW, 'await await fut' isn't something that you likely to see in your life; 'await -fut' is 99.999% just a bug. I'm not sure why Greg is pushing his Grammar idea so aggressively. Yury On 2015-04-29 11:33 AM, Yury Selivanov wrote:

— Sent from Mailbox On Wednesday Apr 29, 2015 at 7:14 PM, Yury Selivanov <yselivanov.ml@gmail.com>, wrote:Also please see this: https://hg.python.org/peps/rev/d048458307b7 FWIW, 'await await fut' isn't something that you likely to see in your life; 'await -fut' is 99.999% just a bug. Agree. I'm not sure why Greg is pushing his Grammar idea so aggressively. Yury On 2015-04-29 11:33 AM, Yury Selivanov wrote:
Ethan,
On 2015-04-29 11:29 AM, Ethan Furman wrote:
Python is not lisp, and await is not a
function, so parens should not be needed in the common case.
Which common case you mean?
Please see this table
https://www.python.org/dev/peps/pep-0492/#syntax-of-await-expression
Yury
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.co...

Yury Selivanov wrote:
I'm not sure why Greg is pushing his Grammar idea so aggressively.
Because I believe that any extra complexity in the grammar needs a very strong justification. It's complexity in the core language, like a new keyword, so it puts a burden on everyone's brain. Saying "I don't think anyone would ever need to write this, therefore we should disallow it" is not enough, given that there is a substantial cost to disallowing it. If you don't think there's a cost, consider that we *both* seem to be having trouble predicting the consequences of your proposed syntax, and you're the one who invented it. That's not a good sign! -- Greg

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I have a slightly different view. A bunch of things *must* work, e.g. f(await g(), await h()) or with await f(): (there's a longer list in the PEP). Other things may be ambiguous to most readers, e.g. what does await f() + g() mean, or can we say await await f(), and the solution is to recommend adding parentheses that make things clear to the parser *and* humans. Yury's proposal satisfies my requirements, and if we really find some unpleasant edge case we can fix it during the 3.5 release (the PEP will be provisional). -- --Guido van Rossum (python.org/~guido)

On Wed, Apr 29, 2015 at 3:46 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
FWIW, now that I've seen the precedence table in the updated PEP, it seems really natural to me: https://www.python.org/dev/peps/pep-0492/#updated-operator-precedence-table According to that, "await" is just a prefix operator that binds more tightly than any arithmetic operation, but less tightly than indexing/funcall/attribute lookup, which seems about right. However, if what I just wrote were true, then that would mean that "await -foo" and "await await foo" would be syntactically legal (though useless). The fact that they apparently are *not* legal means that in fact there is still some weird thing going on in the syntax that I don't understand. And the PEP gives no further details, it just suggests I go read the parser generator source. My preference would be that the PEP be updated so that my one-sentence summary above became correct. But like Guido, I don't necessarily care about the exact details all that much. What I do feel strongly about is that whatever syntax we end up with, there should be *some* accurate human-readable description of *what it is*. AFAICT the PEP currently doesn't have that. -n -- Nathaniel J. Smith -- http://vorpus.org

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
In a PEP you should probably give grammar that is not the actual grammar from the implementation, but matches the grammar used in the reference manual on docs.python.org. -- --Guido van Rossum (python.org/~guido)

On Wed, Apr 29, 2015 at 4:48 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hah, good question :-). Of course we all learned how to parse arithmetic in school, so perhaps it's a bit cheating to refer to that knowledge. Except of course basically all our users *do* have that knowledge (or else are forced to figure it out anyway). So I would be happy with a description of "await" that just says "it's like unary minus but higher precedence". Even if we put aside our trained intuitions about arithmetic, I think it's correct to say that the way unary minus is parsed is: everything to the right of it that has a tighter precedence gets collected up and parsed as an expression, and then it takes that expression as its argument. Still pretty simple. -- Nathaniel J. Smith -- http://vorpus.org

On Wed, Apr 29, 2015 at 5:05 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
So you're saying that "await -fut" and "await await fut" are actually legal syntax after all, contra what the PEP says? Because "- -fut" is totally legal syntax, so if await and unary minus work the same... (Again I don't care about those examples in their own right, I just find it frustrating that I can't answer these questions without asking you each time.) -n -- Nathaniel J. Smith -- http://vorpus.org

Nathaniel Smith wrote:
Tighter or equal, actually: '--a' is allowed. This explains why Yury's syntax disallows 'await -f'. The 'await' operator requires something after it, but there's *nothing* between it and the following '-', which binds less tightly. So it's understandable, but you have to think a bit harder. Why do we have to think harder? I suspect it's because the notion of precedence is normally introduced to resolve ambiguities. Knowing that infix '*' has higher precedence than infix '+' tells us that 'a + b * c' is parsed as 'a + (b * c)' and not '(a + b) * c'. Similarly, knowing that infix '.' has higher precedence than prefix '-' tells us that '-a.b' is parsed as '-(a.b)' rather than '(-a).b'. However, giving prefix 'await' higher precedence than prefix '-' doesn't serve to resolve any ambiguity. '- await f' is parsed as '-(await f)' either way, and 'await f + g' is parsed as '(await f) + g' either way. So when we see 'await -f', we think we already know what it means. There is only one possible order for the operations, so it doesn't look as though precedence comes into it at all, and we don't consider it when judging whether it's a valid expression. What's the conclusion from all this? I think it's that using precedence purely to disallow certain constructs, rather than to resolve ambiguities, leads to a grammar with less-than-intuitive characteristics. -- Greg

On Apr 30, 2015 1:57 AM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
The other reason this threw me is that I've recently been spending time with a shunting yard parser, and in shunting yard parsers unary prefix operators just work in the expected way (their precedence only affects their interaction with later binary operators; a chain of unaries is always allowed). It's just a limitation of the parser generator tech that python uses that it can't handle unary operators in the natural fashion. (OTOH it can handle lots of cases that shunting yard parsers can't -- I'm not criticizing python's choice of parser.) Once I read the new "documentation grammar" this became much clearer.
The actual effect of making "await" a different precedence is to resolve the ambiguity in await x ** 2 If await acted like -, then this would be await (x ** 2) But with the proposed grammar, it's instead (await x) ** 2 Which is probably correct, and produces the IMHO rather nice invariant that "await" binds more tightly than arithmetic in general (instead of having to say that it binds more tightly than arithmetic *except* in this one corner case...). But then given the limitations of Python's parser plus the desire to disambiguate the expression above in the given way, it becomes an arguably regrettable, yet inevitable, consequence that await -fut await +fut await ~fut become parse errors. AFAICT these and the ** case are the only expressions where there's any difference between Yury's proposed grammar and your proposal of treating await like unary minus. -n

On Apr 30, 2015 8:40 PM, "Guido van Rossum" <guido@python.org> wrote:
On Thu, Apr 30, 2015 at 8:30 PM, Nathaniel Smith <njs@pobox.com> wrote:
The actual effect of making "await" a different precedence is to resolve
the ambiguity in that "await" binds more tightly than arithmetic in general (instead of having to say that it binds more tightly than arithmetic *except* in this one corner case...)
Correct.
AFAICT these and the ** case are the only expressions where there's any
difference between Yury's proposed grammar and your proposal of treating await like unary minus. But then given the limitations of Python's parser plus the desire to disambiguate the expression above in the given way, it becomes an arguably regrettable, yet inevitable, consequence that
I didn't say it was regrettable, I said it was arguably regrettable. For proof, see the last week of python-dev ;-). (I guess all else being equal it would be nice if unary operators could stack arbitrarily, since that really is the more natural parse rule IMO and also if things had worked that way then I would have spent this thread less confused. But this is a pure argument from elegance. In practice there's obviously no good reason to write "await -fut" or "-not x", so meh, whatever.) -n

On 04/29, Yury Selivanov wrote:
The common case of not wanting to write parenthesis. ;) Seriously, why are the parens necessary? await (-coro()) from the many examples above that just work, it's a mystery why await -coro() cannot also just work and be the same as the parenthesized version. -- ~Ethan~

On 04/29, Yury Selivanov wrote:
I obviously don't understand your position that "it does not make any sense" -- perhaps you could explain a bit? What I see is a suspension point that is waiting for the results of coro(), which will be negated (and returned/assigned/whatever). What part of that doesn't make sense? -- ~Ethan~

On 29 April 2015 at 19:32, Ethan Furman <ethan@stoneleaf.us> wrote:
Would that not be "-await coro()"? What "await -coro()" would mean is to call coro() (which would return something you can wait on), then apply - to that (then waiting on the result of that negation). But what does negating an awaitable object mean? Obviously you can *define* it to mean something, but you probably didn't. Paul

Hi Ethan, On 2015-04-29 2:32 PM, Ethan Furman wrote:
Because you want operators to be resolved in the order you see them, generally. You want '(await -fut)' to: 1. Suspend on fut; 2. Get the result; 3. Negate it. This is a non-obvious thing. I would myself interpret it as: 1. Get fut.__neg__(); 2. await on it. So I want to make this syntactically incorrect: 'await -fut' would throw a SyntaxError. To do what you want, write a pythonic '- await fut'. Yury

On Apr 29, 2015 11:49 AM, "Yury Selivanov" <yselivanov.ml@gmail.com> wrote:
As a bystander, I don't really care either way about whether await -fut is syntactically valid (since like you say it's semantically nonsense regardless and no one will ever write it). But I would rather like to actually know what the syntax actually is, not just have a list of examples (which kinda gives me perl flashbacks). Is there any simple way to state what the rules for parsing await are? Or do I just have to read the parser code if I want to know that? (I suspect this may also be the impetus behind Greg's request that it just be treated the same as unary minus. IMHO it matters much more that the rules be predictable and teachable than that they allow or disallow every weird edge case in exactly the right way.) -n

Hi Nathaniel, BTW, I'm going to reply to you in the other thread about context-local objects soon. I have some thoughts on the topic. On 2015-04-29 3:14 PM, Nathaniel Smith wrote:
There is a summary of grammar changes in the PEP: https://www.python.org/dev/peps/pep-0492/#grammar-updates You can also see the actual grammar file from the reference implementation: https://github.com/1st1/cpython/blob/await/Grammar/Grammar Thanks, Yury

On 04/29, Yury Selivanov wrote:
Both you and Paul are correct on this, thank you. The proper resolution of await -coro() is indeed to get the result of coro(), call it's __neg__ method, and then await on that. And that is perfectly reasonable, and should not be a SyntaxError; what it might be is an AttributeError (no __neg__ method) or an AsyncError (__neg__ returned non-awaitable object), or might even just work [1]... but it definitely should /not/ be a SyntaxError. -- ~Ethan~ [1] http://stackoverflow.com/q/7719018/208880

On Thu, Apr 30, 2015 at 9:15 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Why not? Unlike some other languages, Python does not have uniform priorities for unary operators, so it's reasonable for some unary operations to have a different priority than others, and certain things will be SyntaxErrors because of that. E.g. you can write "not -x" but you can't write "- not x". This is because they have different priorities: 'not' has a very low priority so it binds less tight than comparisons (which bind less tight than arithmetic), but '-' has a high priority. (There are other quirks, e.g. -2**2 means -(2**2) and you can write 2**-2; but you can't write 2**not x.) -- --Guido van Rossum (python.org/~guido)

On 04/30, Guido van Rossum wrote:
For one, Yury's answer is "- await x" which looks just as nonsensical as "- not x". For another, an error of some type will be raised if either __neg__ doesn't exist or it doesn't return an awaitable, so a SyntaxError is unnecessary. For a third, by making it a SyntaxError you are forcing the use of parens to get what should be the behavior anyway. In other words, a SyntaxError is nat any clearer than "AttributeError: obj has no __neg__ method" and it's not any clearer than "AwaitError: __neg__ returned not-awaitable". Those last two errors tell you exactly what you did wrong. -- ~Ethan~

On 2015-04-30 1:56 PM, Ethan Furman wrote:
"- await x" is a perfectly valid code: result = - await compute_in_db() (same as "result = - (yield from do_something())")
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere. FWIW, I'll fix the 'await (await x)' expression to be parsed without parens.
This is debatable. "obj has no __neg__ method" isn't obvious to everyone (especially to those people who aren't using operator overloading). [1] https://www.python.org/dev/peps/pep-0492/#examples-of-await-expressions Yury

On 04/30, Yury Selivanov wrote:
On 2015-04-30 1:56 PM, Ethan Furman wrote:
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere.
--> await -coro() SyntaxError --> await (-coro()) # not a SyntaxError, therefore parens are # forced
Good news! The error there is actually --> -object() TypeError: bad operand type for unary -: 'object' Which is definitely clear, even for those who don't do operator overloading. -- ~Ethan~

On 1/05/2015 6:04 a.m., Yury Selivanov wrote:
I still want to see where my current grammar forces to use parens. See [1], there are no useless parens anywhere.
It's not about requiring or not requiring parens. It's about making the simplest possible change to the grammar necessary to achieve the desired goals. Keeping the grammar simple makes it easy for humans to reason about. The question is whether syntactically disallowing certain constructs that are unlikely to be needed is a desirable enough goal to be worth complicating the grammar. You think it is, some others of us think it's not.
FWIW, I'll fix the 'await (await x)' expression to be parsed without parens.
I don't particularly care whether 'await -x' or 'await await x' can be written without parens or not. The point is that the simplest grammar change necessary to be able to write the things we *do* want also happens to allow those. I don't see that as a problem worth worrying about. -- Greg

On Thu, Apr 30, 2015 at 6:13 PM, Greg <greg.ewing@canterbury.ac.nz> wrote:
+1. It seems weird to add a whole new precedence level when an existing one works fine. Accidentally negating a future/deferred is not a significant source of errors, so I don't get why that would be a justifying example. -- Devin

On Thu, Apr 30, 2015 at 6:56 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
You can call me weird, but I *like* fine-tuning operator binding rules to suit my intuition for an operator. 'await' is not arithmetic, so I don't see why it should be lumped in with '-'. It's not like the proposed grammar change introducing 'await' is earth-shattering in complexity. -- --Guido van Rossum (python.org/~guido)

On 1/05/2015 5:38 a.m., Guido van Rossum wrote:
you can write "not -x" but you can't write "- not x".
That seems just as arbitrary and unintuitive, though. There are some other unintuitive consequences as well, e.g. you can write not a + b but it's not immediately obvious that this is parsed as 'not (a + b)' rather than '(not a) + b'. The presence of one arbitrary and unintuitive thing in the grammar is not by itself a justification for adding another one. -- Greg

Yury Selivanov wrote:
I don't want this: "await a() * b()" to be parsed, it's not meaningful.
Why not? If a() is a coroutine that returns a number, why shouldn't I be able to multiply it by something? I don't think your currently proposed grammar prevents that anyway. We can have <expr> --> <factor> '*' <factor> --> <power> '*' <power> --> <atom_expr> '*' <atom_expr> --> 'await' <atom> <trailer>* '*' <atom> <trailer>* --> 'await' 'a' '(' ')' '*' 'b' '(' ')' It does, on the other hand, seem to prevent x = - await a() which looks perfectly sensible to me. I don't like the idea of introducing another level of precedence. Python already has too many of those to keep in my brain. Being able to tell people "it's just like unary minus" makes it easy to explain (and therefore possibly a good idea!). -- Greg

Greg, On 2015-04-29 1:40 AM, Greg Ewing wrote:
Sorry, I thought you meant parsing "await a()*b()" as "await (a() * b())".
This works just fine: https://github.com/1st1/cpython/commit/33b3cd052243cd71c064eb385c7a557eec3ce... Current grammar prevents this: "await -fut", and this: "await fut ** 2" being parsed as "await (fut ** 2)
It's just like unary minus ;) Yury
participants (14)
-
Alexander Walters
-
Andrew Svetlov
-
andrew.svetlov@gmail.com
-
Devin Jeanpierre
-
Ethan Furman
-
Greg
-
Greg Ewing
-
Guido van Rossum
-
Nathaniel Smith
-
Nick Coghlan
-
Paul Moore
-
Victor Stinner
-
Yury Selivanov
-
Łukasz Langa