[Python-ideas] async/await in Python

[Yury Selivanov]

Hi Chris,

Thanks for the feedback; answers below:

On 2015-04-17 6:00 PM, Chris Angelico wrote:
> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> The following new syntax is used to declare a coroutine::
>>
>>      async def read_data(db):
>>          pass
>>
>> Key properties of coroutines:
>>
>> * Coroutines are always generators, even if they do not contain ``await``
>>    expressions.
> Do you mean that a coroutine is a special form of generator, or that a
> function declared with "async" is always a coroutine even if it
> doesn't contain an "await"? You're separating coroutines from
> generators in many ways, so maybe I'm just misreading this.

Good catch; I've committed the fix.

Yes, ``async def`` functions are always coroutines no matter if
they have ``await`` exprs in them or not.
>
>> Asynchronous Context Managers and "async with"
>> ----------------------------------------------
>>
>> An *asynchronous context manager* is a context manager that is able to
>> suspend
>> execution in its *enter* and *exit* methods.
> A regular 'with' block guarantees that subsequent code won't execute
> until __exit__ completes. I presume the same guarantee is applied to
> an 'async with' block, and thus it can be used only inside a
> coroutine?
Correct.
>
> (Edit: Oops, missed where that's said further down - SyntaxError to
> use it outside a coroutine. Might still be worth clarifying the
> guarantee somewhere, otherwise ignore me.)
>
>> New Syntax
>> ''''''''''
>>
>> A new statement for iterating through asynchronous iterators is proposed::
>>
>>      async for TARGET in ITER:
>>          BLOCK
>>      else:
>>          BLOCK2
>>
>> which is semantically equivalent to::
>>
>>      iter = (ITER)
>>      iter = await type(iter).__aiter__(iter)
>>      running = True
>>      while running:
>>          try:
>>              TARGET = await type(iter).__anext__(iter)
>>          except StopAsyncIteration:
>>              running = False
>>          else:
>>              BLOCK
>>      else:
>>          BLOCK2
> (Not sure why you don't just use "break" instead of "running = False"?
> Maybe I'm blind to some distinction here.)

Because if you use break, the "else: BLOCK2" won't execute, but
it should, as StopAsyncIteration is not a regular exception, but
something to signal that the iteration is over.
>
>> Why StopAsyncIteration?
>> '''''''''''''''''''''''
>>
>> Coroutines are still based on generators internally.  So, before PEP 479,
>> there
>> was no fundamental difference between
>>
>> ::
>>
>>      def g1():
>>          yield from fut
>>          return 'spam'
>>
>> and
>>
>> ::
>>
>>      def g2():
>>          yield from fut
>>          raise StopIteration('spam')
>>
>> And since PEP 479 is accepted and enabled by default for coroutines, the
>> following example will have its ``StopIteration`` wrapped into a
>> ``RuntimeError``
>>
>> ::
>>
>>      async def a1():
>>          await fut
>>          raise StopIteration('spam')
>>
>> The only way to tell the outside code that the iteration has ended is to
>> raise
>> something other than ``StopIteration``.  Therefore, a new built-in exception
>> class ``StopAsyncIteration`` was added.
>>
>> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
>> raised
>> in coroutines are wrapped in ``RuntimeError``.
> Can't a coroutine simply use 'return'? I'm not hugely familiar with
> them, but I'm not understanding here why you need a completely
> separate exception - and, more importantly, why you wouldn't have the
> same consideration of "StopAsyncIteration leakage". It ought to be
> possible for the coroutine to return, StopIteration be synthesized,
> and the __anext__ function to bubble that out - exactly the way it
> does for generators today.

Unfortunately, there is no way we can enable StopIteration to do
the same thing that StopAsyncIteration does.

'return' in coroutines is implemented with 'StopIteration' exception
internally, so there is no way (without ugly hacks) to distinguish
before plain returns and desire to stop the iteration.
>
>> Glossary
>> ========
>>
>> :Coroutine:
>>      A coroutine function, or just "coroutine", is declared with ``async
>> def``.
>>      It uses ``await`` and ``return value``; see `New Coroutine Declaration
>>      Syntax`_ for details.
> (Using this definition to justify my statement above)
>
>> Why not a __future__ import
>> ---------------------------
>>
>> ``__future__`` imports are inconvenient and easy to forget to add. Also,
>> they
>> are enabled for the whole source file.  Consider that there is a big project
>> with a popular module named "async.py".  With future imports it is required
>> to
>> either import it using ``__import__()`` or ``importlib.import_module()``
>> calls,
>> or to rename the module.  The proposed approach makes it possible to
>> continue
>> using old code and modules without a hassle, while coming up with a
>> migration
>> plan for future python versions.
> The clash would occur only in modules that use the __future__ import,
> wouldn't it? And those modules are going to have to be fixed by
> version X.Y anyway - the whole point of the __future__ import is to
> detect that.
You're right.

Let's see what python-ideas thinks about it.  I'm fine if everybody
wants __future__ imports.  I will only have to rollback my changes
in tokenizer.c and change few tokens in Grammar to make the
reference implementation work.
>> Why magic methods start with "a"
>> --------------------------------
>>
>> New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``,
>> and ``__aexit__`` all start with the same prefix "a".  An alternative
>> proposal
>> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``.
>> However, to align new magic methods with the existing ones, such as
>> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version.
> I support the short names, with the possible exception that
> "__await__" now looks like "asynchronous wait", which may or may not
> be desirable.
>
>> Comprehensions
>> --------------
>>
>> For the sake of restricting the broadness of this PEP there is no new syntax
>> for asynchronous comprehensions.  This should be considered in a separate
>> PEP,
>> if there is a strong demand for this feature.
> Likewise asynchronous lambda functions (or maybe I just missed it).
Right. Right now I'm just not sure they would be useful at all,
whereas I can imagine some usecases for comprehensions ;)

>
>
> Overall comment: The changes appear to be infecting a lot of code.
> Will this suddenly be something that everyone has to think about, or
> can it still be ignored in projects that don't use it?

If you're not writing some asynchronous code or using some
asynchronous library that takes advantage of coroutines, you
can completely ignore the new syntax and protocols.

> There'll be a
> few object types that would benefit from becoming async context
> managers as well as regular context managers (I would expect, for
> instance, that the built-in file type would allow its auto-closing to
> be asynchronous), but if a third-party module completely ignores this
> proposal, everything should continue to work, right?
Exactly. That's why we propose to have separate protocols
__a*__ to completely eliminate possibility of screwing
something up.

>
> I'm seeing the potential for a lot of code duplication here, but
> possibly once things settle into real-world usage, a few easy
> decorators will help out with that (like the total_ordering decorator
> for handling __gt__, __le__, etc etc).
>
> +1 on the proposal as a whole.
Awesome!

Thanks a lot!

Yury