<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div>Hi,<br><br></div>most of the time I am a silent reader but in this discussion I must step in.<br>I use twisted and async stuff a lot over years followed development of asyncio closely.<br><br></div>First it is good to do differentiate async coroutines from generators. So everyone can see it and have this in mind<br></div>and don't mix up booth. It is also easier to explain for new users. Sometimes generators blows their mind and it takes<br></div>a while to get used to them. Async stuff is even harder.<br><br></div>1. I am fine with using something special instead of "yield" or "yield from" for this. C# "await" is ok.<br><br></div>Everything else suggested complicates the language and makes it harder to read.<br><br>2.<br></div>async def f(): is harder to read and something special also it breaks the symmetry in front (def indent).<br></div>Also every existing tooling must be changed to support it. Same for def async, def f() async:<br></div>I thing a decorator is enough here<br></div>@coroutine<br></div>def f():<br></div>is the best solution to mark something as a coroutine. <br><div><div><div><div><div><div><div><div><div><div><div><br><br>3. <br></div><div>async with and async for<br></div><div>Bead idea, we clutter the language even more and it is one more thing every newbie could do wrong.<br></div><div>for x in y:<br></div><div> result = await f()<br></div><div>is enough, every 'async' framework lived without it over years.<br></div><div>Same for with statement.<br><br></div><div>The main use case suggested was for database stuff and this is also where most are best with<br></div><div>defer something to a thread and keep it none async.<br></div><div><br><br></div><div>All together it is very late in the development cycle for 3.5 to incorporate such a big change.<br></div><div>Best is to give all this some more time and defer it to 3.6 and some alpha releases to experiment with.<br><br></div><div>Regards,<br><br></div><div>Wolfgang<br></div><div><br><br></div></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 21, 2015 at 7:26 PM, Yury Selivanov <span dir="ltr"><<a href="mailto:yselivanov.ml@gmail.com" target="_blank">yselivanov.ml@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi python-dev,<br>
<br>
I'm moving the discussion from python-ideas to here.<br>
<br>
The updated version of the PEP should be available shortly<br>
at <a href="https://www.python.org/dev/peps/pep-0492" target="_blank">https://www.python.org/dev/peps/pep-0492</a><br>
and is also pasted in this email.<br>
<br>
Updates:<br>
<br>
1. CO_ASYNC flag was renamed to CO_COROUTINE;<br>
<br>
2. sys.set_async_wrapper() was renamed to<br>
sys.set_coroutine_wrapper();<br>
<br>
3. New function: sys.get_coroutine_wrapper();<br>
<br>
4. types.async_def() renamed to types.coroutine();<br>
<br>
5. New section highlighting differences from<br>
PEP 3152.<br>
<br>
6. New AST node - AsyncFunctionDef; the proposal<br>
now is 100% backwards compatible;<br>
<br>
7. A new section clarifying that coroutine-generators<br>
are not part of the current proposal;<br>
<br>
8. Various small edits/typos fixes.<br>
<br>
<br>
There's is a bug tracker issue to track code review<br>
of the reference implementation (Victor Stinner is<br>
doing the review): <a href="http://bugs.python.org/issue24017" target="_blank">http://bugs.python.org/issue24017</a><br>
While the PEP isn't accepted, we want to make sure<br>
that the reference implementation is ready when such<br>
a decision will be made.<br>
<br>
<br>
Let's discuss some open questions:<br>
<br>
1. Victor raised a question if we should locate<br>
coroutine() function from 'types' module to<br>
'functools'.<br>
<br>
My opinion is that 'types' module is a better<br>
place for 'corotine()', since it adjusts the<br>
type of the passed generator. 'functools' is<br>
about 'partials', 'lru_cache' and 'wraps' kind<br>
of things.<br>
<br>
2. I propose to disallow using of 'for..in' loops,<br>
and builtins like 'list()', 'iter()', 'next()',<br>
'tuple()' etc on coroutines.<br>
<br>
It's possible by modifying PyObject_GetIter to<br>
raise an exception if it receives a coroutine-object.<br>
<br>
'yield from' can also be modified to only accept<br>
coroutine objects if it is called from a generator<br>
with CO_COROUTINE flag.<br>
<br>
This will further separate coroutines from<br>
generators, making it harder to screw something<br>
up by an accident.<br>
<br>
I have a branch of reference implementation<br>
<a href="https://github.com/1st1/cpython/tree/await_noiter" target="_blank">https://github.com/1st1/cpython/tree/await_noiter</a><br>
where this is implemented. I did not observe<br>
any performance drop.<br>
<br>
There is just one possible backwards compatibility<br>
issue here: there will be an exception if some user<br>
of asyncio actually used to iterate over generators<br>
decorated with @coroutine. But I can't imagine<br>
why would someone do that, and even if they did --<br>
it's probably a bug or wrong usage of asyncio.<br>
<br>
<br>
That's it! I'd be happy to hear some feedback!<br>
<br>
Thanks,<br>
Yury<br>
<br>
<br>
<br>
PEP: 492<br>
Title: Coroutines with async and await syntax<br>
Version: $Revision$<br>
Last-Modified: $Date$<br>
Author: Yury Selivanov <<a href="mailto:yselivanov@sprymix.com" target="_blank">yselivanov@sprymix.com</a>><br>
Status: Draft<br>
Type: Standards Track<br>
Content-Type: text/x-rst<br>
Created: 09-Apr-2015<br>
Python-Version: 3.5<br>
Post-History: 17-Apr-2015, 21-Apr-2015<br>
<br>
<br>
Abstract<br>
========<br>
<br>
This PEP introduces new syntax for coroutines, asynchronous ``with``<br>
statements and ``for`` loops. The main motivation behind this proposal<br>
is to streamline writing and maintaining asynchronous code, as well as<br>
to simplify previously hard to implement code patterns.<br>
<br>
<br>
Rationale and Goals<br>
===================<br>
<br>
Current Python supports implementing coroutines via generators (PEP<br>
342), further enhanced by the ``yield from`` syntax introduced in PEP<br>
380. This approach has a number of shortcomings:<br>
<br>
* it is easy to confuse coroutines with regular generators, since they<br>
share the same syntax; async libraries often attempt to alleviate<br>
this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);<br>
<br>
* it is not possible to natively define a coroutine which has no<br>
``yield`` or ``yield from`` statements, again requiring the use of<br>
decorators to fix potential refactoring issues;<br>
<br>
* support for asynchronous calls is limited to expressions where<br>
``yield`` is allowed syntactically, limiting the usefulness of<br>
syntactic features, such as ``with`` and ``for`` statements.<br>
<br>
This proposal makes coroutines a native Python language feature, and<br>
clearly separates them from generators. This removes<br>
generator/coroutine ambiguity, and makes it possible to reliably define<br>
coroutines without reliance on a specific library. This also enables<br>
linters and IDEs to improve static code analysis and refactoring.<br>
<br>
Native coroutines and the associated new syntax features make it<br>
possible to define context manager and iteration protocols in<br>
asynchronous terms. As shown later in this proposal, the new ``async<br>
with`` statement lets Python programs perform asynchronous calls when<br>
entering and exiting a runtime context, and the new ``async for``<br>
statement makes it possible to perform asynchronous calls in iterators.<br>
<br>
<br>
Specification<br>
=============<br>
<br>
This proposal introduces new syntax and semantics to enhance coroutine<br>
support in Python, it does not change the internal implementation of<br>
coroutines, which are still based on generators.<br>
<br>
It is strongly suggested that the reader understands how coroutines are<br>
implemented in Python (PEP 342 and PEP 380). It is also recommended to<br>
read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions).<br>
<br>
>From this point in this document we use the word *coroutine* to refer<br>
to functions declared using the new syntax. *generator-based<br>
coroutine* is used where necessary to refer to coroutines that are<br>
based on generator syntax.<br>
<br>
<br>
New Coroutine Declaration Syntax<br>
--------------------------------<br>
<br>
The following new syntax is used to declare a coroutine::<br>
<br>
async def read_data(db):<br>
pass<br>
<br>
Key properties of coroutines:<br>
<br>
* ``async def`` functions are always coroutines, even if they do not<br>
contain ``await`` expressions.<br>
<br>
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``<br>
expressions in an ``async`` function.<br>
<br>
* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced<br>
to enable runtime detection of coroutines (and migrating existing<br>
code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``<br>
flags set.<br>
<br>
* Regular generators, when called, return a *generator object*;<br>
similarly, coroutines return a *coroutine object*.<br>
<br>
* ``StopIteration`` exceptions are not propagated out of coroutines,<br>
and are replaced with a ``RuntimeError``. For regular generators<br>
such behavior requires a future import (see PEP 479).<br>
<br>
<br>
types.coroutine()<br>
-----------------<br>
<br>
A new function ``coroutine(gen)`` is added to the ``types`` module. It<br>
applies ``CO_COROUTINE`` flag to the passed generator-function's code<br>
object, making it to return a *coroutine object* when called.<br>
<br>
This feature enables an easy upgrade path for existing libraries.<br>
<br>
<br>
Await Expression<br>
----------------<br>
<br>
The following new ``await`` expression is used to obtain a result of<br>
coroutine execution::<br>
<br>
async def read_data(db):<br>
data = await db.fetch('SELECT ...')<br>
...<br>
<br>
``await``, similarly to ``yield from``, suspends execution of<br>
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and<br>
returns the result data.<br>
<br>
It uses the ``yield from`` implementation with an extra step of<br>
validating its argument. ``await`` only accepts an *awaitable*, which<br>
can be one of:<br>
<br>
* A *coroutine object* returned from a *coroutine* or a generator<br>
decorated with ``types.coroutine()``.<br>
<br>
* An object with an ``__await__`` method returning an iterator.<br>
<br>
Any ``yield from`` chain of calls ends with a ``yield``. This is a<br>
fundamental mechanism of how *Futures* are implemented. Since,<br>
internally, coroutines are a special kind of generators, every<br>
``await`` is suspended by a ``yield`` somewhere down the chain of<br>
``await`` calls (please refer to PEP 3156 for a detailed<br>
explanation.)<br>
<br>
To enable this behavior for coroutines, a new magic method called<br>
``__await__`` is added. In asyncio, for instance, to enable Future<br>
objects in ``await`` statements, the only change is to add<br>
``__await__ = __iter__`` line to ``asyncio.Future`` class.<br>
<br>
Objects with ``__await__`` method are called *Future-like* objects in<br>
the rest of this PEP.<br>
<br>
Also, please note that ``__aiter__`` method (see its definition<br>
below) cannot be used for this purpose. It is a different protocol,<br>
and would be like using ``__iter__`` instead of ``__call__`` for<br>
regular callables.<br>
<br>
It is a ``SyntaxError`` to use ``await`` outside of a coroutine.<br>
<br>
<br>
Asynchronous Context Managers and "async with"<br>
----------------------------------------------<br>
<br>
An *asynchronous context manager* is a context manager that is able to<br>
suspend execution in its *enter* and *exit* methods.<br>
<br>
To make this possible, a new protocol for asynchronous context managers<br>
is proposed. Two new magic methods are added: ``__aenter__`` and<br>
``__aexit__``. Both must return an *awaitable*.<br>
<br>
An example of an asynchronous context manager::<br>
<br>
class AsyncContextManager:<br>
async def __aenter__(self):<br>
await log('entering context')<br>
<br>
async def __aexit__(self, exc_type, exc, tb):<br>
await log('exiting context')<br>
<br>
<br>
New Syntax<br>
''''''''''<br>
<br>
A new statement for asynchronous context managers is proposed::<br>
<br>
async with EXPR as VAR:<br>
BLOCK<br>
<br>
<br>
which is semantically equivalent to::<br>
<br>
mgr = (EXPR)<br>
aexit = type(mgr).__aexit__<br>
aenter = type(mgr).__aenter__(mgr)<br>
exc = True<br>
<br>
try:<br>
try:<br>
VAR = await aenter<br>
BLOCK<br>
except:<br>
exc = False<br>
exit_res = await aexit(mgr, *sys.exc_info())<br>
if not exit_res:<br>
raise<br>
<br>
finally:<br>
if exc:<br>
await aexit(mgr, None, None, None)<br>
<br>
<br>
As with regular ``with`` statements, it is possible to specify multiple<br>
context managers in a single ``async with`` statement.<br>
<br>
It is an error to pass a regular context manager without ``__aenter__``<br>
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``<br>
to use ``async with`` outside of a coroutine.<br>
<br>
<br>
Example<br>
'''''''<br>
<br>
With asynchronous context managers it is easy to implement proper<br>
database transaction managers for coroutines::<br>
<br>
async def commit(session, data):<br>
...<br>
<br>
async with session.transaction():<br>
...<br>
await session.update(data)<br>
...<br>
<br>
Code that needs locking also looks lighter::<br>
<br>
async with lock:<br>
...<br>
<br>
instead of::<br>
<br>
with (yield from lock):<br>
...<br>
<br>
<br>
Asynchronous Iterators and "async for"<br>
--------------------------------------<br>
<br>
An *asynchronous iterable* is able to call asynchronous code in its<br>
*iter* implementation, and *asynchronous iterator* can call<br>
asynchronous code in its *next* method. To support asynchronous<br>
iteration:<br>
<br>
1. An object must implement an ``__aiter__`` method returning an<br>
*awaitable* resulting in an *asynchronous iterator object*.<br>
<br>
2. An *asynchronous iterator object* must implement an ``__anext__``<br>
method returning an *awaitable*.<br>
<br>
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``<br>
exception.<br>
<br>
An example of asynchronous iterable::<br>
<br>
class AsyncIterable:<br>
async def __aiter__(self):<br>
return self<br>
<br>
async def __anext__(self):<br>
data = await self.fetch_data()<br>
if data:<br>
return data<br>
else:<br>
raise StopAsyncIteration<br>
<br>
async def fetch_data(self):<br>
...<br>
<br>
<br>
New Syntax<br>
''''''''''<br>
<br>
A new statement for iterating through asynchronous iterators is<br>
proposed::<br>
<br>
async for TARGET in ITER:<br>
BLOCK<br>
else:<br>
BLOCK2<br>
<br>
which is semantically equivalent to::<br>
<br>
iter = (ITER)<br>
iter = await type(iter).__aiter__(iter)<br>
running = True<br>
while running:<br>
try:<br>
TARGET = await type(iter).__anext__(iter)<br>
except StopAsyncIteration:<br>
running = False<br>
else:<br>
BLOCK<br>
else:<br>
BLOCK2<br>
<br>
<br>
It is an error to pass a regular iterable without ``__aiter__`` method<br>
to ``async for``. It is a ``SyntaxError`` to use ``async for`` outside<br>
of a coroutine.<br>
<br>
As for with regular ``for`` statement, ``async for`` has an optional<br>
``else`` clause.<br>
<br>
<br>
Example 1<br>
'''''''''<br>
<br>
With asynchronous iteration protocol it is possible to asynchronously<br>
buffer data during iteration::<br>
<br>
async for data in cursor:<br>
...<br>
<br>
Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows<br>
of data from a database after every ``N`` iterations.<br>
<br>
The following code illustrates new asynchronous iteration protocol::<br>
<br>
class Cursor:<br>
def __init__(self):<br>
self.buffer = collections.deque()<br>
<br>
def _prefetch(self):<br>
...<br>
<br>
async def __aiter__(self):<br>
return self<br>
<br>
async def __anext__(self):<br>
if not self.buffer:<br>
self.buffer = await self._prefetch()<br>
if not self.buffer:<br>
raise StopAsyncIteration<br>
return self.buffer.popleft()<br>
<br>
then the ``Cursor`` class can be used as follows::<br>
<br>
async for row in Cursor():<br>
print(row)<br>
<br>
which would be equivalent to the following code::<br>
<br>
i = await Cursor().__aiter__()<br>
while True:<br>
try:<br>
row = await i.__anext__()<br>
except StopAsyncIteration:<br>
break<br>
else:<br>
print(row)<br>
<br>
<br>
Example 2<br>
'''''''''<br>
<br>
The following is a utility class that transforms a regular iterable to<br>
an asynchronous one. While this is not a very useful thing to do, the<br>
code illustrates the relationship between regular and asynchronous<br>
iterators.<br>
<br>
::<br>
<br>
class AsyncIteratorWrapper:<br>
def __init__(self, obj):<br>
self._it = iter(obj)<br>
<br>
async def __aiter__(self):<br>
return self<br>
<br>
async def __anext__(self):<br>
try:<br>
value = next(self._it)<br>
except StopIteration:<br>
raise StopAsyncIteration<br>
return value<br>
<br>
async for item in AsyncIteratorWrapper("abc"):<br>
print(item)<br>
<br>
<br>
Why StopAsyncIteration?<br>
'''''''''''''''''''''''<br>
<br>
Coroutines are still based on generators internally. So, before PEP<br>
479, there was no fundamental difference between<br>
<br>
::<br>
<br>
def g1():<br>
yield from fut<br>
return 'spam'<br>
<br>
and<br>
<br>
::<br>
<br>
def g2():<br>
yield from fut<br>
raise StopIteration('spam')<br>
<br>
And since PEP 479 is accepted and enabled by default for coroutines,<br>
the following example will have its ``StopIteration`` wrapped into a<br>
``RuntimeError``<br>
<br>
::<br>
<br>
async def a1():<br>
await fut<br>
raise StopIteration('spam')<br>
<br>
The only way to tell the outside code that the iteration has ended is<br>
to raise something other than ``StopIteration``. Therefore, a new<br>
built-in exception class ``StopAsyncIteration`` was added.<br>
<br>
Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions<br>
raised in coroutines are wrapped in ``RuntimeError``.<br>
<br>
<br>
Debugging Features<br>
------------------<br>
<br>
One of the most frequent mistakes that people make when using<br>
generators as coroutines is forgetting to use ``yield from``::<br>
<br>
@asyncio.coroutine<br>
def useful():<br>
asyncio.sleep(1) # this will do noting without 'yield from'<br>
<br>
For debugging this kind of mistakes there is a special debug mode in<br>
asyncio, in which ``@coroutine`` decorator wraps all functions with a<br>
special object with a destructor logging a warning. Whenever a wrapped<br>
generator gets garbage collected, a detailed logging message is<br>
generated with information about where exactly the decorator function<br>
was defined, stack trace of where it was collected, etc. Wrapper<br>
object also provides a convenient ``__repr__`` function with detailed<br>
information about the generator.<br>
<br>
The only problem is how to enable these debug capabilities. Since<br>
debug facilities should be a no-op in production mode, ``@coroutine``<br>
decorator makes the decision of whether to wrap or not to wrap based on<br>
an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is<br>
possible to run asyncio programs with asyncio's own functions<br>
instrumented. ``EventLoop.set_debug``, a different debug facility, has<br>
no impact on ``@coroutine`` decorator's behavior.<br>
<br>
With this proposal, coroutines is a native, distinct from generators,<br>
concept. New methods ``set_coroutine_wrapper`` and<br>
``get_coroutine_wrapper`` are added to the ``sys`` module, with which<br>
frameworks can provide advanced debugging facilities.<br>
<br>
It is also important to make coroutines as fast and efficient as<br>
possible, therefore there are no debug features enabled by default.<br>
<br>
Example::<br>
<br>
async def debug_me():<br>
await asyncio.sleep(1)<br>
<br>
def async_debug_wrap(generator):<br>
return asyncio.AsyncDebugWrapper(generator)<br>
<br>
sys.set_coroutine_wrapper(async_debug_wrap)<br>
<br>
debug_me() # <- this line will likely GC the coroutine object and<br>
# trigger AsyncDebugWrapper's code.<br>
<br>
assert isinstance(debug_me(), AsyncDebugWrapper)<br>
<br>
sys.set_coroutine_wrapper(None) # <- this unsets any<br>
# previously set wrapper<br>
assert not isinstance(debug_me(), AsyncDebugWrapper)<br>
<br>
If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper<br>
replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)``<br>
unsets the wrapper.<br>
<br>
<br>
Glossary<br>
========<br>
<br>
:Coroutine:<br>
A coroutine function, or just "coroutine", is declared with ``async<br>
def``. It uses ``await`` and ``return value``; see `New Coroutine<br>
Declaration Syntax`_ for details.<br>
<br>
:Coroutine object:<br>
Returned from a coroutine function. See `Await Expression`_ for<br>
details.<br>
<br>
:Future-like object:<br>
An object with an ``__await__`` method. Can be consumed by an<br>
``await`` expression in a coroutine. A coroutine waiting for a<br>
Future-like object is suspended until the Future-like object's<br>
``__await__`` completes, and returns the result. See `Await<br>
Expression`_ for details.<br>
<br>
:Awaitable:<br>
A *Future-like* object or a *coroutine object*. See `Await<br>
Expression`_ for details.<br>
<br>
:Generator-based coroutine:<br>
Coroutines based in generator syntax. Most common example is<br>
``@asyncio.coroutine``.<br>
<br>
:Asynchronous context manager:<br>
An asynchronous context manager has ``__aenter__`` and ``__aexit__``<br>
methods and can be used with ``async with``. See `Asynchronous<br>
Context Managers and "async with"`_ for details.<br>
<br>
:Asynchronous iterable:<br>
An object with an ``__aiter__`` method, which must return an<br>
*asynchronous iterator* object. Can be used with ``async for``.<br>
See `Asynchronous Iterators and "async for"`_ for details.<br>
<br>
:Asynchronous iterator:<br>
An asynchronous iterator has an ``__anext__`` method. See<br>
`Asynchronous Iterators and "async for"`_ for details.<br>
<br>
<br>
List of functions and methods<br>
=============================<br>
<br>
================= =================================== =================<br>
Method Can contain Can't contain<br>
================= =================================== =================<br>
async def func await, return value yield, yield from<br>
async def __a*__ await, return value yield, yield from<br>
def __a*__ return awaitable await<br>
def __await__ yield, yield from, return iterable await<br>
generator yield, yield from, return value await<br>
================= =================================== =================<br>
<br>
Where:<br>
<br>
* "async def func": coroutine;<br>
<br>
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,<br>
``__aexit__`` defined with the ``async`` keyword;<br>
<br>
* "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,<br>
``__aexit__`` defined without the ``async`` keyword, must return an<br>
*awaitable*;<br>
<br>
* "def __await__": ``__await__`` method to implement *Future-like*<br>
objects;<br>
<br>
* generator: a "regular" generator, function defined with ``def`` and<br>
which contains a least one ``yield`` or ``yield from`` expression.<br>
<br>
<br>
Transition Plan<br>
===============<br>
<br>
To avoid backwards compatibility issues with ``async`` and ``await``<br>
keywords, it was decided to modify ``tokenizer.c`` in such a way, that<br>
it:<br>
<br>
* recognizes ``async def`` name tokens combination (start of a<br>
coroutine);<br>
<br>
* keeps track of regular functions and coroutines;<br>
<br>
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with<br>
``AWAIT`` when in the process of yielding tokens for coroutines.<br>
<br>
This approach allows for seamless combination of new syntax features<br>
(all of them available only in ``async`` functions) with any existing<br>
code.<br>
<br>
An example of having "async def" and "async" attribute in one piece of<br>
code::<br>
<br>
class Spam:<br>
async = 42<br>
<br>
async def ham():<br>
print(getattr(Spam, 'async'))<br>
<br>
# The coroutine can be executed and will print '42'<br>
<br>
<br>
Backwards Compatibility<br>
-----------------------<br>
<br>
This proposal preserves 100% backwards compatibility.<br>
<br>
<br>
Grammar Updates<br>
---------------<br>
<br>
Grammar changes are also fairly minimal::<br>
<br>
await_expr: AWAIT test<br>
await_stmt: await_expr<br>
<br>
decorated: decorators (classdef | funcdef | async_funcdef)<br>
async_funcdef: ASYNC funcdef<br>
<br>
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)<br>
<br>
compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt |<br>
with_stmt | funcdef | classdef | decorated |<br>
async_stmt)<br>
<br>
atom: ('(' [yield_expr|await_expr|testlist_comp] ')' |<br>
'[' [testlist_comp] ']' |<br>
'{' [dictorsetmaker] '}' |<br>
NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False’)<br>
<br>
expr_stmt: testlist_star_expr<br>
(augassign (yield_expr|await_expr|testlist) |<br>
('=' (yield_expr|await_expr|testlist_star_expr))*)<br>
<br>
<br>
Transition Period Shortcomings<br>
------------------------------<br>
<br>
There is just one.<br>
<br>
Until ``async`` and ``await`` are not proper keywords, it is not<br>
possible (or at least very hard) to fix ``tokenizer.c`` to recognize<br>
them on the **same line** with ``def`` keyword::<br>
<br>
# async and await will always be parsed as variables<br>
<br>
async def outer(): # 1<br>
def nested(a=(await fut)):<br>
pass<br>
<br>
async def foo(): return (await fut) # 2<br>
<br>
Since ``await`` and ``async`` in such cases are parsed as ``NAME``<br>
tokens, a ``SyntaxError`` will be raised.<br>
<br>
To workaround these issues, the above examples can be easily rewritten<br>
to a more readable form::<br>
<br>
async def outer(): # 1<br>
a_default = await fut<br>
def nested(a=a_default):<br>
pass<br>
<br>
async def foo(): # 2<br>
return (await fut)<br>
<br>
This limitation will go away as soon as ``async`` and ``await`` ate<br>
proper keywords. Or if it's decided to use a future import for this<br>
PEP.<br>
<br>
<br>
Deprecation Plans<br>
-----------------<br>
<br>
``async`` and ``await`` names will be softly deprecated in CPython 3.5<br>
and 3.6. In 3.7 we will transform them to proper keywords. Making<br>
``async`` and ``await`` proper keywords before 3.7 might make it harder<br>
for people to port their code to Python 3.<br>
<br>
<br>
asyncio<br>
-------<br>
<br>
``asyncio`` module was adapted and tested to work with coroutines and<br>
new statements. Backwards compatibility is 100% preserved.<br>
<br>
The required changes are mainly:<br>
<br>
1. Modify ``@asyncio.coroutine`` decorator to use new<br>
``types.coroutine()`` function.<br>
<br>
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.<br>
<br>
3. Add ``ensure_task()`` as an alias for ``async()`` function.<br>
Deprecate ``async()`` function.<br>
<br>
<br>
Design Considerations<br>
=====================<br>
<br>
PEP 3152<br>
--------<br>
<br>
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines<br>
(called "cofunctions"). Some key points:<br>
<br>
1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is<br>
always a generator, even if there is no ``cocall`` expressions<br>
inside it. Maps to ``async def`` in this proposal.<br>
<br>
2. A new keyword ``cocall`` to call a *cofunction*. Can only be used<br>
inside a *cofunction*. Maps to ``await`` in this proposal (with<br>
some differences, see below.)<br>
<br>
3. It is not possible to call a *cofunction* without a ``cocall``<br>
keyword.<br>
<br>
4. ``cocall`` grammatically requires parentheses after it::<br>
<br>
atom: cocall | <existing alternatives for atom><br>
cocall: 'cocall' atom cotrailer* '(' [arglist] ')'<br>
cotrailer: '[' subscriptlist ']' | '.' NAME<br>
<br>
5. ``cocall f(*args, **kwds)`` is semantically equivalent to<br>
``yield from f.__cocall__(*args, **kwds)``.<br>
<br>
Differences from this proposal:<br>
<br>
1. There is no equivalent of ``__cocall__`` in this PEP, which is<br>
called and its result is passed to ``yield from`` in the ``cocall``<br>
expression. ``await`` keyword expects an *awaitable* object,<br>
validates the type, and executes ``yield from`` on it. Although,<br>
``__await__`` method is similar to ``__cocall__``, but is only used<br>
to define *Future-like* objects.<br>
<br>
2. ``await`` is defined in almost the same way as ``yield from`` in the<br>
grammar (it is later enforced that ``await`` can only be inside<br>
``async def``). It is possible to simply write ``await future``,<br>
whereas ``cocall`` always requires parentheses.<br>
<br>
3. To make asyncio work with PEP 3152 it would be required to modify<br>
``@asyncio.coroutine`` decorator to wrap all functions in an object<br>
with a ``__cocall__`` method. To call *cofunctions* from existing<br>
generator-based coroutines it would be required to use ``costart``<br>
built-in. In this proposal ``@asyncio.coroutine`` simply sets<br>
``CO_COROUTINE`` on the wrapped function's code object and<br>
everything works automatically.<br>
<br>
4. Since it is impossible to call a *cofunction* without a ``cocall``<br>
keyword, it automatically prevents the common mistake of forgetting<br>
to use ``yield from`` on generator-based coroutines. This proposal<br>
addresses this problem with a different approach, see `Debugging<br>
Features`_.<br>
<br>
5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine<br>
is that if is decided to implement coroutine-generators --<br>
coroutines with ``yield`` or ``async yield`` expressions -- we<br>
wouldn't need a ``cocall`` keyword to call them. So we'll end up<br>
having ``__cocall__`` and no ``__call__`` for regular coroutines,<br>
and having ``__call__`` and no ``__cocall__`` for coroutine-<br>
generators.<br>
<br>
6. There are no equivalents of ``async for`` and ``async with`` in PEP<br>
3152.<br>
<br>
<br>
Coroutine-generators<br>
--------------------<br>
<br>
With ``async for`` keyword it is desirable to have a concept of a<br>
*coroutine-generator* -- a coroutine with ``yield`` and ``yield from``<br>
expressions. To avoid any ambiguity with regular generators, we would<br>
likely require to have an ``async`` keyword before ``yield``, and<br>
``async yield from`` would raise a ``StopAsyncIteration`` exception.<br>
<br>
While it is possible to implement coroutine-generators, we believe that<br>
they are out of scope of this proposal. It is an advanced concept that<br>
should be carefully considered and balanced, with a non-trivial changes<br>
in the implementation of current generator objects. This is a matter<br>
for a separate PEP.<br>
<br>
<br>
No implicit wrapping in Futures<br>
-------------------------------<br>
<br>
There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A<br>
key difference is that JavaScript "async functions" always return a<br>
Promise. While this approach has some advantages, it also implies that<br>
a new Promise object is created on each "async function" invocation.<br>
<br>
We could implement a similar functionality in Python, by wrapping all<br>
coroutines in a Future object, but this has the following<br>
disadvantages:<br>
<br>
1. Performance. A new Future object would be instantiated on each<br>
coroutine call. Moreover, this makes implementation of ``await``<br>
expressions slower (disabling optimizations of ``yield from``).<br>
<br>
2. A new built-in ``Future`` object would need to be added.<br>
<br>
3. Coming up with a generic ``Future`` interface that is usable for any<br>
use case in any framework is a very hard to solve problem.<br>
<br>
4. It is not a feature that is used frequently, when most of the code<br>
is coroutines.<br>
<br>
<br>
Why "async" and "await" keywords<br>
--------------------------------<br>
<br>
async/await is not a new concept in programming languages:<br>
<br>
* C# has it since long time ago [5]_;<br>
<br>
* proposal to add async/await in ECMAScript 7 [2]_;<br>
see also Traceur project [9]_;<br>
<br>
* Facebook's Hack/HHVM [6]_;<br>
<br>
* Google's Dart language [7]_;<br>
<br>
* Scala [8]_;<br>
<br>
* proposal to add async/await to C++ [10]_;<br>
<br>
* and many other less popular languages.<br>
<br>
This is a huge benefit, as some users already have experience with<br>
async/await, and because it makes working with many languages in one<br>
project easier (Python with ECMAScript 7 for instance).<br>
<br>
<br>
Why "__aiter__" is a coroutine<br>
------------------------------<br>
<br>
In principle, ``__aiter__`` could be a regular function. There are<br>
several good reasons to make it a coroutine:<br>
<br>
* as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``<br>
methods are coroutines, users would often make a mistake defining it<br>
as ``async`` anyways;<br>
<br>
* there might be a need to run some asynchronous operations in<br>
``__aiter__``, for instance to prepare DB queries or do some file<br>
operation.<br>
<br>
<br>
Importance of "async" keyword<br>
-----------------------------<br>
<br>
While it is possible to just implement ``await`` expression and treat<br>
all functions with at least one ``await`` as coroutines, this approach<br>
makes APIs design, code refactoring and its long time support harder.<br>
<br>
Let's pretend that Python only has ``await`` keyword::<br>
<br>
def useful():<br>
...<br>
await log(...)<br>
...<br>
<br>
def important():<br>
await useful()<br>
<br>
If ``useful()`` function is refactored and someone removes all<br>
``await`` expressions from it, it would become a regular python<br>
function, and all code that depends on it, including ``important()``<br>
would be broken. To mitigate this issue a decorator similar to<br>
``@asyncio.coroutine`` has to be introduced.<br>
<br>
<br>
Why "async def"<br>
---------------<br>
<br>
For some people bare ``async name(): pass`` syntax might look more<br>
appealing than ``async def name(): pass``. It is certainly easier to<br>
type. But on the other hand, it breaks the symmetry between ``async<br>
def``, ``async with`` and ``async for``, where ``async`` is a modifier,<br>
stating that the statement is asynchronous. It is also more consistent<br>
with the existing grammar.<br>
<br>
<br>
Why not a __future__ import<br>
---------------------------<br>
<br>
``__future__`` imports are inconvenient and easy to forget to add.<br>
Also, they are enabled for the whole source file. Consider that there<br>
is a big project with a popular module named "async.py". With future<br>
imports it is required to either import it using ``__import__()`` or<br>
``importlib.import_module()`` calls, or to rename the module. The<br>
proposed approach makes it possible to continue using old code and<br>
modules without a hassle, while coming up with a migration plan for<br>
future python versions.<br>
<br>
<br>
Why magic methods start with "a"<br>
--------------------------------<br>
<br>
New asynchronous magic methods ``__aiter__``, ``__anext__``,<br>
``__aenter__``, and ``__aexit__`` all start with the same prefix "a".<br>
An alternative proposal is to use "async" prefix, so that ``__aiter__``<br>
becomes ``__async_iter__``. However, to align new magic methods with<br>
the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided<br>
to use a shorter version.<br>
<br>
<br>
Why not reuse existing magic names<br>
----------------------------------<br>
<br>
An alternative idea about new asynchronous iterators and context<br>
managers was to reuse existing magic methods, by adding an ``async``<br>
keyword to their declarations::<br>
<br>
class CM:<br>
async def __enter__(self): # instead of __aenter__<br>
...<br>
<br>
This approach has the following downsides:<br>
<br>
* it would not be possible to create an object that works in both<br>
``with`` and ``async with`` statements;<br>
<br>
* it would look confusing and would require some implicit magic behind<br>
the scenes in the interpreter;<br>
<br>
* one of the main points of this proposal is to make coroutines as<br>
simple and foolproof as possible.<br>
<br>
<br>
Comprehensions<br>
--------------<br>
<br>
For the sake of restricting the broadness of this PEP there is no new<br>
syntax for asynchronous comprehensions. This should be considered in a<br>
separate PEP, if there is a strong demand for this feature.<br>
<br>
<br>
Async lambdas<br>
-------------<br>
<br>
Lambda coroutines are not part of this proposal. In this proposal they<br>
would look like ``async lambda(parameters): expression``. Unless there<br>
is a strong demand to have them as part of this proposal, it is<br>
recommended to consider them later in a separate PEP.<br>
<br>
<br>
Performance<br>
===========<br>
<br>
Overall Impact<br>
--------------<br>
<br>
This proposal introduces no observable performance impact. Here is an<br>
output of python's official set of benchmarks [4]_:<br>
<br>
::<br>
<br>
python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe<br>
<br>
[skipped]<br>
<br>
Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:<br>
Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64<br>
x86_64 i386<br>
<br>
Total CPU cores: 8<br>
<br>
### etree_iterparse ###<br>
Min: 0.365359 -> 0.349168: 1.05x faster<br>
Avg: 0.396924 -> 0.379735: 1.05x faster<br>
Significant (t=9.71)<br>
Stddev: 0.01225 -> 0.01277: 1.0423x larger<br>
<br>
The following not significant results are hidden, use -v to show them:<br>
django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle,<br>
fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.<br>
<br>
<br>
Tokenizer modifications<br>
-----------------------<br>
<br>
There is no observable slowdown of parsing python files with the<br>
modified tokenizer: parsing of one 12Mb file<br>
(``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount<br>
of time.<br>
<br>
<br>
async/await<br>
-----------<br>
<br>
The following micro-benchmark was used to determine performance<br>
difference between "async" functions and generators::<br>
<br>
import sys<br>
import time<br>
<br>
def binary(n):<br>
if n <= 0:<br>
return 1<br>
l = yield from binary(n - 1)<br>
r = yield from binary(n - 1)<br>
return l + 1 + r<br>
<br>
async def abinary(n):<br>
if n <= 0:<br>
return 1<br>
l = await abinary(n - 1)<br>
r = await abinary(n - 1)<br>
return l + 1 + r<br>
<br>
def timeit(gen, depth, repeat):<br>
t0 = time.time()<br>
for _ in range(repeat):<br>
list(gen(depth))<br>
t1 = time.time()<br>
print('{}({}) * {}: total {:.3f}s'.format(<br>
gen.__name__, depth, repeat, t1-t0))<br>
<br>
The result is that there is no observable performance difference.<br>
Minimum timing of 3 runs<br>
<br>
::<br>
<br>
abinary(19) * 30: total 12.985s<br>
binary(19) * 30: total 12.953s<br>
<br>
Note that depth of 19 means 1,048,575 calls.<br>
<br>
<br>
Reference Implementation<br>
========================<br>
<br>
The reference implementation can be found here: [3]_.<br>
<br>
List of high-level changes and new protocols<br>
--------------------------------------------<br>
<br>
1. New syntax for defining coroutines: ``async def`` and new ``await``<br>
keyword.<br>
<br>
2. New ``__await__`` method for Future-like objects.<br>
<br>
3. New syntax for asynchronous context managers: ``async with``. And<br>
associated protocol with ``__aenter__`` and ``__aexit__`` methods.<br>
<br>
4. New syntax for asynchronous iteration: ``async for``. And<br>
associated protocol with ``__aiter__``, ``__aexit__`` and new built-<br>
in exception ``StopAsyncIteration``.<br>
<br>
5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,<br>
``Await``.<br>
<br>
6. New functions: ``sys.set_coroutine_wrapper(callback)``,<br>
``sys.get_coroutine_wrapper()``, and ``types.coroutine(gen)``.<br>
<br>
7. New ``CO_COROUTINE`` bit flag for code objects.<br>
<br>
While the list of changes and new things is not short, it is important<br>
to understand, that most users will not use these features directly.<br>
It is intended to be used in frameworks and libraries to provide users<br>
with convenient to use and unambiguous APIs with ``async def``,<br>
``await``, ``async for`` and ``async with`` syntax.<br>
<br>
<br>
Working example<br>
---------------<br>
<br>
All concepts proposed in this PEP are implemented [3]_ and can be<br>
tested.<br>
<br>
::<br>
<br>
import asyncio<br>
<br>
async def echo_server():<br>
print('Serving on localhost:8000')<br>
await asyncio.start_server(handle_connection,<br>
'localhost', 8000)<br>
<br>
async def handle_connection(reader, writer):<br>
print('New connection...')<br>
<br>
while True:<br>
data = await reader.read(8192)<br>
<br>
if not data:<br>
break<br>
<br>
print('Sending {:.10}... back'.format(repr(data)))<br>
writer.write(data)<br>
<br>
loop = asyncio.get_event_loop()<br>
loop.run_until_complete(echo_server())<br>
try:<br>
loop.run_forever()<br>
finally:<br>
loop.close()<br>
<br>
<br>
References<br>
==========<br>
<br>
.. [1] <a href="https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine" target="_blank">https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine</a><br>
<br>
.. [2] <a href="http://wiki.ecmascript.org/doku.php?id=strawman:async_functions" target="_blank">http://wiki.ecmascript.org/doku.php?id=strawman:async_functions</a><br>
<br>
.. [3] <a href="https://github.com/1st1/cpython/tree/await" target="_blank">https://github.com/1st1/cpython/tree/await</a><br>
<br>
.. [4] <a href="https://hg.python.org/benchmarks" target="_blank">https://hg.python.org/benchmarks</a><br>
<br>
.. [5] <a href="https://msdn.microsoft.com/en-us/library/hh191443.aspx" target="_blank">https://msdn.microsoft.com/en-us/library/hh191443.aspx</a><br>
<br>
.. [6] <a href="http://docs.hhvm.com/manual/en/hack.async.php" target="_blank">http://docs.hhvm.com/manual/en/hack.async.php</a><br>
<br>
.. [7] <a href="https://www.dartlang.org/articles/await-async/" target="_blank">https://www.dartlang.org/articles/await-async/</a><br>
<br>
.. [8] <a href="http://docs.scala-lang.org/sips/pending/async.html" target="_blank">http://docs.scala-lang.org/sips/pending/async.html</a><br>
<br>
.. [9] <a href="https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental" target="_blank">https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental</a><br>
<br>
.. [10] <a href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf" target="_blank">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf</a> (PDF)<br>
<br>
<br>
Acknowledgments<br>
===============<br>
<br>
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew<br>
Svetlov, and Łukasz Langa for their initial feedback.<br>
<br>
<br>
Copyright<br>
=========<br>
<br>
This document has been placed in the public domain.<br>
<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/tds333%2Bpydev%40gmail.com" target="_blank">https://mail.python.org/mailman/options/python-dev/tds333%2Bpydev%40gmail.com</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">bye by Wolfgang</div>
</div>