[Python-Dev] async/await in Python; v2
Andrew Svetlov
andrew.svetlov at gmail.com
Thu Apr 23 11:18:51 CEST 2015
On Thu, Apr 23, 2015 at 10:30 AM, Wolfgang Langner
<tds333+pydev at gmail.com> wrote:
> Hi,
>
> most of the time I am a silent reader but in this discussion I must step in.
> I use twisted and async stuff a lot over years followed development of
> asyncio closely.
>
> First it is good to do differentiate async coroutines from generators. So
> everyone can see it and have this in mind
> and don't mix up booth. It is also easier to explain for new users.
> Sometimes generators blows their mind and it takes
> a while to get used to them. Async stuff is even harder.
>
> 1. I am fine with using something special instead of "yield" or "yield from"
> for this. C# "await" is ok.
>
> Everything else suggested complicates the language and makes it harder to
> read.
>
> 2.
> async def f(): is harder to read and something special also it breaks the
> symmetry in front (def indent).
> Also every existing tooling must be changed to support it. Same for def
> async, def f() async:
> I thing a decorator is enough here
> @coroutine
> def f():
> is the best solution to mark something as a coroutine.
>
Sorry, `@coroutine` decorator is part of Python semantics, not Python
syntax. That means Python compiler cannot relay on @coroutine in
parsing.
`away`, `async for` and `async with` are available only inside `async
def`, not inside regular `def`.
>
> 3.
> async with and async for
> Bead idea, we clutter the language even more and it is one more thing every
> newbie could do wrong.
> for x in y:
> result = await f()
> is enough, every 'async' framework lived without it over years.
async for i in iterable:
pass
is not equal for
for fut in iterable:
i = yield from fut
async for is also suitable when size of iterable sequence is unknown
on iteration start.
Let's look, for example, on Redis SCAN command
(http://redis.io/commands/scan) for iterating over redis keys.
It returns bulk of keys and opaque value for next SCAN call.
In current Python it must be written as
cur = 0
while True:
bulk, cur = yield from redis.scan(cur)
for key in bulk:
process_key(key)
if cur == 0:
break
With new syntax iteration looks much more native:
async for key in redis.scan(cur):
process_key(key)
> Same for with statement.
>
> The main use case suggested was for database stuff and this is also where
> most are best with
> defer something to a thread and keep it none async.
>
`async with` is not only for transactions in relation databases.
As well as plain `with` is used not only for databases but for many
other things -- from files to decimal contexts.
As realistic example not related to databases please recall RabbitMQ.
Every message processing may be acknowledged. The native API for that
may be
while True:
async with channel.consume("my queue") as msg:
process_msg(msg)
with acknowledgement on successful message processing only.
Acknowledgement requires network communication and must be
asynchronous operation.
>
> All together it is very late in the development cycle for 3.5 to incorporate
> such a big change.
> Best is to give all this some more time and defer it to 3.6 and some alpha
> releases to experiment with.
>
> Regards,
>
> Wolfgang
>
>
>
> On Tue, Apr 21, 2015 at 7:26 PM, Yury Selivanov <yselivanov.ml at gmail.com>
> wrote:
>>
>> Hi python-dev,
>>
>> I'm moving the discussion from python-ideas to here.
>>
>> The updated version of the PEP should be available shortly
>> at https://www.python.org/dev/peps/pep-0492
>> and is also pasted in this email.
>>
>> Updates:
>>
>> 1. CO_ASYNC flag was renamed to CO_COROUTINE;
>>
>> 2. sys.set_async_wrapper() was renamed to
>> sys.set_coroutine_wrapper();
>>
>> 3. New function: sys.get_coroutine_wrapper();
>>
>> 4. types.async_def() renamed to types.coroutine();
>>
>> 5. New section highlighting differences from
>> PEP 3152.
>>
>> 6. New AST node - AsyncFunctionDef; the proposal
>> now is 100% backwards compatible;
>>
>> 7. A new section clarifying that coroutine-generators
>> are not part of the current proposal;
>>
>> 8. Various small edits/typos fixes.
>>
>>
>> There's is a bug tracker issue to track code review
>> of the reference implementation (Victor Stinner is
>> doing the review): http://bugs.python.org/issue24017
>> While the PEP isn't accepted, we want to make sure
>> that the reference implementation is ready when such
>> a decision will be made.
>>
>>
>> Let's discuss some open questions:
>>
>> 1. Victor raised a question if we should locate
>> coroutine() function from 'types' module to
>> 'functools'.
>>
>> My opinion is that 'types' module is a better
>> place for 'corotine()', since it adjusts the
>> type of the passed generator. 'functools' is
>> about 'partials', 'lru_cache' and 'wraps' kind
>> of things.
>>
>> 2. I propose to disallow using of 'for..in' loops,
>> and builtins like 'list()', 'iter()', 'next()',
>> 'tuple()' etc on coroutines.
>>
>> It's possible by modifying PyObject_GetIter to
>> raise an exception if it receives a coroutine-object.
>>
>> 'yield from' can also be modified to only accept
>> coroutine objects if it is called from a generator
>> with CO_COROUTINE flag.
>>
>> This will further separate coroutines from
>> generators, making it harder to screw something
>> up by an accident.
>>
>> I have a branch of reference implementation
>> https://github.com/1st1/cpython/tree/await_noiter
>> where this is implemented. I did not observe
>> any performance drop.
>>
>> There is just one possible backwards compatibility
>> issue here: there will be an exception if some user
>> of asyncio actually used to iterate over generators
>> decorated with @coroutine. But I can't imagine
>> why would someone do that, and even if they did --
>> it's probably a bug or wrong usage of asyncio.
>>
>>
>> That's it! I'd be happy to hear some feedback!
>>
>> Thanks,
>> Yury
>>
>>
>>
>> PEP: 492
>> Title: Coroutines with async and await syntax
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Yury Selivanov <yselivanov at sprymix.com>
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 09-Apr-2015
>> Python-Version: 3.5
>> Post-History: 17-Apr-2015, 21-Apr-2015
>>
>>
>> Abstract
>> ========
>>
>> This PEP introduces new syntax for coroutines, asynchronous ``with``
>> statements and ``for`` loops. The main motivation behind this proposal
>> is to streamline writing and maintaining asynchronous code, as well as
>> to simplify previously hard to implement code patterns.
>>
>>
>> Rationale and Goals
>> ===================
>>
>> Current Python supports implementing coroutines via generators (PEP
>> 342), further enhanced by the ``yield from`` syntax introduced in PEP
>> 380. This approach has a number of shortcomings:
>>
>> * it is easy to confuse coroutines with regular generators, since they
>> share the same syntax; async libraries often attempt to alleviate
>> this by using decorators (e.g. ``@asyncio.coroutine`` [1]_);
>>
>> * it is not possible to natively define a coroutine which has no
>> ``yield`` or ``yield from`` statements, again requiring the use of
>> decorators to fix potential refactoring issues;
>>
>> * support for asynchronous calls is limited to expressions where
>> ``yield`` is allowed syntactically, limiting the usefulness of
>> syntactic features, such as ``with`` and ``for`` statements.
>>
>> This proposal makes coroutines a native Python language feature, and
>> clearly separates them from generators. This removes
>> generator/coroutine ambiguity, and makes it possible to reliably define
>> coroutines without reliance on a specific library. This also enables
>> linters and IDEs to improve static code analysis and refactoring.
>>
>> Native coroutines and the associated new syntax features make it
>> possible to define context manager and iteration protocols in
>> asynchronous terms. As shown later in this proposal, the new ``async
>> with`` statement lets Python programs perform asynchronous calls when
>> entering and exiting a runtime context, and the new ``async for``
>> statement makes it possible to perform asynchronous calls in iterators.
>>
>>
>> Specification
>> =============
>>
>> This proposal introduces new syntax and semantics to enhance coroutine
>> support in Python, it does not change the internal implementation of
>> coroutines, which are still based on generators.
>>
>> It is strongly suggested that the reader understands how coroutines are
>> implemented in Python (PEP 342 and PEP 380). It is also recommended to
>> read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions).
>>
>> From this point in this document we use the word *coroutine* to refer
>> to functions declared using the new syntax. *generator-based
>> coroutine* is used where necessary to refer to coroutines that are
>> based on generator syntax.
>>
>>
>> New Coroutine Declaration Syntax
>> --------------------------------
>>
>> The following new syntax is used to declare a coroutine::
>>
>> async def read_data(db):
>> pass
>>
>> Key properties of coroutines:
>>
>> * ``async def`` functions are always coroutines, even if they do not
>> contain ``await`` expressions.
>>
>> * It is a ``SyntaxError`` to have ``yield`` or ``yield from``
>> expressions in an ``async`` function.
>>
>> * Internally, a new code object flag - ``CO_COROUTINE`` - is introduced
>> to enable runtime detection of coroutines (and migrating existing
>> code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``
>> flags set.
>>
>> * Regular generators, when called, return a *generator object*;
>> similarly, coroutines return a *coroutine object*.
>>
>> * ``StopIteration`` exceptions are not propagated out of coroutines,
>> and are replaced with a ``RuntimeError``. For regular generators
>> such behavior requires a future import (see PEP 479).
>>
>>
>> types.coroutine()
>> -----------------
>>
>> A new function ``coroutine(gen)`` is added to the ``types`` module. It
>> applies ``CO_COROUTINE`` flag to the passed generator-function's code
>> object, making it to return a *coroutine object* when called.
>>
>> This feature enables an easy upgrade path for existing libraries.
>>
>>
>> Await Expression
>> ----------------
>>
>> The following new ``await`` expression is used to obtain a result of
>> coroutine execution::
>>
>> async def read_data(db):
>> data = await db.fetch('SELECT ...')
>> ...
>>
>> ``await``, similarly to ``yield from``, suspends execution of
>> ``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
>> returns the result data.
>>
>> It uses the ``yield from`` implementation with an extra step of
>> validating its argument. ``await`` only accepts an *awaitable*, which
>> can be one of:
>>
>> * A *coroutine object* returned from a *coroutine* or a generator
>> decorated with ``types.coroutine()``.
>>
>> * An object with an ``__await__`` method returning an iterator.
>>
>> Any ``yield from`` chain of calls ends with a ``yield``. This is a
>> fundamental mechanism of how *Futures* are implemented. Since,
>> internally, coroutines are a special kind of generators, every
>> ``await`` is suspended by a ``yield`` somewhere down the chain of
>> ``await`` calls (please refer to PEP 3156 for a detailed
>> explanation.)
>>
>> To enable this behavior for coroutines, a new magic method called
>> ``__await__`` is added. In asyncio, for instance, to enable Future
>> objects in ``await`` statements, the only change is to add
>> ``__await__ = __iter__`` line to ``asyncio.Future`` class.
>>
>> Objects with ``__await__`` method are called *Future-like* objects in
>> the rest of this PEP.
>>
>> Also, please note that ``__aiter__`` method (see its definition
>> below) cannot be used for this purpose. It is a different protocol,
>> and would be like using ``__iter__`` instead of ``__call__`` for
>> regular callables.
>>
>> It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
>>
>>
>> Asynchronous Context Managers and "async with"
>> ----------------------------------------------
>>
>> An *asynchronous context manager* is a context manager that is able to
>> suspend execution in its *enter* and *exit* methods.
>>
>> To make this possible, a new protocol for asynchronous context managers
>> is proposed. Two new magic methods are added: ``__aenter__`` and
>> ``__aexit__``. Both must return an *awaitable*.
>>
>> An example of an asynchronous context manager::
>>
>> class AsyncContextManager:
>> async def __aenter__(self):
>> await log('entering context')
>>
>> async def __aexit__(self, exc_type, exc, tb):
>> await log('exiting context')
>>
>>
>> New Syntax
>> ''''''''''
>>
>> A new statement for asynchronous context managers is proposed::
>>
>> async with EXPR as VAR:
>> BLOCK
>>
>>
>> which is semantically equivalent to::
>>
>> mgr = (EXPR)
>> aexit = type(mgr).__aexit__
>> aenter = type(mgr).__aenter__(mgr)
>> exc = True
>>
>> try:
>> try:
>> VAR = await aenter
>> BLOCK
>> except:
>> exc = False
>> exit_res = await aexit(mgr, *sys.exc_info())
>> if not exit_res:
>> raise
>>
>> finally:
>> if exc:
>> await aexit(mgr, None, None, None)
>>
>>
>> As with regular ``with`` statements, it is possible to specify multiple
>> context managers in a single ``async with`` statement.
>>
>> It is an error to pass a regular context manager without ``__aenter__``
>> and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
>> to use ``async with`` outside of a coroutine.
>>
>>
>> Example
>> '''''''
>>
>> With asynchronous context managers it is easy to implement proper
>> database transaction managers for coroutines::
>>
>> async def commit(session, data):
>> ...
>>
>> async with session.transaction():
>> ...
>> await session.update(data)
>> ...
>>
>> Code that needs locking also looks lighter::
>>
>> async with lock:
>> ...
>>
>> instead of::
>>
>> with (yield from lock):
>> ...
>>
>>
>> Asynchronous Iterators and "async for"
>> --------------------------------------
>>
>> An *asynchronous iterable* is able to call asynchronous code in its
>> *iter* implementation, and *asynchronous iterator* can call
>> asynchronous code in its *next* method. To support asynchronous
>> iteration:
>>
>> 1. An object must implement an ``__aiter__`` method returning an
>> *awaitable* resulting in an *asynchronous iterator object*.
>>
>> 2. An *asynchronous iterator object* must implement an ``__anext__``
>> method returning an *awaitable*.
>>
>> 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
>> exception.
>>
>> An example of asynchronous iterable::
>>
>> class AsyncIterable:
>> async def __aiter__(self):
>> return self
>>
>> async def __anext__(self):
>> data = await self.fetch_data()
>> if data:
>> return data
>> else:
>> raise StopAsyncIteration
>>
>> async def fetch_data(self):
>> ...
>>
>>
>> New Syntax
>> ''''''''''
>>
>> A new statement for iterating through asynchronous iterators is
>> proposed::
>>
>> async for TARGET in ITER:
>> BLOCK
>> else:
>> BLOCK2
>>
>> which is semantically equivalent to::
>>
>> iter = (ITER)
>> iter = await type(iter).__aiter__(iter)
>> running = True
>> while running:
>> try:
>> TARGET = await type(iter).__anext__(iter)
>> except StopAsyncIteration:
>> running = False
>> else:
>> BLOCK
>> else:
>> BLOCK2
>>
>>
>> It is an error to pass a regular iterable without ``__aiter__`` method
>> to ``async for``. It is a ``SyntaxError`` to use ``async for`` outside
>> of a coroutine.
>>
>> As for with regular ``for`` statement, ``async for`` has an optional
>> ``else`` clause.
>>
>>
>> Example 1
>> '''''''''
>>
>> With asynchronous iteration protocol it is possible to asynchronously
>> buffer data during iteration::
>>
>> async for data in cursor:
>> ...
>>
>> Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
>> of data from a database after every ``N`` iterations.
>>
>> The following code illustrates new asynchronous iteration protocol::
>>
>> class Cursor:
>> def __init__(self):
>> self.buffer = collections.deque()
>>
>> def _prefetch(self):
>> ...
>>
>> async def __aiter__(self):
>> return self
>>
>> async def __anext__(self):
>> if not self.buffer:
>> self.buffer = await self._prefetch()
>> if not self.buffer:
>> raise StopAsyncIteration
>> return self.buffer.popleft()
>>
>> then the ``Cursor`` class can be used as follows::
>>
>> async for row in Cursor():
>> print(row)
>>
>> which would be equivalent to the following code::
>>
>> i = await Cursor().__aiter__()
>> while True:
>> try:
>> row = await i.__anext__()
>> except StopAsyncIteration:
>> break
>> else:
>> print(row)
>>
>>
>> Example 2
>> '''''''''
>>
>> The following is a utility class that transforms a regular iterable to
>> an asynchronous one. While this is not a very useful thing to do, the
>> code illustrates the relationship between regular and asynchronous
>> iterators.
>>
>> ::
>>
>> class AsyncIteratorWrapper:
>> def __init__(self, obj):
>> self._it = iter(obj)
>>
>> async def __aiter__(self):
>> return self
>>
>> async def __anext__(self):
>> try:
>> value = next(self._it)
>> except StopIteration:
>> raise StopAsyncIteration
>> return value
>>
>> async for item in AsyncIteratorWrapper("abc"):
>> print(item)
>>
>>
>> Why StopAsyncIteration?
>> '''''''''''''''''''''''
>>
>> Coroutines are still based on generators internally. So, before PEP
>> 479, there was no fundamental difference between
>>
>> ::
>>
>> def g1():
>> yield from fut
>> return 'spam'
>>
>> and
>>
>> ::
>>
>> def g2():
>> yield from fut
>> raise StopIteration('spam')
>>
>> And since PEP 479 is accepted and enabled by default for coroutines,
>> the following example will have its ``StopIteration`` wrapped into a
>> ``RuntimeError``
>>
>> ::
>>
>> async def a1():
>> await fut
>> raise StopIteration('spam')
>>
>> The only way to tell the outside code that the iteration has ended is
>> to raise something other than ``StopIteration``. Therefore, a new
>> built-in exception class ``StopAsyncIteration`` was added.
>>
>> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
>> raised in coroutines are wrapped in ``RuntimeError``.
>>
>>
>> Debugging Features
>> ------------------
>>
>> One of the most frequent mistakes that people make when using
>> generators as coroutines is forgetting to use ``yield from``::
>>
>> @asyncio.coroutine
>> def useful():
>> asyncio.sleep(1) # this will do noting without 'yield from'
>>
>> For debugging this kind of mistakes there is a special debug mode in
>> asyncio, in which ``@coroutine`` decorator wraps all functions with a
>> special object with a destructor logging a warning. Whenever a wrapped
>> generator gets garbage collected, a detailed logging message is
>> generated with information about where exactly the decorator function
>> was defined, stack trace of where it was collected, etc. Wrapper
>> object also provides a convenient ``__repr__`` function with detailed
>> information about the generator.
>>
>> The only problem is how to enable these debug capabilities. Since
>> debug facilities should be a no-op in production mode, ``@coroutine``
>> decorator makes the decision of whether to wrap or not to wrap based on
>> an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is
>> possible to run asyncio programs with asyncio's own functions
>> instrumented. ``EventLoop.set_debug``, a different debug facility, has
>> no impact on ``@coroutine`` decorator's behavior.
>>
>> With this proposal, coroutines is a native, distinct from generators,
>> concept. New methods ``set_coroutine_wrapper`` and
>> ``get_coroutine_wrapper`` are added to the ``sys`` module, with which
>> frameworks can provide advanced debugging facilities.
>>
>> It is also important to make coroutines as fast and efficient as
>> possible, therefore there are no debug features enabled by default.
>>
>> Example::
>>
>> async def debug_me():
>> await asyncio.sleep(1)
>>
>> def async_debug_wrap(generator):
>> return asyncio.AsyncDebugWrapper(generator)
>>
>> sys.set_coroutine_wrapper(async_debug_wrap)
>>
>> debug_me() # <- this line will likely GC the coroutine object and
>> # trigger AsyncDebugWrapper's code.
>>
>> assert isinstance(debug_me(), AsyncDebugWrapper)
>>
>> sys.set_coroutine_wrapper(None) # <- this unsets any
>> # previously set wrapper
>> assert not isinstance(debug_me(), AsyncDebugWrapper)
>>
>> If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper
>> replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)``
>> unsets the wrapper.
>>
>>
>> Glossary
>> ========
>>
>> :Coroutine:
>> A coroutine function, or just "coroutine", is declared with ``async
>> def``. It uses ``await`` and ``return value``; see `New Coroutine
>> Declaration Syntax`_ for details.
>>
>> :Coroutine object:
>> Returned from a coroutine function. See `Await Expression`_ for
>> details.
>>
>> :Future-like object:
>> An object with an ``__await__`` method. Can be consumed by an
>> ``await`` expression in a coroutine. A coroutine waiting for a
>> Future-like object is suspended until the Future-like object's
>> ``__await__`` completes, and returns the result. See `Await
>> Expression`_ for details.
>>
>> :Awaitable:
>> A *Future-like* object or a *coroutine object*. See `Await
>> Expression`_ for details.
>>
>> :Generator-based coroutine:
>> Coroutines based in generator syntax. Most common example is
>> ``@asyncio.coroutine``.
>>
>> :Asynchronous context manager:
>> An asynchronous context manager has ``__aenter__`` and ``__aexit__``
>> methods and can be used with ``async with``. See `Asynchronous
>> Context Managers and "async with"`_ for details.
>>
>> :Asynchronous iterable:
>> An object with an ``__aiter__`` method, which must return an
>> *asynchronous iterator* object. Can be used with ``async for``.
>> See `Asynchronous Iterators and "async for"`_ for details.
>>
>> :Asynchronous iterator:
>> An asynchronous iterator has an ``__anext__`` method. See
>> `Asynchronous Iterators and "async for"`_ for details.
>>
>>
>> List of functions and methods
>> =============================
>>
>> ================= =================================== =================
>> Method Can contain Can't contain
>> ================= =================================== =================
>> async def func await, return value yield, yield from
>> async def __a*__ await, return value yield, yield from
>> def __a*__ return awaitable await
>> def __await__ yield, yield from, return iterable await
>> generator yield, yield from, return value await
>> ================= =================================== =================
>>
>> Where:
>>
>> * "async def func": coroutine;
>>
>> * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
>> ``__aexit__`` defined with the ``async`` keyword;
>>
>> * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
>> ``__aexit__`` defined without the ``async`` keyword, must return an
>> *awaitable*;
>>
>> * "def __await__": ``__await__`` method to implement *Future-like*
>> objects;
>>
>> * generator: a "regular" generator, function defined with ``def`` and
>> which contains a least one ``yield`` or ``yield from`` expression.
>>
>>
>> Transition Plan
>> ===============
>>
>> To avoid backwards compatibility issues with ``async`` and ``await``
>> keywords, it was decided to modify ``tokenizer.c`` in such a way, that
>> it:
>>
>> * recognizes ``async def`` name tokens combination (start of a
>> coroutine);
>>
>> * keeps track of regular functions and coroutines;
>>
>> * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
>> ``AWAIT`` when in the process of yielding tokens for coroutines.
>>
>> This approach allows for seamless combination of new syntax features
>> (all of them available only in ``async`` functions) with any existing
>> code.
>>
>> An example of having "async def" and "async" attribute in one piece of
>> code::
>>
>> class Spam:
>> async = 42
>>
>> async def ham():
>> print(getattr(Spam, 'async'))
>>
>> # The coroutine can be executed and will print '42'
>>
>>
>> Backwards Compatibility
>> -----------------------
>>
>> This proposal preserves 100% backwards compatibility.
>>
>>
>> Grammar Updates
>> ---------------
>>
>> Grammar changes are also fairly minimal::
>>
>> await_expr: AWAIT test
>> await_stmt: await_expr
>>
>> decorated: decorators (classdef | funcdef | async_funcdef)
>> async_funcdef: ASYNC funcdef
>>
>> async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
>>
>> compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt |
>> with_stmt | funcdef | classdef | decorated |
>> async_stmt)
>>
>> atom: ('(' [yield_expr|await_expr|testlist_comp] ')' |
>> '[' [testlist_comp] ']' |
>> '{' [dictorsetmaker] '}' |
>> NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False’)
>>
>> expr_stmt: testlist_star_expr
>> (augassign (yield_expr|await_expr|testlist) |
>> ('=' (yield_expr|await_expr|testlist_star_expr))*)
>>
>>
>> Transition Period Shortcomings
>> ------------------------------
>>
>> There is just one.
>>
>> Until ``async`` and ``await`` are not proper keywords, it is not
>> possible (or at least very hard) to fix ``tokenizer.c`` to recognize
>> them on the **same line** with ``def`` keyword::
>>
>> # async and await will always be parsed as variables
>>
>> async def outer(): # 1
>> def nested(a=(await fut)):
>> pass
>>
>> async def foo(): return (await fut) # 2
>>
>> Since ``await`` and ``async`` in such cases are parsed as ``NAME``
>> tokens, a ``SyntaxError`` will be raised.
>>
>> To workaround these issues, the above examples can be easily rewritten
>> to a more readable form::
>>
>> async def outer(): # 1
>> a_default = await fut
>> def nested(a=a_default):
>> pass
>>
>> async def foo(): # 2
>> return (await fut)
>>
>> This limitation will go away as soon as ``async`` and ``await`` ate
>> proper keywords. Or if it's decided to use a future import for this
>> PEP.
>>
>>
>> Deprecation Plans
>> -----------------
>>
>> ``async`` and ``await`` names will be softly deprecated in CPython 3.5
>> and 3.6. In 3.7 we will transform them to proper keywords. Making
>> ``async`` and ``await`` proper keywords before 3.7 might make it harder
>> for people to port their code to Python 3.
>>
>>
>> asyncio
>> -------
>>
>> ``asyncio`` module was adapted and tested to work with coroutines and
>> new statements. Backwards compatibility is 100% preserved.
>>
>> The required changes are mainly:
>>
>> 1. Modify ``@asyncio.coroutine`` decorator to use new
>> ``types.coroutine()`` function.
>>
>> 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
>>
>> 3. Add ``ensure_task()`` as an alias for ``async()`` function.
>> Deprecate ``async()`` function.
>>
>>
>> Design Considerations
>> =====================
>>
>> PEP 3152
>> --------
>>
>> PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
>> (called "cofunctions"). Some key points:
>>
>> 1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is
>> always a generator, even if there is no ``cocall`` expressions
>> inside it. Maps to ``async def`` in this proposal.
>>
>> 2. A new keyword ``cocall`` to call a *cofunction*. Can only be used
>> inside a *cofunction*. Maps to ``await`` in this proposal (with
>> some differences, see below.)
>>
>> 3. It is not possible to call a *cofunction* without a ``cocall``
>> keyword.
>>
>> 4. ``cocall`` grammatically requires parentheses after it::
>>
>> atom: cocall | <existing alternatives for atom>
>> cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
>> cotrailer: '[' subscriptlist ']' | '.' NAME
>>
>> 5. ``cocall f(*args, **kwds)`` is semantically equivalent to
>> ``yield from f.__cocall__(*args, **kwds)``.
>>
>> Differences from this proposal:
>>
>> 1. There is no equivalent of ``__cocall__`` in this PEP, which is
>> called and its result is passed to ``yield from`` in the ``cocall``
>> expression. ``await`` keyword expects an *awaitable* object,
>> validates the type, and executes ``yield from`` on it. Although,
>> ``__await__`` method is similar to ``__cocall__``, but is only used
>> to define *Future-like* objects.
>>
>> 2. ``await`` is defined in almost the same way as ``yield from`` in the
>> grammar (it is later enforced that ``await`` can only be inside
>> ``async def``). It is possible to simply write ``await future``,
>> whereas ``cocall`` always requires parentheses.
>>
>> 3. To make asyncio work with PEP 3152 it would be required to modify
>> ``@asyncio.coroutine`` decorator to wrap all functions in an object
>> with a ``__cocall__`` method. To call *cofunctions* from existing
>> generator-based coroutines it would be required to use ``costart``
>> built-in. In this proposal ``@asyncio.coroutine`` simply sets
>> ``CO_COROUTINE`` on the wrapped function's code object and
>> everything works automatically.
>>
>> 4. Since it is impossible to call a *cofunction* without a ``cocall``
>> keyword, it automatically prevents the common mistake of forgetting
>> to use ``yield from`` on generator-based coroutines. This proposal
>> addresses this problem with a different approach, see `Debugging
>> Features`_.
>>
>> 5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine
>> is that if is decided to implement coroutine-generators --
>> coroutines with ``yield`` or ``async yield`` expressions -- we
>> wouldn't need a ``cocall`` keyword to call them. So we'll end up
>> having ``__cocall__`` and no ``__call__`` for regular coroutines,
>> and having ``__call__`` and no ``__cocall__`` for coroutine-
>> generators.
>>
>> 6. There are no equivalents of ``async for`` and ``async with`` in PEP
>> 3152.
>>
>>
>> Coroutine-generators
>> --------------------
>>
>> With ``async for`` keyword it is desirable to have a concept of a
>> *coroutine-generator* -- a coroutine with ``yield`` and ``yield from``
>> expressions. To avoid any ambiguity with regular generators, we would
>> likely require to have an ``async`` keyword before ``yield``, and
>> ``async yield from`` would raise a ``StopAsyncIteration`` exception.
>>
>> While it is possible to implement coroutine-generators, we believe that
>> they are out of scope of this proposal. It is an advanced concept that
>> should be carefully considered and balanced, with a non-trivial changes
>> in the implementation of current generator objects. This is a matter
>> for a separate PEP.
>>
>>
>> No implicit wrapping in Futures
>> -------------------------------
>>
>> There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A
>> key difference is that JavaScript "async functions" always return a
>> Promise. While this approach has some advantages, it also implies that
>> a new Promise object is created on each "async function" invocation.
>>
>> We could implement a similar functionality in Python, by wrapping all
>> coroutines in a Future object, but this has the following
>> disadvantages:
>>
>> 1. Performance. A new Future object would be instantiated on each
>> coroutine call. Moreover, this makes implementation of ``await``
>> expressions slower (disabling optimizations of ``yield from``).
>>
>> 2. A new built-in ``Future`` object would need to be added.
>>
>> 3. Coming up with a generic ``Future`` interface that is usable for any
>> use case in any framework is a very hard to solve problem.
>>
>> 4. It is not a feature that is used frequently, when most of the code
>> is coroutines.
>>
>>
>> Why "async" and "await" keywords
>> --------------------------------
>>
>> async/await is not a new concept in programming languages:
>>
>> * C# has it since long time ago [5]_;
>>
>> * proposal to add async/await in ECMAScript 7 [2]_;
>> see also Traceur project [9]_;
>>
>> * Facebook's Hack/HHVM [6]_;
>>
>> * Google's Dart language [7]_;
>>
>> * Scala [8]_;
>>
>> * proposal to add async/await to C++ [10]_;
>>
>> * and many other less popular languages.
>>
>> This is a huge benefit, as some users already have experience with
>> async/await, and because it makes working with many languages in one
>> project easier (Python with ECMAScript 7 for instance).
>>
>>
>> Why "__aiter__" is a coroutine
>> ------------------------------
>>
>> In principle, ``__aiter__`` could be a regular function. There are
>> several good reasons to make it a coroutine:
>>
>> * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``
>> methods are coroutines, users would often make a mistake defining it
>> as ``async`` anyways;
>>
>> * there might be a need to run some asynchronous operations in
>> ``__aiter__``, for instance to prepare DB queries or do some file
>> operation.
>>
>>
>> Importance of "async" keyword
>> -----------------------------
>>
>> While it is possible to just implement ``await`` expression and treat
>> all functions with at least one ``await`` as coroutines, this approach
>> makes APIs design, code refactoring and its long time support harder.
>>
>> Let's pretend that Python only has ``await`` keyword::
>>
>> def useful():
>> ...
>> await log(...)
>> ...
>>
>> def important():
>> await useful()
>>
>> If ``useful()`` function is refactored and someone removes all
>> ``await`` expressions from it, it would become a regular python
>> function, and all code that depends on it, including ``important()``
>> would be broken. To mitigate this issue a decorator similar to
>> ``@asyncio.coroutine`` has to be introduced.
>>
>>
>> Why "async def"
>> ---------------
>>
>> For some people bare ``async name(): pass`` syntax might look more
>> appealing than ``async def name(): pass``. It is certainly easier to
>> type. But on the other hand, it breaks the symmetry between ``async
>> def``, ``async with`` and ``async for``, where ``async`` is a modifier,
>> stating that the statement is asynchronous. It is also more consistent
>> with the existing grammar.
>>
>>
>> Why not a __future__ import
>> ---------------------------
>>
>> ``__future__`` imports are inconvenient and easy to forget to add.
>> Also, they are enabled for the whole source file. Consider that there
>> is a big project with a popular module named "async.py". With future
>> imports it is required to either import it using ``__import__()`` or
>> ``importlib.import_module()`` calls, or to rename the module. The
>> proposed approach makes it possible to continue using old code and
>> modules without a hassle, while coming up with a migration plan for
>> future python versions.
>>
>>
>> Why magic methods start with "a"
>> --------------------------------
>>
>> New asynchronous magic methods ``__aiter__``, ``__anext__``,
>> ``__aenter__``, and ``__aexit__`` all start with the same prefix "a".
>> An alternative proposal is to use "async" prefix, so that ``__aiter__``
>> becomes ``__async_iter__``. However, to align new magic methods with
>> the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided
>> to use a shorter version.
>>
>>
>> Why not reuse existing magic names
>> ----------------------------------
>>
>> An alternative idea about new asynchronous iterators and context
>> managers was to reuse existing magic methods, by adding an ``async``
>> keyword to their declarations::
>>
>> class CM:
>> async def __enter__(self): # instead of __aenter__
>> ...
>>
>> This approach has the following downsides:
>>
>> * it would not be possible to create an object that works in both
>> ``with`` and ``async with`` statements;
>>
>> * it would look confusing and would require some implicit magic behind
>> the scenes in the interpreter;
>>
>> * one of the main points of this proposal is to make coroutines as
>> simple and foolproof as possible.
>>
>>
>> Comprehensions
>> --------------
>>
>> For the sake of restricting the broadness of this PEP there is no new
>> syntax for asynchronous comprehensions. This should be considered in a
>> separate PEP, if there is a strong demand for this feature.
>>
>>
>> Async lambdas
>> -------------
>>
>> Lambda coroutines are not part of this proposal. In this proposal they
>> would look like ``async lambda(parameters): expression``. Unless there
>> is a strong demand to have them as part of this proposal, it is
>> recommended to consider them later in a separate PEP.
>>
>>
>> Performance
>> ===========
>>
>> Overall Impact
>> --------------
>>
>> This proposal introduces no observable performance impact. Here is an
>> output of python's official set of benchmarks [4]_:
>>
>> ::
>>
>> python perf.py -r -b default ../cpython/python.exe
>> ../cpython-aw/python.exe
>>
>> [skipped]
>>
>> Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
>> Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
>> x86_64 i386
>>
>> Total CPU cores: 8
>>
>> ### etree_iterparse ###
>> Min: 0.365359 -> 0.349168: 1.05x faster
>> Avg: 0.396924 -> 0.379735: 1.05x faster
>> Significant (t=9.71)
>> Stddev: 0.01225 -> 0.01277: 1.0423x larger
>>
>> The following not significant results are hidden, use -v to show them:
>> django_v2, 2to3, etree_generate, etree_parse, etree_process,
>> fastpickle,
>> fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
>>
>>
>> Tokenizer modifications
>> -----------------------
>>
>> There is no observable slowdown of parsing python files with the
>> modified tokenizer: parsing of one 12Mb file
>> (``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount
>> of time.
>>
>>
>> async/await
>> -----------
>>
>> The following micro-benchmark was used to determine performance
>> difference between "async" functions and generators::
>>
>> import sys
>> import time
>>
>> def binary(n):
>> if n <= 0:
>> return 1
>> l = yield from binary(n - 1)
>> r = yield from binary(n - 1)
>> return l + 1 + r
>>
>> async def abinary(n):
>> if n <= 0:
>> return 1
>> l = await abinary(n - 1)
>> r = await abinary(n - 1)
>> return l + 1 + r
>>
>> def timeit(gen, depth, repeat):
>> t0 = time.time()
>> for _ in range(repeat):
>> list(gen(depth))
>> t1 = time.time()
>> print('{}({}) * {}: total {:.3f}s'.format(
>> gen.__name__, depth, repeat, t1-t0))
>>
>> The result is that there is no observable performance difference.
>> Minimum timing of 3 runs
>>
>> ::
>>
>> abinary(19) * 30: total 12.985s
>> binary(19) * 30: total 12.953s
>>
>> Note that depth of 19 means 1,048,575 calls.
>>
>>
>> Reference Implementation
>> ========================
>>
>> The reference implementation can be found here: [3]_.
>>
>> List of high-level changes and new protocols
>> --------------------------------------------
>>
>> 1. New syntax for defining coroutines: ``async def`` and new ``await``
>> keyword.
>>
>> 2. New ``__await__`` method for Future-like objects.
>>
>> 3. New syntax for asynchronous context managers: ``async with``. And
>> associated protocol with ``__aenter__`` and ``__aexit__`` methods.
>>
>> 4. New syntax for asynchronous iteration: ``async for``. And
>> associated protocol with ``__aiter__``, ``__aexit__`` and new built-
>> in exception ``StopAsyncIteration``.
>>
>> 5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,
>> ``Await``.
>>
>> 6. New functions: ``sys.set_coroutine_wrapper(callback)``,
>> ``sys.get_coroutine_wrapper()``, and ``types.coroutine(gen)``.
>>
>> 7. New ``CO_COROUTINE`` bit flag for code objects.
>>
>> While the list of changes and new things is not short, it is important
>> to understand, that most users will not use these features directly.
>> It is intended to be used in frameworks and libraries to provide users
>> with convenient to use and unambiguous APIs with ``async def``,
>> ``await``, ``async for`` and ``async with`` syntax.
>>
>>
>> Working example
>> ---------------
>>
>> All concepts proposed in this PEP are implemented [3]_ and can be
>> tested.
>>
>> ::
>>
>> import asyncio
>>
>> async def echo_server():
>> print('Serving on localhost:8000')
>> await asyncio.start_server(handle_connection,
>> 'localhost', 8000)
>>
>> async def handle_connection(reader, writer):
>> print('New connection...')
>>
>> while True:
>> data = await reader.read(8192)
>>
>> if not data:
>> break
>>
>> print('Sending {:.10}... back'.format(repr(data)))
>> writer.write(data)
>>
>> loop = asyncio.get_event_loop()
>> loop.run_until_complete(echo_server())
>> try:
>> loop.run_forever()
>> finally:
>> loop.close()
>>
>>
>> References
>> ==========
>>
>> .. [1]
>> https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
>>
>> .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
>>
>> .. [3] https://github.com/1st1/cpython/tree/await
>>
>> .. [4] https://hg.python.org/benchmarks
>>
>> .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
>>
>> .. [6] http://docs.hhvm.com/manual/en/hack.async.php
>>
>> .. [7] https://www.dartlang.org/articles/await-async/
>>
>> .. [8] http://docs.scala-lang.org/sips/pending/async.html
>>
>> .. [9]
>> https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental
>>
>> .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf
>> (PDF)
>>
>>
>> Acknowledgments
>> ===============
>>
>> I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
>> Svetlov, and Łukasz Langa for their initial feedback.
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed in the public domain.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/tds333%2Bpydev%40gmail.com
>
>
>
>
> --
> bye by Wolfgang
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>
--
Thanks,
Andrew Svetlov
More information about the Python-Dev
mailing list