[Python-ideas] async/await in Python

Ludovic Gasc gmludo at gmail.com
Sat Apr 18 00:38:21 CEST 2015


As a lambda end-user of AsyncIO, it seems promising.
As I said to Andrew Svetlov during PyCON 2015, I really prefer this new 
syntax.

For a new comer, it will be more easier to handle because you don't need to 
explain the first time what is a decorator, a yield, and a yield from in 
the same time that Futures, Coroutines and Tasks.
It's a (small) distraction about Python internal plumber about AsyncIO 
implementation, instead to concentrate on async business logic 
implementation.

Nevertheless, the only small fear I have is a potential "schism" in AsyncIO 
community like we have with Python 2/3.
For end-products source code, no problems: For new projects you can use the 
new syntax, or migrate easily old projects, thanks to compatibility plan.
But for open source libraries, you must keep old syntax is you want Python 
3.3+ compatibility.

I know it's a small change compare to Python 3 changes, and AsyncIO 
community is enough small for now and almost AsyncIO users are early 
adopters for now: It means it will be more fluid for the shift.

Because async keyword integration will change the end-user source code, I 
vote for an integration in Python 3.5 (not 3.6): it will reduce the waiting 
time for the end-users to use that.

In an ideal world, it should be useful if we could also use that with 
Python 3.3 and 3.4 to help migration, because mainstream Linux 
distributions for production (Debian Jessie, Ubuntu 14.04 and CentOS/RHEL 
7) have packages for Python 3.4, not Python 3.5, and it will change only 
for the new versions of theses distributions.
Nevertheless, I understand the technical and political reasons that it's 
impossible to do that: It's an internal change of CPython, can't be added 
via a third library.

Hopefully, it's really easy to compile CPython, an AsyncIO early adopter 
can't be afraid to do that ;-)
--
Ludovic Gasc (GMLudo)
http://www.gmludo.eu/

On Friday, April 17, 2015 at 8:59:29 PM UTC+2, Yury Selivanov wrote:
>
> Hello python-ideas,
>
> Here's my proposal to add async/await in Python.
>
> I believe that PEPs 380 and 3156 were a major breakthrough for Python 3,
> and I hope that this PEP can be another big step.  Who knows, maybe it
> will be one of the reasons to drive people's interest towards Python 3.
>
>
> PEP: XXX
> Title: Coroutines with async and await syntax
> Version: $Revision$
> Last-Modified: $Date$
> Author: Yury Selivanov <yseli... at sprymix.com <javascript:>>
> Discussions-To: Python-Dev <pytho... at python.org <javascript:>>
> Python-Version: 3.5
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 09-Apr-2015
> Post-History:
> Resolution:
>
>
> Abstract
> ========
>
> This PEP introduces new syntax for coroutines, asynchronous ``with``
> statements and ``for`` loops.  The main motivation behind this proposal 
> is to
> streamline writing and maintaining asynchronous code, as well as to 
> simplify
> previously hard to implement code patterns.
>
>
> Rationale and Goals
> ===================
>
> Current Python supports implementing coroutines via generators (PEP 342),
> further enhanced by the ``yield from`` syntax introduced in PEP 380.
> This approach has a number of shortcomings:
>
> * it is easy to confuse coroutines with regular generators, since they 
> share
>    the same syntax; async libraries often attempt to alleviate this by 
> using
>    decorators (e.g. ``@asyncio.coroutine`` [1]_);
>
> * it is not possible to natively define a coroutine which has no ``yield``
>    or  ``yield from`` statements, again requiring the use of decorators to
>    fix potential refactoring issues;
>
> * support for asynchronous calls is limited to expressions where 
> ``yield`` is
>    allowed syntactically, limiting the usefulness of syntactic features, 
> such
>    as ``with`` and ``for`` statements.
>
> This proposal makes coroutines a native Python language feature, and 
> clearly
> separates them from generators.  This removes generator/coroutine 
> ambiguity,
> and makes it possible to reliably define coroutines without reliance on a
> specific library.  This also enables linters and IDEs to improve static 
> code
> analysis and refactoring.
>
> Native coroutines and the associated new syntax features make it possible
> to define context manager and iteration protocols in asynchronous terms.
> As shown later in this proposal, the new ``async with`` statement lets 
> Python
> programs perform asynchronous calls when entering and exiting a runtime
> context, and the new ``async for`` statement makes it possible to perform
> asynchronous calls in iterators.
>
>
> Specification
> =============
>
> This proposal introduces new syntax and semantics to enhance coroutine 
> support
> in Python, it does not change the internal implementation of coroutines, 
> which
> are still based on generators.
>
> It is strongly suggested that the reader understands how coroutines are
> implemented in Python (PEP 342 and PEP 380).  It is also recommended to 
> read
> PEP 3156 (asyncio framework).
>
>  From this point in this document we use the word *coroutine* to refer to
> functions declared using the new syntax.  *generator-based coroutine* is 
> used
> where necessary to refer to coroutines that are based on generator syntax.
>
>
> New Coroutine Declaration Syntax
> --------------------------------
>
> The following new syntax is used to declare a coroutine::
>
>      async def read_data(db):
>          pass
>
> Key properties of coroutines:
>
> * Coroutines are always generators, even if they do not contain ``await``
>    expressions.
>
> * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions 
> in
>    an ``async`` function.
>
> * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to 
> enable
>    runtime detection of coroutines (and migrating existing code).
>    All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set.
>
> * Regular generators, when called, return a *generator object*; similarly,
>    coroutines return a *coroutine object*.
>
> * ``StopIteration`` exceptions are not propagated out of coroutines, and 
> are
>    replaced with a ``RuntimeError``.  For regular generators such behavior
>    requires a future import (see PEP 479).
>
>
> types.async_def()
> -----------------
>
> A new function ``async_def(gen)`` is added to the ``types`` module. It
> applies ``CO_ASYNC`` flag to the passed generator's code object, so that it
> returns a *coroutine object* when called.
>
> This feature enables an easy upgrade path for existing libraries.
>
>
> Await Expression
> ----------------
>
> The following new ``await`` expression is used to obtain a result of 
> coroutine
> execution::
>
>      async def read_data(db):
>          data = await db.fetch('SELECT ...')
>          ...
>
> ``await``, similarly to ``yield from``, suspends execution of ``read_data``
> coroutine until ``db.fetch`` *awaitable* completes and returns the result
> data.
>
> It uses the ``yield from`` implementation with an extra step of 
> validating its
> argument.  ``await`` only accepts an *awaitable*, which can be one of:
>
> * A *coroutine object* returned from a coroutine or a generator 
> decorated with
>    ``types.async_def()``.
>
> * An object with an ``__await__`` method returning an iterator.
>
>    Any ``yield from`` chain of calls ends with a ``yield``.  This is a
>    fundamental mechanism of how *Futures* are implemented.  Since, 
> internally,
>    coroutines are a special kind of generators, every ``await`` is 
> suspended by
>    a ``yield`` somewhere down the chain of ``await`` calls (please refer 
> to PEP
>    3156 for a detailed explanation.)
>
>    To enable this behavior for coroutines, a new magic method called
>    ``__await__`` is added.  In asyncio, for instance, to enable Future 
> objects
>    in ``await`` statements, the only change is to add ``__await__ = 
> __iter__``
>    line to ``asyncio.Future`` class.
>
>    Objects with ``__await__`` method are called *Future-like* objects in 
> the
>    rest of this PEP.
>
>    Also, please note that ``__aiter__`` method (see its definition 
> below) cannot
>    be used for this purpose.  It is a different protocol, and would be like
>    using ``__iter__`` instead of ``__call__`` for regular callables.
>
> It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
>
>
> Asynchronous Context Managers and "async with"
> ----------------------------------------------
>
> An *asynchronous context manager* is a context manager that is able to 
> suspend
> execution in its *enter* and *exit* methods.
>
> To make this possible, a new protocol for asynchronous context managers is
> proposed.  Two new magic methods are added: ``__aenter__`` and 
> ``__aexit__``.
> Both must return an *awaitable*.
>
> An example of an asynchronous context manager::
>
>      class AsyncContextManager:
>          async def __aenter__(self):
>              await log('entering context')
>
>          async def __aexit__(self, exc_type, exc, tb):
>              await log('exiting context')
>
>
> New Syntax
> ''''''''''
>
> A new statement for asynchronous context managers is proposed::
>
>      async with EXPR as VAR:
>          BLOCK
>
>
> which is semantically equivalent to::
>
>      mgr = (EXPR)
>      aexit = type(mgr).__aexit__
>      aenter = type(mgr).__aenter__(mgr)
>      exc = True
>
>      try:
>          try:
>              VAR = await aenter
>              BLOCK
>          except:
>              exc = False
>              exit_res = await aexit(mgr, *sys.exc_info())
>              if not exit_res:
>                  raise
>
>      finally:
>          if exc:
>              await aexit(mgr, None, None, None)
>
>
> As with regular ``with`` statements, it is possible to specify multiple 
> context
> managers in a single ``async with`` statement.
>
> It is an error to pass a regular context manager without ``__aenter__`` and
> ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError`` to use
> ``async with`` outside of a coroutine.
>
>
> Example
> '''''''
>
> With asynchronous context managers it is easy to implement proper database
> transaction managers for coroutines::
>
>      async def commit(session, data):
>          ...
>
>          async with session.transaction():
>              ...
>              await session.update(data)
>              ...
>
> Code that needs locking also looks lighter::
>
>      async with lock:
>          ...
>
> instead of::
>
>      with (yield from lock):
>          ...
>
>
> Asynchronous Iterators and "async for"
> --------------------------------------
>
> An *asynchronous iterable* is able to call asynchronous code in its *iter*
> implementation, and *asynchronous iterator* can call asynchronous code 
> in its
> *next* method.  To support asynchronous iteration:
>
> 1. An object must implement an  ``__aiter__`` method returning an 
> *awaitable*
>     resulting in an *asynchronous iterator object*.
>
> 2. An *asynchronous iterator object* must implement an ``__anext__`` method
>     returning an *awaitable*.
>
> 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration``
>     exception.
>
> An example of asynchronous iterable::
>
>      class AsyncIterable:
>          async def __aiter__(self):
>              return self
>
>          async def __anext__(self):
>              data = await self.fetch_data()
>              if data:
>                  return data
>              else:
>                  raise StopAsyncIteration
>
>          async def fetch_data(self):
>              ...
>
>
> New Syntax
> ''''''''''
>
> A new statement for iterating through asynchronous iterators is proposed::
>
>      async for TARGET in ITER:
>          BLOCK
>      else:
>          BLOCK2
>
> which is semantically equivalent to::
>
>      iter = (ITER)
>      iter = await type(iter).__aiter__(iter)
>      running = True
>      while running:
>          try:
>              TARGET = await type(iter).__anext__(iter)
>          except StopAsyncIteration:
>              running = False
>          else:
>              BLOCK
>      else:
>          BLOCK2
>
>
> It is an error to pass a regular iterable without ``__aiter__`` method to
> ``async for``.  It is a ``SyntaxError`` to use ``async for`` outside of a
> coroutine.
>
> As for with regular ``for`` statement, ``async for`` has an optional 
> ``else``
> clause.
>
>
> Example 1
> '''''''''
>
> With asynchronous iteration protocol it is possible to asynchronously 
> buffer
> data during iteration::
>
>      async for data in cursor:
>          ...
>
> Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
> of data from a database after every ``N`` iterations.
>
> The following code illustrates new asynchronous iteration protocol::
>
>      class Cursor:
>          def __init__(self):
>              self.buffer = collections.deque()
>
>          def _prefetch(self):
>              ...
>
>          async def __aiter__(self):
>              return self
>
>          async def __anext__(self):
>              if not self.buffer:
>                  self.buffer = await self._prefetch()
>                  if not self.buffer:
>                      raise StopAsyncIteration
>              return self.buffer.popleft()
>
> then the ``Cursor`` class can be used as follows::
>
>      async for row in Cursor():
>          print(row)
>
> which would be equivalent to the following code::
>
>      i = await Cursor().__aiter__()
>      while True:
>          try:
>              row = await i.__anext__()
>          except StopAsyncIteration:
>              break
>          else:
>              print(row)
>
>
> Example 2
> '''''''''
>
> The following is a utility class that transforms a regular iterable to an
> asynchronous one.  While this is not a very useful thing to do, the code
> illustrates the relationship between regular and asynchronous iterators.
>
> ::
>
>      class AsyncIteratorWrapper:
>          def __init__(self, obj):
>              self._it = iter(obj)
>
>          async def __aiter__(self):
>              return self
>
>          async def __anext__(self):
>              try:
>                  value = next(self._it)
>              except StopIteration:
>                  raise StopAsyncIteration
>              return value
>
>      data = "abc"
>      it = AsyncIteratorWrapper("abc")
>      async for item in it:
>          print(it)
>
>
> Why StopAsyncIteration?
> '''''''''''''''''''''''
>
> Coroutines are still based on generators internally.  So, before PEP 
> 479, there
> was no fundamental difference between
>
> ::
>
>      def g1():
>          yield from fut
>          return 'spam'
>
> and
>
> ::
>
>      def g2():
>          yield from fut
>          raise StopIteration('spam')
>
> And since PEP 479 is accepted and enabled by default for coroutines, the
> following example will have its ``StopIteration`` wrapped into a
> ``RuntimeError``
>
> ::
>
>      async def a1():
>          await fut
>          raise StopIteration('spam')
>
> The only way to tell the outside code that the iteration has ended is to 
> raise
> something other than ``StopIteration``.  Therefore, a new built-in 
> exception
> class ``StopAsyncIteration`` was added.
>
> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions 
> raised
> in coroutines are wrapped in ``RuntimeError``.
>
>
> Debugging Features
> ------------------
>
> One of the most frequent mistakes that people make when using generators as
> coroutines is forgetting to use ``yield from``::
>
>      @asyncio.coroutine
>      def useful():
>          asyncio.sleep(1) # this will do noting without 'yield from'
>
> For debugging this kind of mistakes there is a special debug mode in 
> asyncio,
> in which ``@coroutine`` decorator wraps all functions with a special object
> with a destructor logging a warning.  Whenever a wrapped generator gets 
> garbage
> collected, a detailed logging message is generated with information 
> about where
> exactly the decorator function was defined, stack trace of where it was
> collected, etc.  Wrapper object also provides a convenient ``__repr__``
> function with detailed information about the generator.
>
> The only problem is how to enable these debug capabilities.  Since debug
> facilities should be a no-op in production mode, ``@coroutine`` 
> decorator makes
> the decision of whether to wrap or not to wrap based on an OS environment
> variable ``PYTHONASYNCIODEBUG``.  This way it is possible to run asyncio
> programs with asyncio's own functions instrumented. 
> ``EventLoop.set_debug``, a
> different debug facility, has no impact on ``@coroutine`` decorator's 
> behavior.
>
> With this proposal, coroutines is a native, distinct from generators,
> concept.  A new method ``set_async_wrapper`` is added to the ``sys`` 
> module,
> with which frameworks can provide advanced debugging facilities.
>
> It is also important to make coroutines as fast and efficient as possible,
> therefore there are no debug features enabled by default.
>
> Example::
>
>      async def debug_me():
>          await asyncio.sleep(1)
>
>      def async_debug_wrap(generator):
>          return asyncio.AsyncDebugWrapper(generator)
>
>      sys.set_async_wrapper(async_debug_wrap)
>
>      debug_me()  # <- this line will likely GC the coroutine object and
>                  # trigger AsyncDebugWrapper's code.
>
>      assert isinstance(debug_me(), AsyncDebugWrapper)
>
>      sys.set_async_wrapper(None)   # <- this unsets any previously set 
> wrapper
>      assert not isinstance(debug_me(), AsyncDebugWrapper)
>
> If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces 
> the
> previous wrapper.  ``sys.set_async_wrapper(None)`` unsets the wrapper.
>
>
> Glossary
> ========
>
> :Coroutine:
>      A coroutine function, or just "coroutine", is declared with ``async 
> def``.
>      It uses ``await`` and ``return value``; see `New Coroutine Declaration
>      Syntax`_ for details.
>
> :Coroutine object:
>      Returned from a coroutine function. See `Await Expression`_ for 
> details.
>
> :Future-like object:
>      An object with an ``__await__`` method.  It is consumed by 
> ``await`` in a
>      coroutine. A coroutine waiting for a Future-like object is 
> suspended until
>      the Future-like object's ``__await__`` completes.  ``await`` 
> returns the
>      result of the Future-like object.  See `Await Expression`_ for 
> details.
>
> :Awaitable:
>      A *future-like* object or a *coroutine object*.  See `Await 
> Expression`_
>      for details.
>
> :Generator-based coroutine:
>      Coroutines based in generator syntax.  Most common example is
>      ``@asyncio.coroutine``.
>
> :Asynchronous context manager:
>     An asynchronous context manager has ``__aenter__`` and ``__aexit__`` 
> methods
>     and can be used with ``async with``.  See
>     `Asynchronous Context Managers and "async with"`_ for details.
>
> :Asynchronous iterable:
>      An object with an ``__aiter__`` method, which must return an 
> *asynchronous
>      iterator* object.  Can be used with ``async for``. See
>      `Asynchronous Iterators and "async for"`_ for details.
>
> :Asynchronous iterator:
>      An asynchronous iterator has an ``__anext__`` method.See
>      `Asynchronous Iterators and "async for"`_ for details.
>
>
> List of functions and methods
> =============================
>
> ================= ======================================= =================
> Method            Can contain                              Can't contain
> ================= ======================================= =================
> async def func    await, return value                      yield, yield 
> from
> async def __a*__  await, return value                      yield, yield 
> from
> def __a*__        return Future-like                       await
> def __await__     yield, yield from, return iterable       await
> generator         yield, yield from, return value          await
> ================= ======================================= =================
>
> Where:
>
> * ""async def func": coroutine;
>
> * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
>    ``__aexit__`` defined with the ``async`` keyword;
>
> * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__``
>    defined without the ``async`` keyword, must return an *awaitable*;
>
> * "def __await__": ``__await__`` method to implement *Future-like* objects;
>
> * generator: a "regular" generator, function defined with ``def`` and which
>    contains a least one ``yield`` or ``yield from`` expression.
>
> *Future-like* is an object with an ``__await__`` method, see
> `Await Expression`_ section for details.
>
>
> Transition Plan
> ===============
>
> To avoid backwards compatibility issues with ``async`` and ``await`` 
> keywords,
> it was decided to modify ``tokenizer.c`` in such a way, that it:
>
> * recognizes ``async def`` name tokens combination (start of a coroutine);
>
> * keeps track of regular functions and coroutines;
>
> * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
>    ``AWAIT`` when in the process of yielding tokens for coroutines.
>
> This approach allows for seamless combination of new syntax features (all 
> of
> them available only in ``async`` functions) with any existing code.
>
> An example of having "async def" and "async" attribute in one piece of 
> code::
>
>      class Spam:
>          async = 42
>
>      async def ham():
>          print(getattr(Spam, 'async'))
>
>      # The coroutine can be executed and will print '42'
>
>
> Backwards Compatibility
> -----------------------
>
> The only backwards incompatible change is an extra argument ``is_async`` to
> ``FunctionDef`` AST node.  But since it is a documented fact that the 
> structure
> of AST nodes is an implementation detail and subject to change, this 
> should not
> be considered a serious issue.
>
>
> Grammar Updates
> ---------------
>
> Grammar changes are also fairly minimal::
>
>      await_expr: AWAIT test
>      await_stmt: await_expr
>
>      decorated: decorators (classdef | funcdef | async_funcdef)
>      async_funcdef: ASYNC funcdef
>
>      async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later
>
>      compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
>                     | funcdef | classdef | decorated | async_stmt)
>
>      atom: ('(' [yield_expr|await_expr|testlist_comp] ')' |
>            '[' [testlist_comp] ']' |
>            '{' [dictorsetmaker] '}' |
>            NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False’)
>
>      expr_stmt: testlist_star_expr (augassign 
> (yield_expr|await_expr|testlist) |
>                          ('=' (yield_expr|await_expr|testlist_star_expr))*)
>
>
> Transition Period Shortcomings
> ------------------------------
>
> There is just one.
>
> Until ``async`` and ``await`` are not proper keywords, it is not 
> possible (or
> at least very hard) to fix ``tokenizer.c`` to recognize them on the **same
> line** with ``def`` keyword::
>
>      # async and await will always be parsed as variables
>
>      async def outer():                             # 1
>          def nested(a=(await fut)):
>              pass
>
>      async def foo(): return (await fut)            # 2
>
> Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, 
> a
> ``SyntaxError`` will be raised.
>
> To workaround these issues, the above examples can be easily rewritten to a
> more readable form::
>
>      async def outer():                             # 1
>          a_default = await fut
>          def nested(a=a_default):
>              pass
>
>      async def foo():                               # 2
>          return (await fut)
>
> This limitation will go away as soon as ``async`` and ``await`` ate proper
> keywords.  Or if it's decided to use a future import for this PEP.
>
>
> Deprecation Plans
> -----------------
>
> ``async`` and ``await`` names will be softly deprecated in CPython 3.5 
> and 3.6.
> In 3.7 we will transform them to proper keywords.  Making ``async`` and
> ``await`` proper keywords before 3.7 might make it harder for people to 
> port
> their code to Python 3.
>
>
> asyncio
> -------
>
> ``asyncio`` module was adapted and tested to work with coroutines and new
> statements.  Backwards compatibility is 100% preserved.
>
> The required changes are mainly:
>
> 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()``
>     function.
>
> 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
>
> 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate
>     ``async()`` function.
>
>
> Design Considerations
> =====================
>
> No implicit wrapping in Futures
> -------------------------------
>
> There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key
> difference is that JavaScript "async functions" always return a Promise. 
> While
> this approach has some advantages, it also implies that a new Promise 
> object is
> created on each "async function" invocation.
>
> We could implement a similar functionality in Python, by wrapping all
> coroutines in a Future object, but this has the following disadvantages:
>
> 1. Performance.  A new Future object would be instantiated on each 
> coroutine
>     call.  Moreover, this makes implementation of ``await`` expressions 
> slower
>     (disabling optimizations of ``yield from``).
>
> 2. A new built-in ``Future`` object would need to be added.
>
> 3. Coming up with a generic ``Future`` interface that is usable for any use
>     case in any framework is a very hard to solve problem.
>
> 4. It is not a feature that is used frequently, when most of the code is
>     coroutines.
>
>
> Why "async" and "await" keywords
> --------------------------------
>
> async/await is not a new concept in programming languages:
>
> * C# has it since long time ago [5]_;
>
> * proposal to add async/await in ECMAScript 7 [2]_;
>    see also Traceur project [9]_;
>
> * Facebook's Hack/HHVM [6]_;
>
> * Google's Dart language [7]_;
>
> * Scala [8]_;
>
> * proposal to add async/await to C++ [10]_;
>
> * and many other less popular languages.
>
> This is a huge benefit, as some users already have experience with 
> async/await,
> and because it makes working with many languages in one project easier 
> (Python
> with ECMAScript 7 for instance).
>
>
> Why "__aiter__" is a coroutine
> ------------------------------
>
> In principle, ``__aiter__`` could be a regular function.  There are several
> good reasons to make it a coroutine:
>
> * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` 
> methods are
>    coroutines, users would often make a mistake defining it as ``async``
>    anyways;
>
> * there might be a need to run some asynchronous operations in 
> ``__aiter__``,
>    for instance to prepare DB queries or do some file operation.
>
>
> Importance of "async" keyword
> -----------------------------
>
> While it is possible to just implement ``await`` expression and treat all
> functions with at least one ``await`` as coroutines, this approach makes
> APIs design, code refactoring and its long time support harder.
>
> Let's pretend that Python only has ``await`` keyword::
>
>      def useful():
>          ...
>          await log(...)
>          ...
>
>      def important():
>          await useful()
>
> If ``useful()`` function is refactored and someone removes all ``await``
> expressions from it, it would become a regular python function, and all 
> code
> that depends on it, including ``important()`` would be broken.  To 
> mitigate this
> issue a decorator similar to ``@asyncio.coroutine`` has to be introduced.
>
>
> Why "async def"
> ---------------
>
> For some people bare ``async name(): pass`` syntax might look more 
> appealing
> than ``async def name(): pass``.  It is certainly easier to type. But on 
> the
> other hand, it breaks the symmetry between ``async def``, ``async with`` 
> and
> ``async for``, where ``async`` is a modifier, stating that the statement is
> asynchronous.  It is also more consistent with the existing grammar.
>
>
> Why not a __future__ import
> ---------------------------
>
> ``__future__`` imports are inconvenient and easy to forget to add. Also, 
> they
> are enabled for the whole source file.  Consider that there is a big 
> project
> with a popular module named "async.py".  With future imports it is 
> required to
> either import it using ``__import__()`` or ``importlib.import_module()`` 
> calls,
> or to rename the module.  The proposed approach makes it possible to 
> continue
> using old code and modules without a hassle, while coming up with a 
> migration
> plan for future python versions.
>
>
> Why magic methods start with "a"
> --------------------------------
>
> New asynchronous magic methods ``__aiter__``, ``__anext__``, 
> ``__aenter__``,
> and ``__aexit__`` all start with the same prefix "a".  An alternative 
> proposal
> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``.
> However, to align new magic methods with the existing ones, such as
> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version.
>
>
> Why not reuse existing magic names
> ----------------------------------
>
> An alternative idea about new asynchronous iterators and context 
> managers was
> to reuse existing magic methods, by adding an ``async`` keyword to their
> declarations::
>
>      class CM:
>          async def __enter__(self): # instead of __aenter__
>              ...
>
> This approach has the following downsides:
>
> * it would not be possible to create an object that works in both 
> ``with`` and
>    ``async with`` statements;
>
> * it would look confusing and would require some implicit magic behind the
>    scenes in the interpreter;
>
> * one of the main points of this proposal is to make coroutines as simple
>    and foolproof as possible.
>
>
> Comprehensions
> --------------
>
> For the sake of restricting the broadness of this PEP there is no new 
> syntax
> for asynchronous comprehensions.  This should be considered in a 
> separate PEP,
> if there is a strong demand for this feature.
>
>
> Performance
> ===========
>
> Overall Impact
> --------------
>
> This proposal introduces no observable performance impact.  Here is an 
> output
> of python's official set of benchmarks [4]_:
>
> ::
>
>      python perf.py -r -b default ../cpython/python.exe 
> ../cpython-aw/python.exe
>
>      [skipped]
>
>      Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
>      Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
>      x86_64 i386
>
>      Total CPU cores: 8
>
>      ### etree_iterparse ###
>      Min: 0.365359 -> 0.349168: 1.05x faster
>      Avg: 0.396924 -> 0.379735: 1.05x faster
>      Significant (t=9.71)
>      Stddev: 0.01225 -> 0.01277: 1.0423x larger
>
>      The following not significant results are hidden, use -v to show them:
>      django_v2, 2to3, etree_generate, etree_parse, etree_process, 
> fastpickle,
>      fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
>
>
> Tokenizer modifications
> -----------------------
>
> There is no observable slowdown of parsing python files with the modified
> tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated 
> 1000
> times) takes the same amount of time.
>
>
> async/await
> -----------
>
> The following micro-benchmark was used to determine performance difference
> between "async" functions and generators::
>
>      import sys
>      import time
>
>      def binary(n):
>          if n <= 0:
>              return 1
>          l = yield from binary(n - 1)
>          r = yield from binary(n - 1)
>          return l + 1 + r
>
>      async def abinary(n):
>          if n <= 0:
>              return 1
>          l = await abinary(n - 1)
>          r = await abinary(n - 1)
>          return l + 1 + r
>
>      def timeit(gen, depth, repeat):
>          t0 = time.time()
>          for _ in range(repeat):
>              list(gen(depth))
>          t1 = time.time()
>          print('{}({}) * {}: total {:.3f}s'.format(
>              gen.__name__, depth, repeat, t1-t0))
>
> The result is that there is no observable performance difference. Minimum
> timing of 3 runs
>
> ::
>
>      abinary(19) * 30: total 12.985s
>      binary(19) * 30: total 12.953s
>
> Note that depth of 19 means 1,048,575 calls.
>
>
> Reference Implementation
> ========================
>
> The reference implementation can be found here: [3]_.
>
> List of high-level changes and new protocols
> --------------------------------------------
>
> 1. New syntax for defining coroutines: ``async def`` and new ``await``
>     keyword.
>
> 2. New ``__await__`` method for Future-like objects.
>
> 3. New syntax for asynchronous context managers: ``async with``. And
>     associated protocol with ``__aenter__`` and ``__aexit__`` methods.
>
> 4. New syntax for asynchronous iteration: ``async for``.  And associated
>     protocol with ``__aiter__``, ``__aexit__`` and new built-in exception
>     ``StopAsyncIteration``.
>
> 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; 
> ``FunctionDef`` AST
>     node got a new argument ``is_async``.
>
> 6. New functions: ``sys.set_async_wrapper(callback)`` and
>     ``types.async_def(gen)``.
>
> 7. New ``CO_ASYNC`` bit flag for code objects.
>
> While the list of changes and new things is not short, it is important to
> understand, that most users will not use these features directly. It is
> intended to be used in frameworks and libraries to provide users with
> convenient to use and unambiguous APIs with ``async def``, ``await``, 
> ``async
> for`` and ``async with`` syntax.
>
>
> Working example
> ---------------
>
> All concepts proposed in this PEP are implemented [3]_ and can be tested.
>
> ::
>
>      import asyncio
>
>
>      async def echo_server():
>          print('Serving on localhost:8000')
>          await asyncio.start_server(handle_connection, 'localhost', 8000)
>
>
>      async def handle_connection(reader, writer):
>          print('New connection...')
>
>          while True:
>              data = await reader.read(8192)
>
>              if not data:
>                  break
>
>              print('Sending {:.10}... back'.format(repr(data)))
>              writer.write(data)
>
>
>      loop = asyncio.get_event_loop()
>      loop.run_until_complete(echo_server())
>      try:
>          loop.run_forever()
>      finally:
>          loop.close()
>
>
> References
> ==========
>
> .. [1] 
> https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
>
> .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
>
> .. [3] https://github.com/1st1/cpython/tree/await
>
> .. [4] https://hg.python.org/benchmarks
>
> .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
>
> .. [6] http://docs.hhvm.com/manual/en/hack.async.php
>
> .. [7] https://www.dartlang.org/articles/await-async/
>
> .. [8] http://docs.scala-lang.org/sips/pending/async.html
>
> .. [9] 
>
> https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental
>
> .. [10] 
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
>
>
> Acknowledgments
> ===============
>
> I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew 
> Svetlov,
> and Łukasz Langa for their initial feedback.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
> _______________________________________________
> Python-ideas mailing list
> Python... at python.org <javascript:>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150417/99451521/attachment-0001.html>


More information about the Python-ideas mailing list