https://hg.python.org/peps/rev/f37b0c1fb126 changeset: 5810:f37b0c1fb126 user: Yury Selivanov <yselivanov@sprymix.com> date: Wed Apr 29 21:11:53 2015 -0400 summary: pep-0492: v4. files: pep-0492.txt | 348 ++++++++++++++++++++++---------------- 1 files changed, 201 insertions(+), 147 deletions(-) diff --git a/pep-0492.txt b/pep-0492.txt --- a/pep-0492.txt +++ b/pep-0492.txt @@ -8,7 +8,7 @@ Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 -Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015 +Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015 Abstract @@ -57,8 +57,7 @@ ============= This proposal introduces new syntax and semantics to enhance coroutine -support in Python, it does not change the internal implementation of -coroutines, which are still based on generators. +support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax @@ -66,21 +65,22 @@ the "Cofunctions" proposal (PEP 3152, now rejected in favor of this specification). -From this point in this document we use the word *coroutine* to refer -to functions declared using the new syntax. *generator-based +From this point in this document we use the word *native coroutine* to +refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are -based on generator syntax. +based on generator syntax. *coroutine* is used in contexts where both +definitions are applicable. New Coroutine Declaration Syntax -------------------------------- -The following new syntax is used to declare a coroutine:: +The following new syntax is used to declare a *native coroutine*:: async def read_data(db): pass -Key properties of coroutines: +Key properties of *native coroutines*: * ``async def`` functions are always coroutines, even if they do not contain ``await`` expressions. @@ -88,10 +88,16 @@ * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. -* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced - to enable runtime detection of coroutines (and migrating existing - code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR`` - flags set. +* Internally, two new code object flags were introduced: + + - ``CO_COROUTINE`` is used to enable runtime detection of + *coroutines* (and migrating existing code). + + - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines* + (defined with new syntax.) + + All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and + ``CO_GENERATOR`` flags set. * Regular generators, when called, return a *generator object*; similarly, coroutines return a *coroutine object*. @@ -100,15 +106,25 @@ and are replaced with a ``RuntimeError``. For regular generators such behavior requires a future import (see PEP 479). +* See also `Coroutine objects`_ section. + types.coroutine() ----------------- A new function ``coroutine(gen)`` is added to the ``types`` module. It -applies ``CO_COROUTINE`` flag to the passed generator-function's code -object, making it to return a *coroutine object* when called. +allows interoperability between existing *generator-based coroutines* +in asyncio and *native coroutines* introduced by this PEP. -This feature enables an easy upgrade path for existing libraries. +The function applies ``CO_COROUTINE`` flag to generator-function's code +object, making it return a *coroutine object*. + +The function can be used as a decorator, since it modifies generator- +functions in-place and returns them. + +Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by +``types.coroutine()`` to make it possible to separate *native +coroutines* defined with new syntax, from *generator-based coroutines*. Await Expression @@ -129,7 +145,9 @@ validating its argument. ``await`` only accepts an *awaitable*, which can be one of: -* A *coroutine object* returned from a *coroutine* or a generator +* A *native coroutine object* returned from a *native coroutine*. + +* A *generator-based coroutine object* returned from a generator decorated with ``types.coroutine()``. * An object with an ``__await__`` method returning an iterator. @@ -142,7 +160,7 @@ explanation.) To enable this behavior for coroutines, a new magic method called - ``__await__`` is added. In asyncio, for instance, to enable Future + ``__await__`` is added. In asyncio, for instance, to enable *Future* objects in ``await`` statements, the only change is to add ``__await__ = __iter__`` line to ``asyncio.Future`` class. @@ -160,7 +178,9 @@ * Objects defined with CPython C API with a ``tp_await`` function, returning an iterator (similar to ``__await__`` method). -It is a ``SyntaxError`` to use ``await`` outside of a coroutine. +It is a ``SyntaxError`` to use ``await`` outside of an ``async def`` +function (like it is a ``SyntaxError`` to use ``yield`` outside of +``def`` function.) It is a ``TypeError`` to pass anything other than an *awaitable* object to an ``await`` expression. @@ -169,11 +189,22 @@ Updated operator precedence table ''''''''''''''''''''''''''''''''' -``await`` keyword is defined differently from ``yield`` and ``yield -from`` in the Grammar. +``await`` keyword is defined as follows:: -The key difference is that *await expressions* do not require -parentheses around them most of the times. + power ::= await ["**" u_expr] + await ::= ["await"] primary + +where "primary" represents the most tightly bound operations of the +language. Its syntax is:: + + primary ::= atom | attributeref | subscription | slicing | call + +See Python Documentation [12]_ and `Grammar Updates`_ section of this +proposal for details. + +The key ``await`` difference from ``yield`` and ``yield from`` +operators is that *await expressions* do not require parentheses around +them most of the times. Also, ``yield from`` allows any expression as its argument, including expressions like ``yield from a() + b()``, that would be parsed as @@ -186,7 +217,8 @@ +------------------------------+-----------------------------------+ | Operator | Description | +==============================+===================================+ -| ``yield``, ``yield from`` | Yield expression | +| ``yield`` ``x``, | Yield expression | +| ``yield from`` ``x`` | | +------------------------------+-----------------------------------+ | ``lambda`` | Lambda expression | +------------------------------+-----------------------------------+ @@ -221,7 +253,7 @@ +------------------------------+-----------------------------------+ | ``**`` | Exponentiation | +------------------------------+-----------------------------------+ -| ``await`` | Await expression | +| ``await`` ``x`` | Await expression | +------------------------------+-----------------------------------+ | ``x[index]``, | Subscription, slicing, | | ``x[index:index]``, | call, attribute reference | @@ -234,8 +266,6 @@ | ``{expressions...}`` | set display | +------------------------------+-----------------------------------+ -See `Grammar Updates`_ section for details. - Examples of "await" expressions ''''''''''''''''''''''''''''''' @@ -266,8 +296,6 @@ ``await -coro()`` ``await (-coro())`` ================================== ================================== -See `Grammar Updates`_ section for details. - Asynchronous Context Managers and "async with" ---------------------------------------------- @@ -306,18 +334,13 @@ exc = True try: - try: - VAR = await aenter - BLOCK - except: - exc = False - exit_res = await aexit(mgr, *sys.exc_info()) - if not exit_res: - raise - - finally: - if exc: - await aexit(mgr, None, None, None) + VAR = await aenter + BLOCK + except: + if not await aexit(mgr, *sys.exc_info()): + raise + else: + await aexit(mgr, None, None, None) As with regular ``with`` statements, it is possible to specify multiple @@ -325,13 +348,13 @@ It is an error to pass a regular context manager without ``__aenter__`` and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` -to use ``async with`` outside of a coroutine. +to use ``async with`` outside of an ``async def`` function. Example ''''''' -With asynchronous context managers it is easy to implement proper +With *asynchronous context managers* it is easy to implement proper database transaction managers for coroutines:: async def commit(session, data): @@ -416,7 +439,7 @@ It is a ``TypeError`` to pass a regular iterable without ``__aiter__`` method to ``async for``. It is a ``SyntaxError`` to use ``async for`` -outside of a coroutine. +outside of an ``async def`` function. As for with regular ``for`` statement, ``async for`` has an optional ``else`` clause. @@ -536,11 +559,61 @@ raised in coroutines are wrapped in ``RuntimeError``. +Coroutine objects +----------------- + +Differences from generators +''''''''''''''''''''''''''' + +This section applies only to *native coroutines* with +``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def`` +syntax. + +**The behavior of existing *generator-based coroutines* in asyncio +remains unchanged.** + +Great effort has been made to make sure that coroutines and +generators are treated as distinct concepts: + +1. *Native coroutine objects* do not implement ``__iter__`` and + ``__next__`` methods. Therefore, they cannot be iterated over or + passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins. + They also cannot be used in a ``for..in`` loop. + + An attempt to use ``__iter__`` or ``__next__`` on a *native + coroutine object* will result in a ``TypeError``. + +2. *Plain generators* cannot ``yield from`` *native coroutine objects*: + doing so will result in a ``TypeError``. + +3. *generator-based coroutines* (for asyncio code must be decorated + with ``@asyncio.coroutine``) can ``yield from`` *native coroutine + objects*. + +4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()`` + return ``False`` for *native coroutine objects* and *native + coroutine functions*. + + +Coroutine object methods +'''''''''''''''''''''''' + +Coroutines are based on generators internally, thus they share the +implementation. Similarly to generator objects, coroutine objects have +``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and +``GeneratorExit`` play the same role for coroutine objects (although +PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380, +and Python Documentation [11]_ for details. + +``throw()``, ``send()`` methods for coroutine objects are used to push +values and raise errors into *Future-like* objects. + + Debugging Features ------------------ -One of the most frequent mistakes that people make when using -generators as coroutines is forgetting to use ``yield from``:: +A common beginner mistake is forgetting to use ``yield from`` on +coroutines:: @asyncio.coroutine def useful(): @@ -590,65 +663,58 @@ # previously set wrapper assert not isinstance(debug_me(), asyncio.CoroWrapper) -If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper -replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)`` -unsets the wrapper. +New Standard Library Functions +------------------------------ -inspect.iscoroutine() and inspect.iscoroutineobject() ------------------------------------------------------ - -Two new functions are added to the ``inspect`` module: +* ``types.coroutine(gen)``. See `types.coroutine()`_ section for + details. * ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a - coroutine object. + *coroutine object*. -* ``inspect.iscoroutinefunction(obj)`` returns ``True`` is ``obj`` is a - coroutine function. +* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a + *coroutine function*. +* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used + in ``await`` expression. See `Await Expression`_ for details. -Differences between coroutines and generators ---------------------------------------------- +* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of + *coroutine objects*. ``wraper`` must be a callable that accepts one + argument: a *coroutine object* or ``None``. ``None`` resets the + wrapper. If called twice, the new wrapper replaces the previous one. + See `Debugging Features`_ for more details. -A great effort has been made to make sure that coroutines and -generators are separate concepts: - -1. Coroutine objects do not implement ``__iter__`` and ``__next__`` - methods. Therefore they cannot be iterated over or passed to - ``iter()``, ``list()``, ``tuple()`` and other built-ins. They - also cannot be used in a ``for..in`` loop. - -2. ``yield from`` does not accept coroutine objects (unless it is used - in a generator-based coroutine decorated with ``types.coroutine``.) - -3. ``yield from`` does not accept coroutine objects from plain Python - generators (*not* generator-based coroutines.) - -4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()`` - return ``False`` for coroutine objects and coroutine functions. - - -Coroutine objects ------------------ - -Coroutines are based on generators internally, thus they share the -implementation. Similarly to generator objects, coroutine objects have -``throw``, ``send`` and ``close`` methods. ``StopIteration`` and -``GeneratorExit`` play the same role for coroutine objects (although -PEP 479 is enabled by default for coroutines). +* ``sys.get_coroutine_wrapper()`` returns the current wrapper object. + Returns ``None`` if no wrapper was set. See `Debugging Features`_ + for more details. Glossary ======== +:Native coroutine: + A coroutine function is declared with ``async def``. It uses + ``await`` and ``return value``; see `New Coroutine Declaration + Syntax`_ for details. + +:Native coroutine object: + Returned from a native coroutine function. See `Await Expression`_ + for details. + +:Generator-based coroutine: + Coroutines based on generator syntax. Most common example are + functions decorated with ``@asyncio.coroutine``. + +:Generator-based coroutine object: + Returned from a generator-based coroutine function. + :Coroutine: - A coroutine function, or just "coroutine", is declared with ``async - def``. It uses ``await`` and ``return value``; see `New Coroutine - Declaration Syntax`_ for details. + Either *native coroutine* or *generator-based coroutine*. :Coroutine object: - Returned from a coroutine function. See `Await Expression`_ for - details. + Either *native coroutine object* or *generator-based coroutine + object*. :Future-like object: An object with an ``__await__`` method, or a C object with @@ -662,10 +728,6 @@ A *Future-like* object or a *coroutine object*. See `Await Expression`_ for details. -:Generator-based coroutine: - Coroutines based in generator syntax. Most common example is - ``@asyncio.coroutine``. - :Asynchronous context manager: An asynchronous context manager has ``__aenter__`` and ``__aexit__`` methods and can be used with ``async with``. See `Asynchronous @@ -696,7 +758,7 @@ Where: -* "async def func": coroutine; +* "async def func": native coroutine; * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` defined with the ``async`` keyword; @@ -720,12 +782,13 @@ it: * recognizes ``async def`` name tokens combination (start of a - coroutine); + native coroutine); -* keeps track of regular functions and coroutines; +* keeps track of regular functions and native coroutines; * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with - ``AWAIT`` when in the process of yielding tokens for coroutines. + ``AWAIT`` when in the process of yielding tokens for native + coroutines. This approach allows for seamless combination of new syntax features (all of them available only in ``async`` functions) with any existing @@ -749,6 +812,34 @@ This proposal preserves 100% backwards compatibility. +asyncio +------- + +``asyncio`` module was adapted and tested to work with coroutines and +new statements. Backwards compatibility is 100% preserved, i.e. all +existing code will work as-is. + +The required changes are mainly: + +1. Modify ``@asyncio.coroutine`` decorator to use new + ``types.coroutine()`` function. + +2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. + +3. Add ``ensure_task()`` as an alias for ``async()`` function. + Deprecate ``async()`` function. + + +Migration strategy +'''''''''''''''''' + +Because *plain generators* cannot ``yield from`` *native coroutine +objects* (see `Differences from generators`_ section for more details), +it is advised to make sure that all generator-based coroutines are +decorated with ``@asyncio.coroutine`` *before* starting to use the new +syntax. + + Grammar Updates --------------- @@ -811,23 +902,6 @@ for people to port their code to Python 3. -asyncio -------- - -``asyncio`` module was adapted and tested to work with coroutines and -new statements. Backwards compatibility is 100% preserved. - -The required changes are mainly: - -1. Modify ``@asyncio.coroutine`` decorator to use new - ``types.coroutine()`` function. - -2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. - -3. Add ``ensure_task()`` as an alias for ``async()`` function. - Deprecate ``async()`` function. - - Design Considerations ===================== @@ -928,31 +1002,6 @@ for a separate PEP. -No implicit wrapping in Futures -------------------------------- - -There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A -key difference is that JavaScript "async functions" always return a -Promise. While this approach has some advantages, it also implies that -a new Promise object is created on each "async function" invocation. - -We could implement a similar functionality in Python, by wrapping all -coroutines in a Future object, but this has the following -disadvantages: - -1. Performance. A new Future object would be instantiated on each - coroutine call. Moreover, this makes implementation of ``await`` - expressions slower (disabling optimizations of ``yield from``). - -2. A new built-in ``Future`` object would need to be added. - -3. Coming up with a generic ``Future`` interface that is usable for any - use case in any framework is a very hard to solve problem. - -4. It is not a feature that is used frequently, when most of the code - is coroutines. - - Why "async" and "await" keywords -------------------------------- @@ -978,8 +1027,8 @@ project easier (Python with ECMAScript 7 for instance). -Why "__aiter__" is a coroutine ------------------------------- +Why "__aiter__" returns awaitable +--------------------------------- In principle, ``__aiter__`` could be a regular function. There are several good reasons to make it a coroutine: @@ -1098,9 +1147,9 @@ returning a Future-like objects from ``__enter__`` and/or ``__exit__`` in Python <= 3.4; -* one of the main points of this proposal is to make coroutines as - simple and foolproof as possible, hence the clear separation of the - protocols. +* one of the main points of this proposal is to make native coroutines + as simple and foolproof as possible, hence the clear separation of + the protocols. Why not reuse existing "for" and "with" statements @@ -1120,8 +1169,8 @@ construct is outside of the scope of this PEP. -Async lambdas -------------- +Async lambda functions +---------------------- Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP. @@ -1236,9 +1285,11 @@ 6. New functions: ``sys.set_coroutine_wrapper(callback)``, ``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``, - ``inspect.iscoroutinefunction()``, and ``inspect.iscoroutine()``. + ``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``, + and ``inspect.isawaitable()``. -7. New ``CO_COROUTINE`` bit flag for code objects. +7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code + objects. While the list of changes and new things is not short, it is important to understand, that most users will not use these features directly. @@ -1305,6 +1356,9 @@ .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) +.. [11] https://docs.python.org/3/reference/expressions.html#generator-iterator-meth... + +.. [12] https://docs.python.org/3/reference/expressions.html#primaries Acknowledgments =============== -- Repository URL: https://hg.python.org/peps