[Python-checkins] peps: pep-0492: v4.
yury.selivanov
python-checkins at python.org
Thu Apr 30 03:30:29 CEST 2015
https://hg.python.org/peps/rev/f37b0c1fb126
changeset: 5810:f37b0c1fb126
user: Yury Selivanov <yselivanov at sprymix.com>
date: Wed Apr 29 21:11:53 2015 -0400
summary:
pep-0492: v4.
files:
pep-0492.txt | 348 ++++++++++++++++++++++----------------
1 files changed, 201 insertions(+), 147 deletions(-)
diff --git a/pep-0492.txt b/pep-0492.txt
--- a/pep-0492.txt
+++ b/pep-0492.txt
@@ -8,7 +8,7 @@
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
-Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015
+Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015
Abstract
@@ -57,8 +57,7 @@
=============
This proposal introduces new syntax and semantics to enhance coroutine
-support in Python, it does not change the internal implementation of
-coroutines, which are still based on generators.
+support in Python.
This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
@@ -66,21 +65,22 @@
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).
-From this point in this document we use the word *coroutine* to refer
-to functions declared using the new syntax. *generator-based
+From this point in this document we use the word *native coroutine* to
+refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
-based on generator syntax.
+based on generator syntax. *coroutine* is used in contexts where both
+definitions are applicable.
New Coroutine Declaration Syntax
--------------------------------
-The following new syntax is used to declare a coroutine::
+The following new syntax is used to declare a *native coroutine*::
async def read_data(db):
pass
-Key properties of coroutines:
+Key properties of *native coroutines*:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
@@ -88,10 +88,16 @@
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
-* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced
- to enable runtime detection of coroutines (and migrating existing
- code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``
- flags set.
+* Internally, two new code object flags were introduced:
+
+ - ``CO_COROUTINE`` is used to enable runtime detection of
+ *coroutines* (and migrating existing code).
+
+ - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
+ (defined with new syntax.)
+
+ All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
+ ``CO_GENERATOR`` flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
@@ -100,15 +106,25 @@
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
+* See also `Coroutine objects`_ section.
+
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
-applies ``CO_COROUTINE`` flag to the passed generator-function's code
-object, making it to return a *coroutine object* when called.
+allows interoperability between existing *generator-based coroutines*
+in asyncio and *native coroutines* introduced by this PEP.
-This feature enables an easy upgrade path for existing libraries.
+The function applies ``CO_COROUTINE`` flag to generator-function's code
+object, making it return a *coroutine object*.
+
+The function can be used as a decorator, since it modifies generator-
+functions in-place and returns them.
+
+Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
+``types.coroutine()`` to make it possible to separate *native
+coroutines* defined with new syntax, from *generator-based coroutines*.
Await Expression
@@ -129,7 +145,9 @@
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
-* A *coroutine object* returned from a *coroutine* or a generator
+* A *native coroutine object* returned from a *native coroutine*.
+
+* A *generator-based coroutine object* returned from a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
@@ -142,7 +160,7 @@
explanation.)
To enable this behavior for coroutines, a new magic method called
- ``__await__`` is added. In asyncio, for instance, to enable Future
+ ``__await__`` is added. In asyncio, for instance, to enable *Future*
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
@@ -160,7 +178,9 @@
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
-It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
+It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
+function (like it is a ``SyntaxError`` to use ``yield`` outside of
+``def`` function.)
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
@@ -169,11 +189,22 @@
Updated operator precedence table
'''''''''''''''''''''''''''''''''
-``await`` keyword is defined differently from ``yield`` and ``yield
-from`` in the Grammar.
+``await`` keyword is defined as follows::
-The key difference is that *await expressions* do not require
-parentheses around them most of the times.
+ power ::= await ["**" u_expr]
+ await ::= ["await"] primary
+
+where "primary" represents the most tightly bound operations of the
+language. Its syntax is::
+
+ primary ::= atom | attributeref | subscription | slicing | call
+
+See Python Documentation [12]_ and `Grammar Updates`_ section of this
+proposal for details.
+
+The key ``await`` difference from ``yield`` and ``yield from``
+operators is that *await expressions* do not require parentheses around
+them most of the times.
Also, ``yield from`` allows any expression as its argument, including
expressions like ``yield from a() + b()``, that would be parsed as
@@ -186,7 +217,8 @@
+------------------------------+-----------------------------------+
| Operator | Description |
+==============================+===================================+
-| ``yield``, ``yield from`` | Yield expression |
+| ``yield`` ``x``, | Yield expression |
+| ``yield from`` ``x`` | |
+------------------------------+-----------------------------------+
| ``lambda`` | Lambda expression |
+------------------------------+-----------------------------------+
@@ -221,7 +253,7 @@
+------------------------------+-----------------------------------+
| ``**`` | Exponentiation |
+------------------------------+-----------------------------------+
-| ``await`` | Await expression |
+| ``await`` ``x`` | Await expression |
+------------------------------+-----------------------------------+
| ``x[index]``, | Subscription, slicing, |
| ``x[index:index]``, | call, attribute reference |
@@ -234,8 +266,6 @@
| ``{expressions...}`` | set display |
+------------------------------+-----------------------------------+
-See `Grammar Updates`_ section for details.
-
Examples of "await" expressions
'''''''''''''''''''''''''''''''
@@ -266,8 +296,6 @@
``await -coro()`` ``await (-coro())``
================================== ==================================
-See `Grammar Updates`_ section for details.
-
Asynchronous Context Managers and "async with"
----------------------------------------------
@@ -306,18 +334,13 @@
exc = True
try:
- try:
- VAR = await aenter
- BLOCK
- except:
- exc = False
- exit_res = await aexit(mgr, *sys.exc_info())
- if not exit_res:
- raise
-
- finally:
- if exc:
- await aexit(mgr, None, None, None)
+ VAR = await aenter
+ BLOCK
+ except:
+ if not await aexit(mgr, *sys.exc_info()):
+ raise
+ else:
+ await aexit(mgr, None, None, None)
As with regular ``with`` statements, it is possible to specify multiple
@@ -325,13 +348,13 @@
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
-to use ``async with`` outside of a coroutine.
+to use ``async with`` outside of an ``async def`` function.
Example
'''''''
-With asynchronous context managers it is easy to implement proper
+With *asynchronous context managers* it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
@@ -416,7 +439,7 @@
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
-outside of a coroutine.
+outside of an ``async def`` function.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
@@ -536,11 +559,61 @@
raised in coroutines are wrapped in ``RuntimeError``.
+Coroutine objects
+-----------------
+
+Differences from generators
+'''''''''''''''''''''''''''
+
+This section applies only to *native coroutines* with
+``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
+syntax.
+
+**The behavior of existing *generator-based coroutines* in asyncio
+remains unchanged.**
+
+Great effort has been made to make sure that coroutines and
+generators are treated as distinct concepts:
+
+1. *Native coroutine objects* do not implement ``__iter__`` and
+ ``__next__`` methods. Therefore, they cannot be iterated over or
+ passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
+ They also cannot be used in a ``for..in`` loop.
+
+ An attempt to use ``__iter__`` or ``__next__`` on a *native
+ coroutine object* will result in a ``TypeError``.
+
+2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
+ doing so will result in a ``TypeError``.
+
+3. *generator-based coroutines* (for asyncio code must be decorated
+ with ``@asyncio.coroutine``) can ``yield from`` *native coroutine
+ objects*.
+
+4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
+ return ``False`` for *native coroutine objects* and *native
+ coroutine functions*.
+
+
+Coroutine object methods
+''''''''''''''''''''''''
+
+Coroutines are based on generators internally, thus they share the
+implementation. Similarly to generator objects, coroutine objects have
+``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and
+``GeneratorExit`` play the same role for coroutine objects (although
+PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380,
+and Python Documentation [11]_ for details.
+
+``throw()``, ``send()`` methods for coroutine objects are used to push
+values and raise errors into *Future-like* objects.
+
+
Debugging Features
------------------
-One of the most frequent mistakes that people make when using
-generators as coroutines is forgetting to use ``yield from``::
+A common beginner mistake is forgetting to use ``yield from`` on
+coroutines::
@asyncio.coroutine
def useful():
@@ -590,65 +663,58 @@
# previously set wrapper
assert not isinstance(debug_me(), asyncio.CoroWrapper)
-If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper
-replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)``
-unsets the wrapper.
+New Standard Library Functions
+------------------------------
-inspect.iscoroutine() and inspect.iscoroutineobject()
------------------------------------------------------
-
-Two new functions are added to the ``inspect`` module:
+* ``types.coroutine(gen)``. See `types.coroutine()`_ section for
+ details.
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
- coroutine object.
+ *coroutine object*.
-* ``inspect.iscoroutinefunction(obj)`` returns ``True`` is ``obj`` is a
- coroutine function.
+* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
+ *coroutine function*.
+* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
+ in ``await`` expression. See `Await Expression`_ for details.
-Differences between coroutines and generators
----------------------------------------------
+* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of
+ *coroutine objects*. ``wraper`` must be a callable that accepts one
+ argument: a *coroutine object* or ``None``. ``None`` resets the
+ wrapper. If called twice, the new wrapper replaces the previous one.
+ See `Debugging Features`_ for more details.
-A great effort has been made to make sure that coroutines and
-generators are separate concepts:
-
-1. Coroutine objects do not implement ``__iter__`` and ``__next__``
- methods. Therefore they cannot be iterated over or passed to
- ``iter()``, ``list()``, ``tuple()`` and other built-ins. They
- also cannot be used in a ``for..in`` loop.
-
-2. ``yield from`` does not accept coroutine objects (unless it is used
- in a generator-based coroutine decorated with ``types.coroutine``.)
-
-3. ``yield from`` does not accept coroutine objects from plain Python
- generators (*not* generator-based coroutines.)
-
-4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
- return ``False`` for coroutine objects and coroutine functions.
-
-
-Coroutine objects
------------------
-
-Coroutines are based on generators internally, thus they share the
-implementation. Similarly to generator objects, coroutine objects have
-``throw``, ``send`` and ``close`` methods. ``StopIteration`` and
-``GeneratorExit`` play the same role for coroutine objects (although
-PEP 479 is enabled by default for coroutines).
+* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
+ Returns ``None`` if no wrapper was set. See `Debugging Features`_
+ for more details.
Glossary
========
+:Native coroutine:
+ A coroutine function is declared with ``async def``. It uses
+ ``await`` and ``return value``; see `New Coroutine Declaration
+ Syntax`_ for details.
+
+:Native coroutine object:
+ Returned from a native coroutine function. See `Await Expression`_
+ for details.
+
+:Generator-based coroutine:
+ Coroutines based on generator syntax. Most common example are
+ functions decorated with ``@asyncio.coroutine``.
+
+:Generator-based coroutine object:
+ Returned from a generator-based coroutine function.
+
:Coroutine:
- A coroutine function, or just "coroutine", is declared with ``async
- def``. It uses ``await`` and ``return value``; see `New Coroutine
- Declaration Syntax`_ for details.
+ Either *native coroutine* or *generator-based coroutine*.
:Coroutine object:
- Returned from a coroutine function. See `Await Expression`_ for
- details.
+ Either *native coroutine object* or *generator-based coroutine
+ object*.
:Future-like object:
An object with an ``__await__`` method, or a C object with
@@ -662,10 +728,6 @@
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
-:Generator-based coroutine:
- Coroutines based in generator syntax. Most common example is
- ``@asyncio.coroutine``.
-
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
@@ -696,7 +758,7 @@
Where:
-* "async def func": coroutine;
+* "async def func": native coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
@@ -720,12 +782,13 @@
it:
* recognizes ``async def`` name tokens combination (start of a
- coroutine);
+ native coroutine);
-* keeps track of regular functions and coroutines;
+* keeps track of regular functions and native coroutines;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
- ``AWAIT`` when in the process of yielding tokens for coroutines.
+ ``AWAIT`` when in the process of yielding tokens for native
+ coroutines.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
@@ -749,6 +812,34 @@
This proposal preserves 100% backwards compatibility.
+asyncio
+-------
+
+``asyncio`` module was adapted and tested to work with coroutines and
+new statements. Backwards compatibility is 100% preserved, i.e. all
+existing code will work as-is.
+
+The required changes are mainly:
+
+1. Modify ``@asyncio.coroutine`` decorator to use new
+ ``types.coroutine()`` function.
+
+2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
+
+3. Add ``ensure_task()`` as an alias for ``async()`` function.
+ Deprecate ``async()`` function.
+
+
+Migration strategy
+''''''''''''''''''
+
+Because *plain generators* cannot ``yield from`` *native coroutine
+objects* (see `Differences from generators`_ section for more details),
+it is advised to make sure that all generator-based coroutines are
+decorated with ``@asyncio.coroutine`` *before* starting to use the new
+syntax.
+
+
Grammar Updates
---------------
@@ -811,23 +902,6 @@
for people to port their code to Python 3.
-asyncio
--------
-
-``asyncio`` module was adapted and tested to work with coroutines and
-new statements. Backwards compatibility is 100% preserved.
-
-The required changes are mainly:
-
-1. Modify ``@asyncio.coroutine`` decorator to use new
- ``types.coroutine()`` function.
-
-2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
-
-3. Add ``ensure_task()`` as an alias for ``async()`` function.
- Deprecate ``async()`` function.
-
-
Design Considerations
=====================
@@ -928,31 +1002,6 @@
for a separate PEP.
-No implicit wrapping in Futures
--------------------------------
-
-There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A
-key difference is that JavaScript "async functions" always return a
-Promise. While this approach has some advantages, it also implies that
-a new Promise object is created on each "async function" invocation.
-
-We could implement a similar functionality in Python, by wrapping all
-coroutines in a Future object, but this has the following
-disadvantages:
-
-1. Performance. A new Future object would be instantiated on each
- coroutine call. Moreover, this makes implementation of ``await``
- expressions slower (disabling optimizations of ``yield from``).
-
-2. A new built-in ``Future`` object would need to be added.
-
-3. Coming up with a generic ``Future`` interface that is usable for any
- use case in any framework is a very hard to solve problem.
-
-4. It is not a feature that is used frequently, when most of the code
- is coroutines.
-
-
Why "async" and "await" keywords
--------------------------------
@@ -978,8 +1027,8 @@
project easier (Python with ECMAScript 7 for instance).
-Why "__aiter__" is a coroutine
-------------------------------
+Why "__aiter__" returns awaitable
+---------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
@@ -1098,9 +1147,9 @@
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
-* one of the main points of this proposal is to make coroutines as
- simple and foolproof as possible, hence the clear separation of the
- protocols.
+* one of the main points of this proposal is to make native coroutines
+ as simple and foolproof as possible, hence the clear separation of
+ the protocols.
Why not reuse existing "for" and "with" statements
@@ -1120,8 +1169,8 @@
construct is outside of the scope of this PEP.
-Async lambdas
--------------
+Async lambda functions
+----------------------
Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.
@@ -1236,9 +1285,11 @@
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
- ``inspect.iscoroutinefunction()``, and ``inspect.iscoroutine()``.
+ ``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
+ and ``inspect.isawaitable()``.
-7. New ``CO_COROUTINE`` bit flag for code objects.
+7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
+ objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
@@ -1305,6 +1356,9 @@
.. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
+.. [11] https://docs.python.org/3/reference/expressions.html#generator-iterator-methods
+
+.. [12] https://docs.python.org/3/reference/expressions.html#primaries
Acknowledgments
===============
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list