PEP 492: async/await in Python; version 4

Hi python-dev, New version of the PEP is attached. Summary of updates: 1. Terminology: - *native coroutine* term is used for "async def" functions. - *generator-based coroutine* term is used for PEP 380 coroutines used in asyncio. - *coroutine* is used when *native coroutine* or *generator based coroutine* can be used in the same context. I think that it's not really productive to discuss the terminology that we use in the PEP. Its only purpose is to disambiguate concepts used in the PEP. We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted. Although if you notice that somewhere in the PEP it is not crystal clear what "coroutine" means please give me a heads up! 2. Syntax of await expressions is now thoroghly defined in the PEP. See "Await Expression", "Updated operator precedence table", and "Examples of "await" expressions" sections. I like the current approach. I'm still not convinced that we should make 'await' the same grammatically as unary minus. I don't understand why we should allow parsing things like 'await -fut'; this should be a SyntaxError. I'm fine to modify the grammar to allow 'await await fut' syntax, though. And I'm open to discussion :) 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. 4. asyncio / Migration strategy. Existing code can be used with PEP 492 as is, everything will work as expected. However, since *plain generator*s (not decorated with @asyncio.coroutine) cannot 'yield from' native coroutines (async def functions), it might break some code *while adapting it to the new syntax*. I'm open to just throw a RuntimeWarning in this case in 3.5, and raise a TypeError in 3.6. Please see the "Differences from generators" section of the PEP. 5. inspect.isawaitable() function. And, all new functions are now listed in the "New Standard Library Functions" section. 6. Lot's of small updates and tweaks throughout the PEP. Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yselivanov@sprymix.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015 Abstract ======== This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals =================== Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification ============= This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the "Cofunctions" proposal (PEP 3152, now rejected in favor of this specification). From this point in this document we use the word *native coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. *coroutine* is used in contexts where both definitions are applicable. New Coroutine Declaration Syntax -------------------------------- The following new syntax is used to declare a *native coroutine*:: async def read_data(db): pass Key properties of *native coroutines*: * ``async def`` functions are always coroutines, even if they do not contain ``await`` expressions. * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. * Internally, two new code object flags were introduced: - ``CO_COROUTINE`` is used to enable runtime detection of *coroutines* (and migrating existing code). - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines* (defined with new syntax.) All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and ``CO_GENERATOR`` flags set. * Regular generators, when called, return a *generator object*; similarly, coroutines return a *coroutine object*. * ``StopIteration`` exceptions are not propagated out of coroutines, and are replaced with a ``RuntimeError``. For regular generators such behavior requires a future import (see PEP 479). * See also `Coroutine objects`_ section. types.coroutine() ----------------- A new function ``coroutine(gen)`` is added to the ``types`` module. It allows interoperability between existing *generator-based coroutines* in asyncio and *native coroutines* introduced by this PEP. The function applies ``CO_COROUTINE`` flag to generator-function's code object, making it return a *coroutine object*. The function can be used as a decorator, since it modifies generator- functions in-place and returns them. Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by ``types.coroutine()`` to make it possible to separate *native coroutines* defined with new syntax, from *generator-based coroutines*. Await Expression ---------------- The following new ``await`` expression is used to obtain a result of coroutine execution:: async def read_data(db): data = await db.fetch('SELECT ...') ... ``await``, similarly to ``yield from``, suspends execution of ``read_data`` coroutine until ``db.fetch`` *awaitable* completes and returns the result data. It uses the ``yield from`` implementation with an extra step of validating its argument. ``await`` only accepts an *awaitable*, which can be one of: * A *native coroutine object* returned from a *native coroutine*. * A *generator-based coroutine object* returned from a generator decorated with ``types.coroutine()``. * An object with an ``__await__`` method returning an iterator. Any ``yield from`` chain of calls ends with a ``yield``. This is a fundamental mechanism of how *Futures* are implemented. Since, internally, coroutines are a special kind of generators, every ``await`` is suspended by a ``yield`` somewhere down the chain of ``await`` calls (please refer to PEP 3156 for a detailed explanation.) To enable this behavior for coroutines, a new magic method called ``__await__`` is added. In asyncio, for instance, to enable *Future* objects in ``await`` statements, the only change is to add ``__await__ = __iter__`` line to ``asyncio.Future`` class. Objects with ``__await__`` method are called *Future-like* objects in the rest of this PEP. Also, please note that ``__aiter__`` method (see its definition below) cannot be used for this purpose. It is a different protocol, and would be like using ``__iter__`` instead of ``__call__`` for regular callables. It is a ``TypeError`` if ``__await__`` returns anything but an iterator. * Objects defined with CPython C API with a ``tp_await`` function, returning an iterator (similar to ``__await__`` method). It is a ``SyntaxError`` to use ``await`` outside of an ``async def`` function (like it is a ``SyntaxError`` to use ``yield`` outside of ``def`` function.) It is a ``TypeError`` to pass anything other than an *awaitable* object to an ``await`` expression. Updated operator precedence table ''''''''''''''''''''''''''''''''' ``await`` keyword is defined as follows:: power ::= await ["**" u_expr] await ::= ["await"] primary where "primary" represents the most tightly bound operations of the language. Its syntax is:: primary ::= atom | attributeref | subscription | slicing | call See Python Documentation [12]_ and `Grammar Updates`_ section of this proposal for details. The key ``await`` difference from ``yield`` and ``yield from`` operators is that *await expressions* do not require parentheses around them most of the times. Also, ``yield from`` allows any expression as its argument, including expressions like ``yield from a() + b()``, that would be parsed as ``yield from (a() + b())``, which is almost always a bug. In general, the result of any arithmetic operation is not an *awaitable* object. To avoid this kind of mistakes, it was decided to make ``await`` precedence lower than ``[]``, ``()``, and ``.``, but higher than ``**`` operators. +------------------------------+-----------------------------------+ | Operator | Description | +==============================+===================================+ | ``yield`` ``x``, | Yield expression | | ``yield from`` ``x`` | | +------------------------------+-----------------------------------+ | ``lambda`` | Lambda expression | +------------------------------+-----------------------------------+ | ``if`` -- ``else`` | Conditional expression | +------------------------------+-----------------------------------+ | ``or`` | Boolean OR | +------------------------------+-----------------------------------+ | ``and`` | Boolean AND | +------------------------------+-----------------------------------+ | ``not`` ``x`` | Boolean NOT | +------------------------------+-----------------------------------+ | ``in``, ``not in``, | Comparisons, including membership | | ``is``, ``is not``, ``<``, | tests and identity tests | | ``<=``, ``>``, ``>=``, | | | ``!=``, ``==`` | | +------------------------------+-----------------------------------+ | ``|`` | Bitwise OR | +------------------------------+-----------------------------------+ | ``^`` | Bitwise XOR | +------------------------------+-----------------------------------+ | ``&`` | Bitwise AND | +------------------------------+-----------------------------------+ | ``<<``, ``>>`` | Shifts | +------------------------------+-----------------------------------+ | ``+``, ``-`` | Addition and subtraction | +------------------------------+-----------------------------------+ | ``*``, ``@``, ``/``, ``//``, | Multiplication, matrix | | ``%`` | multiplication, division, | | | remainder | +------------------------------+-----------------------------------+ | ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT | +------------------------------+-----------------------------------+ | ``**`` | Exponentiation | +------------------------------+-----------------------------------+ | ``await`` ``x`` | Await expression | +------------------------------+-----------------------------------+ | ``x[index]``, | Subscription, slicing, | | ``x[index:index]``, | call, attribute reference | | ``x(arguments...)``, | | | ``x.attribute`` | | +------------------------------+-----------------------------------+ | ``(expressions...)``, | Binding or tuple display, | | ``[expressions...]``, | list display, | | ``{key: value...}``, | dictionary display, | | ``{expressions...}`` | set display | +------------------------------+-----------------------------------+ Examples of "await" expressions ''''''''''''''''''''''''''''''' Valid syntax examples: ================================== ================================== Expression Will be parsed as ================================== ================================== ``if await fut: pass`` ``if (await fut): pass`` ``if await fut + 1: pass`` ``if (await fut) + 1: pass`` ``pair = await fut, 'spam'`` ``pair = (await fut), 'spam'`` ``with await fut, open(): pass`` ``with (await fut), open(): pass`` ``await foo()['spam'].baz()()`` ``await ( foo()['spam'].baz()() )`` ``return await coro()`` ``return ( await coro() )`` ``res = await coro() ** 2`` ``res = (await coro()) ** 2`` ``func(a1=await coro(), a2=0)`` ``func(a1=(await coro()), a2=0)`` ``await foo() + await bar()`` ``(await foo()) + (await bar())`` ``-await foo()`` ``-(await foo())`` ================================== ================================== Invalid syntax examples: ================================== ================================== Expression Should be written as ================================== ================================== ``await await coro()`` ``await (await coro())`` ``await -coro()`` ``await (-coro())`` ================================== ================================== Asynchronous Context Managers and "async with" ---------------------------------------------- An *asynchronous context manager* is a context manager that is able to suspend execution in its *enter* and *exit* methods. To make this possible, a new protocol for asynchronous context managers is proposed. Two new magic methods are added: ``__aenter__`` and ``__aexit__``. Both must return an *awaitable*. An example of an asynchronous context manager:: class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context') New Syntax '''''''''' A new statement for asynchronous context managers is proposed:: async with EXPR as VAR: BLOCK which is semantically equivalent to:: mgr = (EXPR) aexit = type(mgr).__aexit__ aenter = type(mgr).__aenter__(mgr) exc = True try: VAR = await aenter BLOCK except: if not await aexit(mgr, *sys.exc_info()): raise else: await aexit(mgr, None, None, None) As with regular ``with`` statements, it is possible to specify multiple context managers in a single ``async with`` statement. It is an error to pass a regular context manager without ``__aenter__`` and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use ``async with`` outside of an ``async def`` function. Example ''''''' With *asynchronous context managers* it is easy to implement proper database transaction managers for coroutines:: async def commit(session, data): ... async with session.transaction(): ... await session.update(data) ... Code that needs locking also looks lighter:: async with lock: ... instead of:: with (yield from lock): ... Asynchronous Iterators and "async for" -------------------------------------- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` exception. An example of asynchronous iterable:: class AsyncIterable: async def __aiter__(self): return self async def __anext__(self): data = await self.fetch_data() if data: return data else: raise StopAsyncIteration async def fetch_data(self): ... New Syntax '''''''''' A new statement for iterating through asynchronous iterators is proposed:: async for TARGET in ITER: BLOCK else: BLOCK2 which is semantically equivalent to:: iter = (ITER) iter = await type(iter).__aiter__(iter) running = True while running: try: TARGET = await type(iter).__anext__(iter) except StopAsyncIteration: running = False else: BLOCK else: BLOCK2 It is a ``TypeError`` to pass a regular iterable without ``__aiter__`` method to ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of an ``async def`` function. As for with regular ``for`` statement, ``async for`` has an optional ``else`` clause. Example 1 ''''''''' With asynchronous iteration protocol it is possible to asynchronously buffer data during iteration:: async for data in cursor: ... Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows of data from a database after every ``N`` iterations. The following code illustrates new asynchronous iteration protocol:: class Cursor: def __init__(self): self.buffer = collections.deque() def _prefetch(self): ... async def __aiter__(self): return self async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopAsyncIteration return self.buffer.popleft() then the ``Cursor`` class can be used as follows:: async for row in Cursor(): print(row) which would be equivalent to the following code:: i = await Cursor().__aiter__() while True: try: row = await i.__anext__() except StopAsyncIteration: break else: print(row) Example 2 ''''''''' The following is a utility class that transforms a regular iterable to an asynchronous one. While this is not a very useful thing to do, the code illustrates the relationship between regular and asynchronous iterators. :: class AsyncIteratorWrapper: def __init__(self, obj): self._it = iter(obj) async def __aiter__(self): return self async def __anext__(self): try: value = next(self._it) except StopIteration: raise StopAsyncIteration return value async for letter in AsyncIteratorWrapper("abc"): print(letter) Why StopAsyncIteration? ''''''''''''''''''''''' Coroutines are still based on generators internally. So, before PEP 479, there was no fundamental difference between :: def g1(): yield from fut return 'spam' and :: def g2(): yield from fut raise StopIteration('spam') And since PEP 479 is accepted and enabled by default for coroutines, the following example will have its ``StopIteration`` wrapped into a ``RuntimeError`` :: async def a1(): await fut raise StopIteration('spam') The only way to tell the outside code that the iteration has ended is to raise something other than ``StopIteration``. Therefore, a new built-in exception class ``StopAsyncIteration`` was added. Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions raised in coroutines are wrapped in ``RuntimeError``. Coroutine objects ----------------- Differences from generators ''''''''''''''''''''''''''' This section applies only to *native coroutines* with ``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def`` syntax. **The behavior of existing *generator-based coroutines* in asyncio remains unchanged.** Great effort has been made to make sure that coroutines and generators are treated as distinct concepts: 1. *Native coroutine objects* do not implement ``__iter__`` and ``__next__`` methods. Therefore, they cannot be iterated over or passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins. They also cannot be used in a ``for..in`` loop. An attempt to use ``__iter__`` or ``__next__`` on a *native coroutine object* will result in a ``TypeError``. 2. *Plain generators* cannot ``yield from`` *native coroutine objects*: doing so will result in a ``TypeError``. 3. *generator-based coroutines* (for asyncio code must be decorated with ``@asyncio.coroutine``) can ``yield from`` *native coroutine objects*. 4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()`` return ``False`` for *native coroutine objects* and *native coroutine functions*. Coroutine object methods '''''''''''''''''''''''' Coroutines are based on generators internally, thus they share the implementation. Similarly to generator objects, coroutine objects have ``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and ``GeneratorExit`` play the same role for coroutine objects (although PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380, and Python Documentation [11]_ for details. ``throw()``, ``send()`` methods for coroutine objects are used to push values and raise errors into *Future-like* objects. Debugging Features ------------------ A common beginner mistake is forgetting to use ``yield from`` on coroutines:: @asyncio.coroutine def useful(): asyncio.sleep(1) # this will do noting without 'yield from' For debugging this kind of mistakes there is a special debug mode in asyncio, in which ``@coroutine`` decorator wraps all functions with a special object with a destructor logging a warning. Whenever a wrapped generator gets garbage collected, a detailed logging message is generated with information about where exactly the decorator function was defined, stack trace of where it was collected, etc. Wrapper object also provides a convenient ``__repr__`` function with detailed information about the generator. The only problem is how to enable these debug capabilities. Since debug facilities should be a no-op in production mode, ``@coroutine`` decorator makes the decision of whether to wrap or not to wrap based on an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio programs with asyncio's own functions instrumented. ``EventLoop.set_debug``, a different debug facility, has no impact on ``@coroutine`` decorator's behavior. With this proposal, coroutines is a native, distinct from generators, concept. New methods ``set_coroutine_wrapper`` and ``get_coroutine_wrapper`` are added to the ``sys`` module, with which frameworks can provide advanced debugging facilities. It is also important to make coroutines as fast and efficient as possible, therefore there are no debug features enabled by default. Example:: async def debug_me(): await asyncio.sleep(1) def async_debug_wrap(generator): return asyncio.CoroWrapper(generator) sys.set_coroutine_wrapper(async_debug_wrap) debug_me() # <- this line will likely GC the coroutine object and # trigger asyncio.CoroWrapper's code. assert isinstance(debug_me(), asyncio.CoroWrapper) sys.set_coroutine_wrapper(None) # <- this unsets any # previously set wrapper assert not isinstance(debug_me(), asyncio.CoroWrapper) New Standard Library Functions ------------------------------ * ``types.coroutine(gen)``. See `types.coroutine()`_ section for details. * ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a *coroutine object*. * ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a *coroutine function*. * ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used in ``await`` expression. See `Await Expression`_ for details. * ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of *coroutine objects*. ``wraper`` must be a callable that accepts one argument: a *coroutine object* or ``None``. ``None`` resets the wrapper. If called twice, the new wrapper replaces the previous one. See `Debugging Features`_ for more details. * ``sys.get_coroutine_wrapper()`` returns the current wrapper object. Returns ``None`` if no wrapper was set. See `Debugging Features`_ for more details. Glossary ======== :Native coroutine: A coroutine function is declared with ``async def``. It uses ``await`` and ``return value``; see `New Coroutine Declaration Syntax`_ for details. :Native coroutine object: Returned from a native coroutine function. See `Await Expression`_ for details. :Generator-based coroutine: Coroutines based on generator syntax. Most common example are functions decorated with ``@asyncio.coroutine``. :Generator-based coroutine object: Returned from a generator-based coroutine function. :Coroutine: Either *native coroutine* or *generator-based coroutine*. :Coroutine object: Either *native coroutine object* or *generator-based coroutine object*. :Future-like object: An object with an ``__await__`` method, or a C object with ``tp_await`` function, returning an iterator. Can be consumed by an ``await`` expression in a coroutine. A coroutine waiting for a Future-like object is suspended until the Future-like object's ``__await__`` completes, and returns the result. See `Await Expression`_ for details. :Awaitable: A *Future-like* object or a *coroutine object*. See `Await Expression`_ for details. :Asynchronous context manager: An asynchronous context manager has ``__aenter__`` and ``__aexit__`` methods and can be used with ``async with``. See `Asynchronous Context Managers and "async with"`_ for details. :Asynchronous iterable: An object with an ``__aiter__`` method, which must return an *asynchronous iterator* object. Can be used with ``async for``. See `Asynchronous Iterators and "async for"`_ for details. :Asynchronous iterator: An asynchronous iterator has an ``__anext__`` method. See `Asynchronous Iterators and "async for"`_ for details. List of functions and methods ============================= ================= =================================== ================= Method Can contain Can't contain ================= =================================== ================= async def func await, return value yield, yield from async def __a*__ await, return value yield, yield from def __a*__ return awaitable await def __await__ yield, yield from, return iterable await generator yield, yield from, return value await ================= =================================== ================= Where: * "async def func": native coroutine; * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` defined with the ``async`` keyword; * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` defined without the ``async`` keyword, must return an *awaitable*; * "def __await__": ``__await__`` method to implement *Future-like* objects; * generator: a "regular" generator, function defined with ``def`` and which contains a least one ``yield`` or ``yield from`` expression. Transition Plan =============== To avoid backwards compatibility issues with ``async`` and ``await`` keywords, it was decided to modify ``tokenizer.c`` in such a way, that it: * recognizes ``async def`` name tokens combination (start of a native coroutine); * keeps track of regular functions and native coroutines; * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with ``AWAIT`` when in the process of yielding tokens for native coroutines. This approach allows for seamless combination of new syntax features (all of them available only in ``async`` functions) with any existing code. An example of having "async def" and "async" attribute in one piece of code:: class Spam: async = 42 async def ham(): print(getattr(Spam, 'async')) # The coroutine can be executed and will print '42' Backwards Compatibility ----------------------- This proposal preserves 100% backwards compatibility. asyncio ------- ``asyncio`` module was adapted and tested to work with coroutines and new statements. Backwards compatibility is 100% preserved, i.e. all existing code will work as-is. The required changes are mainly: 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.coroutine()`` function. 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate ``async()`` function. Migration strategy '''''''''''''''''' Because *plain generators* cannot ``yield from`` *native coroutine objects* (see `Differences from generators`_ section for more details), it is advised to make sure that all generator-based coroutines are decorated with ``@asyncio.coroutine`` *before* starting to use the new syntax. Grammar Updates --------------- Grammar changes are also fairly minimal:: decorated: decorators (classdef | funcdef | async_funcdef) async_funcdef: ASYNC funcdef compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt) async_stmt: ASYNC (funcdef | with_stmt | for_stmt) power: atom_expr ['**' factor] atom_expr: [AWAIT] atom trailer* Transition Period Shortcomings ------------------------------ There is just one. Until ``async`` and ``await`` are not proper keywords, it is not possible (or at least very hard) to fix ``tokenizer.c`` to recognize them on the **same line** with ``def`` keyword:: # async and await will always be parsed as variables async def outer(): # 1 def nested(a=(await fut)): pass async def foo(): return (await fut) # 2 Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, a ``SyntaxError`` will be raised. To workaround these issues, the above examples can be easily rewritten to a more readable form:: async def outer(): # 1 a_default = await fut def nested(a=a_default): pass async def foo(): # 2 return (await fut) This limitation will go away as soon as ``async`` and ``await`` are proper keywords. Or if it's decided to use a future import for this PEP. Deprecation Plans ----------------- ``async`` and ``await`` names will be softly deprecated in CPython 3.5 and 3.6. In 3.7 we will transform them to proper keywords. Making ``async`` and ``await`` proper keywords before 3.7 might make it harder for people to port their code to Python 3. Design Considerations ===================== PEP 3152 -------- PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines (called "cofunctions"). Some key points: 1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is always a generator, even if there is no ``cocall`` expressions inside it. Maps to ``async def`` in this proposal. 2. A new keyword ``cocall`` to call a *cofunction*. Can only be used inside a *cofunction*. Maps to ``await`` in this proposal (with some differences, see below.) 3. It is not possible to call a *cofunction* without a ``cocall`` keyword. 4. ``cocall`` grammatically requires parentheses after it:: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME 5. ``cocall f(*args, **kwds)`` is semantically equivalent to ``yield from f.__cocall__(*args, **kwds)``. Differences from this proposal: 1. There is no equivalent of ``__cocall__`` in this PEP, which is called and its result is passed to ``yield from`` in the ``cocall`` expression. ``await`` keyword expects an *awaitable* object, validates the type, and executes ``yield from`` on it. Although, ``__await__`` method is similar to ``__cocall__``, but is only used to define *Future-like* objects. 2. ``await`` is defined in almost the same way as ``yield from`` in the grammar (it is later enforced that ``await`` can only be inside ``async def``). It is possible to simply write ``await future``, whereas ``cocall`` always requires parentheses. 3. To make asyncio work with PEP 3152 it would be required to modify ``@asyncio.coroutine`` decorator to wrap all functions in an object with a ``__cocall__`` method, or to implement ``__cocall__`` on generators. To call *cofunctions* from existing generator-based coroutines it would be required to use ``costart(cofunc, *args, **kwargs)`` built-in. 4. Since it is impossible to call a *cofunction* without a ``cocall`` keyword, it automatically prevents the common mistake of forgetting to use ``yield from`` on generator-based coroutines. This proposal addresses this problem with a different approach, see `Debugging Features`_. 5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine is that if is decided to implement coroutine-generators -- coroutines with ``yield`` or ``async yield`` expressions -- we wouldn't need a ``cocall`` keyword to call them. So we'll end up having ``__cocall__`` and no ``__call__`` for regular coroutines, and having ``__call__`` and no ``__cocall__`` for coroutine- generators. 6. Requiring parentheses grammatically also introduces a whole lot of new problems. The following code:: await fut await function_returning_future() await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2)) would look like:: cocall fut() # or cocall costart(fut) cocall (function_returning_future())() cocall asyncio.gather(costart(coro1, arg1, arg2), costart(coro2, arg1, arg2)) 7. There are no equivalents of ``async for`` and ``async with`` in PEP 3152. Coroutine-generators -------------------- With ``async for`` keyword it is desirable to have a concept of a *coroutine-generator* -- a coroutine with ``yield`` and ``yield from`` expressions. To avoid any ambiguity with regular generators, we would likely require to have an ``async`` keyword before ``yield``, and ``async yield from`` would raise a ``StopAsyncIteration`` exception. While it is possible to implement coroutine-generators, we believe that they are out of scope of this proposal. It is an advanced concept that should be carefully considered and balanced, with a non-trivial changes in the implementation of current generator objects. This is a matter for a separate PEP. Why "async" and "await" keywords -------------------------------- async/await is not a new concept in programming languages: * C# has it since long time ago [5]_; * proposal to add async/await in ECMAScript 7 [2]_; see also Traceur project [9]_; * Facebook's Hack/HHVM [6]_; * Google's Dart language [7]_; * Scala [8]_; * proposal to add async/await to C++ [10]_; * and many other less popular languages. This is a huge benefit, as some users already have experience with async/await, and because it makes working with many languages in one project easier (Python with ECMAScript 7 for instance). Why "__aiter__" returns awaitable --------------------------------- In principle, ``__aiter__`` could be a regular function. There are several good reasons to make it a coroutine: * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` methods are coroutines, users would often make a mistake defining it as ``async`` anyways; * there might be a need to run some asynchronous operations in ``__aiter__``, for instance to prepare DB queries or do some file operation. Importance of "async" keyword ----------------------------- While it is possible to just implement ``await`` expression and treat all functions with at least one ``await`` as coroutines, this approach makes APIs design, code refactoring and its long time support harder. Let's pretend that Python only has ``await`` keyword:: def useful(): ... await log(...) ... def important(): await useful() If ``useful()`` function is refactored and someone removes all ``await`` expressions from it, it would become a regular python function, and all code that depends on it, including ``important()`` would be broken. To mitigate this issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. Why "async def" --------------- For some people bare ``async name(): pass`` syntax might look more appealing than ``async def name(): pass``. It is certainly easier to type. But on the other hand, it breaks the symmetry between ``async def``, ``async with`` and ``async for``, where ``async`` is a modifier, stating that the statement is asynchronous. It is also more consistent with the existing grammar. Why not "await for" and "await with" ------------------------------------ ``async`` is an adjective, and hence it is a better choice for a *statement qualifier* keyword. ``await for/with`` would imply that something is awaiting for a completion of a ``for`` or ``with`` statement. Why "async def" and not "def async" ----------------------------------- ``async`` keyword is a *statement qualifier*. A good analogy to it are "static", "public", "unsafe" keywords from other languages. "async for" is an asynchronous "for" statement, "async with" is an asynchronous "with" statement, "async def" is an asynchronous function. Having "async" after the main statement keyword might introduce some confusion, like "for async item in iterator" can be read as "for each asynchronous item in iterator". Having ``async`` keyword before ``def``, ``with`` and ``for`` also makes the language grammar simpler. And "async def" better separates coroutines from regular functions visually. Why not a __future__ import --------------------------- ``__future__`` imports are inconvenient and easy to forget to add. Also, they are enabled for the whole source file. Consider that there is a big project with a popular module named "async.py". With future imports it is required to either import it using ``__import__()`` or ``importlib.import_module()`` calls, or to rename the module. The proposed approach makes it possible to continue using old code and modules without a hassle, while coming up with a migration plan for future python versions. Why magic methods start with "a" -------------------------------- New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, and ``__aexit__`` all start with the same prefix "a". An alternative proposal is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. However, to align new magic methods with the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. Why not reuse existing magic names ---------------------------------- An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an ``async`` keyword to their declarations:: class CM: async def __enter__(self): # instead of __aenter__ ... This approach has the following downsides: * it would not be possible to create an object that works in both ``with`` and ``async with`` statements; * it would break backwards compatibility, as nothing prohibits from returning a Future-like objects from ``__enter__`` and/or ``__exit__`` in Python <= 3.4; * one of the main points of this proposal is to make native coroutines as simple and foolproof as possible, hence the clear separation of the protocols. Why not reuse existing "for" and "with" statements -------------------------------------------------- The vision behind existing generator-based coroutines and this proposal is to make it easy for users to see where the code might be suspended. Making existing "for" and "with" statements to recognize asynchronous iterators and context managers will inevitably create implicit suspend points, making it harder to reason about the code. Comprehensions -------------- Syntax for asynchronous comprehensions could be provided, but this construct is outside of the scope of this PEP. Async lambda functions ---------------------- Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP. Performance =========== Overall Impact -------------- This proposal introduces no observable performance impact. Here is an output of python's official set of benchmarks [4]_: :: python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe [skipped] Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### etree_iterparse ### Min: 0.365359 -> 0.349168: 1.05x faster Avg: 0.396924 -> 0.379735: 1.05x faster Significant (t=9.71) Stddev: 0.01225 -> 0.01277: 1.0423x larger The following not significant results are hidden, use -v to show them: django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle, fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. Tokenizer modifications ----------------------- There is no observable slowdown of parsing python files with the modified tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount of time. async/await ----------- The following micro-benchmark was used to determine performance difference between "async" functions and generators:: import sys import time def binary(n): if n <= 0: return 1 l = yield from binary(n - 1) r = yield from binary(n - 1) return l + 1 + r async def abinary(n): if n <= 0: return 1 l = await abinary(n - 1) r = await abinary(n - 1) return l + 1 + r def timeit(gen, depth, repeat): t0 = time.time() for _ in range(repeat): list(gen(depth)) t1 = time.time() print('{}({}) * {}: total {:.3f}s'.format( gen.__name__, depth, repeat, t1-t0)) The result is that there is no observable performance difference. Minimum timing of 3 runs :: abinary(19) * 30: total 12.985s binary(19) * 30: total 12.953s Note that depth of 19 means 1,048,575 calls. Reference Implementation ======================== The reference implementation can be found here: [3]_. List of high-level changes and new protocols -------------------------------------------- 1. New syntax for defining coroutines: ``async def`` and new ``await`` keyword. 2. New ``__await__`` method for Future-like objects, and new ``tp_await`` slot in ``PyTypeObject``. 3. New syntax for asynchronous context managers: ``async with``. And associated protocol with ``__aenter__`` and ``__aexit__`` methods. 4. New syntax for asynchronous iteration: ``async for``. And associated protocol with ``__aiter__``, ``__aexit__`` and new built- in exception ``StopAsyncIteration``. 5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``, ``Await``. 6. New functions: ``sys.set_coroutine_wrapper(callback)``, ``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``, ``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``, and ``inspect.isawaitable()``. 7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code objects. While the list of changes and new things is not short, it is important to understand, that most users will not use these features directly. It is intended to be used in frameworks and libraries to provide users with convenient to use and unambiguous APIs with ``async def``, ``await``, ``async for`` and ``async with`` syntax. Working example --------------- All concepts proposed in this PEP are implemented [3]_ and can be tested. :: import asyncio async def echo_server(): print('Serving on localhost:8000') await asyncio.start_server(handle_connection, 'localhost', 8000) async def handle_connection(reader, writer): print('New connection...') while True: data = await reader.read(8192) if not data: break print('Sending {:.10}... back'.format(repr(data))) writer.write(data) loop = asyncio.get_event_loop() loop.run_until_complete(echo_server()) try: loop.run_forever() finally: loop.close() References ========== .. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions .. [3] https://github.com/1st1/cpython/tree/await .. [4] https://hg.python.org/benchmarks .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx .. [6] http://docs.hhvm.com/manual/en/hack.async.php .. [7] https://www.dartlang.org/articles/await-async/ .. [8] http://docs.scala-lang.org/sips/pending/async.html .. [9] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-funct... .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) .. [11] https://docs.python.org/3/reference/expressions.html#generator-iterator-meth... .. [12] https://docs.python.org/3/reference/expressions.html#primaries Acknowledgments =============== I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, and Łukasz Langa for their initial feedback. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

One more thing to discuss: 7. StopAsyncIteration vs AsyncStopIteration. I don't have a strong opinion on this, I prefer the former because it reads better. There was no consensus on which one we should use. Thanks, Yury

Yury Selivanov wrote:
I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility". -- Greg

On 2015-04-30 7:24 PM, Greg Ewing wrote:
Ah, alright. You quoted this: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I wrote "full backwards compatibility" for that particular point #3 -- existing @asyncio.coroutines will have __iter__ and __next__ working just fine. Sorry if this was misleading.
I covered this in point #4. I also touched this in https://www.python.org/dev/peps/pep-0492/#migration-strategy I'm still waiting for feedback on this from Guido. If he decides to go with RuntimeWarnings, then it's 100% backwards compatible. If we keep TypeErrors -- then *existing code will work on 3.5*, but something *might* break during adopting new syntax. I'll update the Backwards Compatibility section. Thanks, Yury

On Thu, Apr 30, 2015 at 4:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Greg, you seem to have an odd notion of "full backwards compatibility". The term means that old code won't break. It doesn't imply that old and new code can always seamlessly interact (that would be an impossibly high bar for almost any change). That said, interoperability between old code and new code is an area of interest. But if the only thing that's standing between old code and new code is the @coroutine decorator, things are looking pretty good -- that decorator is already strongly required for coroutines intended for use with the asyncio package, and older versions of the asyncio package also define that decorator, so if there's old code out there that needs to be able to call the new coroutines (by whatever name, e.g. async functions :-), adding the @coroutine decorator to the old code doesn't look like too much of a burden. I assume there might be code out there that uses yield-from-based coroutines but does not use the asyncio package, but I doubt there is much code like that (I haven't seen much mention of yield-from outside its use in asyncio). So I think the interop problem is mostly limited to asyncio-using code that plays loose with the @coroutine decorator requirement and now wants to work with the new async functions. That's easy enough to address. -- --Guido van Rossum (python.org/~guido)

Yury Selivanov schrieb am 30.04.2015 um 03:30:
What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"? ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee. Stefan

On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. Thus, aware of the choice, I would still prefer a separate __anext__ method. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum schrieb am 01.05.2015 um 17:28:
My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap. But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ ("tp_iternext"), but not for __anext__. Stefan

Stefan, I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Yury On 2015-05-01 1:10 PM, Stefan Behnel wrote:

Yury Selivanov schrieb am 01.05.2015 um 20:52:
Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan

On 1 May 2015 at 21:27, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I having been arguing for merging two different protocols. I'm saying that allowing an object to be both normal and async iterable is not an argument for having separate protocols because it's not a good thing. Cheers, -- Arnaud

Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang

Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang

Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote:
I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury

On 5 May 2015 at 19:25, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it". Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that?
Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur. Paul

On Tue, May 5, 2015 at 3:14 PM Paul Moore <p.f.moore@gmail.com> wrote:
Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly).
I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib.
Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get "turn the wheel left to make the car go left".
I think the goal is "make coroutines easier to use" and does not directly relate to asyncio.
Yep.
I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that
You forgot to append "... yet" to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there.
Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that* difficult of a topic to teach people. This is just one of those PEPs where you have to trust the people with experience in the area are making good design decisions for those of us who aren't in a position to contribute directly without more experience in the domain. -Brett

Jumping in to correct one fact. On Tue, May 5, 2015 at 12:44 PM, Brett Cannon <brett@python.org> wrote:
The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n <= 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 21:38, Guido van Rossum <guido@python.org> wrote:
Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the "callback" terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says "this method is a coroutine" rather than "this method returns a Future"[1]. Paul [1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me.

On Tue, May 5, 2015 at 1:44 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for "either of the above" so we end up using one or the other as a shorthand. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). In the PEP 492 world, these concepts map as follows: - Future translates to "something with an __await__ method" (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - "either of the above" maps to "awaitable". -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 22:12, Guido van Rossum <guido@python.org> wrote:
Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen.
OK, that makes a lot of sense.
Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example, https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-...). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned?
OK. Although "future" is a nicer term than "something with an __await__ method" and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is "awaitable", it's still a net improvement. So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using "Future" for the generic term for "something with an __await__ method" at best, and quite possibly as equivalent to "awaitable", unfortunately). Paul

On Tue, May 5, 2015 at 2:29 PM, Paul Moore <p.f.moore@gmail.com> wrote:
No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running. Such complexity doesn't belong in such a simple example though.
I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as "complete" (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield from and await that will explain this.) It's likely that you could write a much simpler event loop by assuming just coroutines (either the kind implemented by generators using yield from or the PEP 492 kind). The reason asyncio uses callbacks at the lower levels is the hope of fostering interoperability with Twisted and Tornado (and even gevent, which also has an event loop at the bottom of everything). -- --Guido van Rossum (python.org/~guido)

On May 5, 2015 2:14 PM, "Guido van Rossum" <guido@python.org> wrote:
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. -n

On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith <njs@pobox.com> wrote: On May 5, 2015 2:14 PM, "Guido van Rossum" <guido@python.org> wrote:
The current PEP 492 proposal does endow the object returned by calling an async function (let's call it a coroutine object) with an __await__ method. And there's a good reason for this -- the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, which are used to allow send() and throw() calls on the outer generator to be passed into the inner generator with minimal fuss. (This is the reason why "yield from X" is *not* equivalent to "for x in X: yield x".) @Yury: I have a feeling the PEP could use more clarity here -- perhaps the section "Await Expression" should explain what the interepreter does for each type of awaitable? -- --Guido van Rossum (python.org/~guido)

(Yury gave similar responses, so (a) I'll just respond here, and (b) it's encouraging that you both responded so quickly with the same message) On 5 May 2015 at 20:44, Brett Cannon <brett@python.org> wrote:
I'll watch that - it should be fun. But I have seen things like that before, and I've got an idea how to write an event loop. You're right that it's easy to lose track of the fundamentally simple idea in all the complex discussions. To me that feels like a peculiar failure of the abstraction, in that in some circumstances it makes things feel *more* complex than they are :-)
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate "what is an event loop mechanism" from "what is asyncio specific". For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented "basic event loop" interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I guess the other canonical event loop use case is GUI system message dispatchers.
Fair point. If only I could avoid driving into walls :-)
OK. But in that case, some examples using a non-asyncio toy "just schedule tasks" event loop might help.
... and so you can't use it with async/await?
Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a "shared source" async and synchronous implementation of requests. So the typical "please support Python 3" issue that motivates projects to move forward doesn't exist in the same way. It's not to say that there won't be async versions of important libraries, it's just hard to see how the dynamics will work. I can't see myself raising an issue on cx_Oracle saying "please add asyncio support", and I don't know who else I would ask...
That's also a fair point, and it seems to me that there *is* reasonably general feeling that the experts can be trusted on the basic principles. There's also a huge amount of bikeshedding, but that's pretty much inevitable :-) But I do think that unless someone does something to offer some non-asyncio examples of coroutine-based asynchronous programming in Python, the link in people's minds between async and asyncio will become more and more entrenched. While asyncio is the only real event loop implementation, saying "async can be used for things other than asyncio" is a rather theoretical point. Is there anyone who feels they could write a stripped down but working example of a valid Python event loop *without* the asyncio aspects? Or is that what David Beazley's talk does? (I got the impression from what you said that he was aiming at async IO rather than just a non-IO event loop). Can asyncio.Future and asyncio.Task be reused with such an event loop, or would those need to be reimplemented as well? Writing your own event loop seems like a plausible exercise. Writing your own version of the whole task/future/coroutine/queue/synchronisation mechanisms seems like a lot to expect. And the event loop policy mechanism says that it works with loops that implement asyncio.BaseEventLoop (which as noted includes things like create_connection, etc). Paul

On Tue, May 5, 2015 at 1:39 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about.
What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 21:57, Guido van Rossum <guido@python.org> wrote:
That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). And I don't see where the PEP 492 additions would fit in (OK, "replace yield from with await" is part of it, but I don't see the rest). We may be talking at cross purposes here. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to "coroutine programming in Python" because they are referenced in any discussion of coroutines (you yield from a future, for example). If you see them as purely asyncio related (and not directly usable from outside of an asyncio context) then that may explain some of my confusion (but at the cost of reducing the coroutine concept to something pretty trivial in the absence of a library that independently implements these concepts). In some ways I wish there had been an "asyncio" library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just "async", maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. But that's water under the bridge now. Paul

Paul Moore wrote:
Because a minimal event loop doesn't *need* those. In my little scheduler, a "task" is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object. And there's really no need for the concept of a "future" at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a "future" is really just "an object that can be passed to yield-from". There is no need for a concrete Future class, it's just a protocol.
That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'.
Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate. When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-(
As I said before, I don't think it's really possible to factor an event loop into those kinds of parts. You may be able to factor the *API* that way, but any given implementation has to address all the parts at once. -- Greg

On 6 May 2015 at 09:20, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
It doesn't *need* them, but as abstractions they allow easier building of reusable higher-level libraries. You can write an event loop with nothing but coroutines, but to build reusable libraries on top of it, you need some common interfaces.
Agreed, you don't need a Future class, all you need is to agree what reusable code is allowed to do with the core objects you are passing around - that's how duck typing works. The objects *I* can see are futures (in a PEP 492 world, "awaitables" which may or may not be equivalent in terms of the operations you'd want to focus on) and the event loop itself. In your example, the event loop is implicit (as it's a singleton, you use global functions rather than methods on the loop object) but that's a minor detail.
So, "things you can wait on" have one operation - "wait for a result". That's OK. You can create such things as coroutines, which is also fine. You may want to create such things explicitly (equivalent to generators vs __iter__) - maybe that's where __aiter__ comes in in PEP 492 and the Future class in asyncio. Again, all fine. You also need operations like "schedule a thing to run", which is the event loop "interface". Your sample has the following basic event loop methods that I can see: run, schedule, unschedule, and expire_timeslice (that last one may be an implementation detail, but the other 3 seem pretty basic). PEP 492 has nothing to say on the event loop side of things (something that became clear to me during this discussion).
Futures seem to me to be (modulo a few details) what "awaitables" are in PEP 492. I can't see how you can meaningfully talk about event loops in a Python context without having *some* term for "things you wait for". Maybe Future wasn't a good name, and maybe the parallel with concurrent.futures.Future wasn't helpful (I think both things were fine, but you may not) but we had to have *some* way of talking about them, and of constructing standalone awaitables. PEP 492 has new, and hopefully better, ways, but I think that awaitables *have* to be central to any model where you wait for things... By the way, it feels to me like I'm now arguing in favour of PEP 492 with a reasonable understanding of what it "means". Assuming what I said above isn't complete rubbish, thanks to everyone who's helped me get to this point of understanding through this thread! (And if I haven't understood, that's my fault, and still thanks to everyone for their efforts :-)) Paul

Paul Moore wrote:
Take a look at the example I developed when working on the yield-from pep: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... The first version of the event loop presented there does exactly that, just schedules tasks in a round- robin fashion. Then I gradually add more features to it.
I don't it's possible to answer that question, because there isn't a single answer. The minimal set of features that an event loop needs depends on what you want to achieve with it. Even the notion of "just schedules tasks" is ambiguous. What does "schedule" mean? Does it just mean round-robin switching between them, or should they be able to synchronise with each other in some way? Should it be possible for a task to suspend itself for an interval of real world time, or does that come under the heading of I/O (since you're waiting for an external event, i.e. the computer's clock reaching some time)? Etc.
And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?)
I don't think *any* particular event loop features are needed. You can't pick out any of these features as being "core". For example, it would be possible to have an event loop that handled socket I/O but *didn't* do round-robin scheduling -- it could just keep on running the same task, even if it yielded, until it blocked waiting for an external event. Such a scheduler would probably be quite adequate for many use cases. It seems to me that the idea of "generator-based tasks managed by an event loop" is more of a design pattern than something you can write a detailed API specification for. Another problem with the "core" idea is that you can't start with an event loop that "just does scheduling" and then add on other features such as I/O *from the outside*. There has to be some point at which everything comes together, which means choosing something like select() or poll() or I/O completion queues, and build that into the heart of your event loop. At that point it's no longer something with a simple core. -- Greg

On Tue, May 5, 2015 at 1:39 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Twisted has a pretty good taxonomy of event loop methods, in the interfaces at the bottom of this page: http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html and the comparison matrix at http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html The asyncio event loops implement most of these (not the exact interfaces, but the same functionality). Tornado implements FDSet, Time, and part of Threads in the IOLoop itself, with the rest of the functionality coming from separate classes. (You may wonder then why Twisted and asyncio put everything in the core event loop? It's necessary to abstract over certain platform differences, which is one big reason why Tornado has poor support for Windows). -Ben

On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:
Another reason people don't realize it is that the PEP goes out of its way to avoid saying so. I understand that you (and Yuri) don't want to tie the PEP too tightly to the specific event loop implementation in asyncio.events.AbstractEventLoop, but ... that particular conflation isn't really what people are confused about. "coroutines" often brings up thoughts of independent tasks. Yuri may well know that "(Python has asymmetric coroutines, that's it)", but others have posted that this was a surprise -- and the people posting here have far more python experience than most readers will. Anyone deeply involved enough to recognize that this PEP is only about (1) a particular type of co-routine -- a subset even of prior python usage (2) used for a particular purpose (3) coordinated via an external scheduler will already know that they can substitute other event loops. Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

Jim, On 2015-05-05 5:09 PM, Jim J. Jewett wrote:
Thank you for this suggestion. I've added it to the PEP: https://hg.python.org/peps/rev/7ac132b24f1f Yury

Paul, On 2015-05-05 3:14 PM, Paul Moore wrote:
I agree. We have to improve asyncio docs in this area.
Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions.
There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user.
You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes.
There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient
It's chicken and egg problem. Right now, current coroutines via generators approach is cumbersome, it's harder to write async code than it should be. It stops the innovation in this area. Some languages like Go were specifically designed to make network programming easier, and they now steal users from Python. There is no absence of libraries for Go (and it's a new language!), btw. Give people the right tools and they will build what they need. Yury [1] https://github.com/KeepSafe/aiohttp

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:
Not just "How?", but "Why?". Why do they *need* to be disambiguated? With the benefit of having recently read all that discussion (as opposed to just the PEP), my answer is ... uh ... that generators vs "async def" is NOT an important distinction. What matters (as best I can tell) is: "something using yield (or yield from) to mark execution context switches" vs "other kinds of callables, including those using yield to make an iterator" I'm not quite sure that the actual proposal even really separates them effectively, in part because the terminology keeps suggesting other distinctions instead. (The glossary does help; just not enough.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

On 2015-05-05 5:31 PM, Jim J. Jewett wrote:
To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury

Hi Yury, On 05.05.2015 20:25, Yury Selivanov wrote:
I think monkeypatching and the gevent way is wrong. And explicit is better than implicit. I have to clear this. I meant how can we make this async stuff more accessible to the average sync user. Sometime even without writing or knowing how to write coroutines or other async stuff. Let me explain it with a deeper example (Some of it is related to Python 2 and twisted): I had the same problem for a server application using twisted. Provide easy interfaces to my users most not aware of async stuff. My solution was to write my own decorator similar to twisted's @inlineCallbacks. On top of it I added one more level to the decorator and distinguish if it was called from the main thread (also running the mainloop) and other threads. This object usable also as decorator is called "task" and has some utility methods. And returns a "deferred". (in asyncio this would be a Future) Resulting in code like: @task def myAsyncFunction(webaddress): result = yield fetch(webaddress) # only to show sleep example yield task.sleep(0.1) task.Return(result) Usable in a sync context (extra script thread): def do(): content = myAsyncFunction("http://xyz") or async context: @task def ado(): content = yield myAsyncFunction("http://xyz") The task decorator has functionality to check if something is called from the main thread (-> also a task and async) or it is called from another thread (-> sync or script). So this async interface is usable from both worlds. If someone operates async he/she must only know the task decorator and when to yield. If he/she uses it in sync mode nothing special has to be done. To allow all this the server starts the async main loop in the main thread and executes the script in an extra script thread. The user has every time his own thread, also for rpc stuff. The only way to switch into the main loop is to decorate a function as @task, every task is a coroutine and executed in the main thread (also thread of main loop). Benefit of all this: - Easy to write a async task it is marked as one and special stuff belongs to the task object. (task.Return is needed because we are in Python 2) - The normal user executes his stuff in his own thread and he/she can program in sync mode. No problem it is an extra thread and the main loop does not block. - A network client or other stuff has to be written only once, most time this can be a @task in the async world. But this should not care the end user. We don't have to implement all twice once for async and once for the sync world. -> Less overhead This is what I mean if I say we must address the bridging problem between the worlds. It think it is the wrong way to divide it in async and sync stuff and duplicate all networking libraries in the sync and async ones. For me the answer is to write one async netowrk library and use it in both, a sync script and in an async main loop. With an easy interface and not forcing the user to know this is an async library I have to do something special. And in future go one step further and use all this combined with PyParallel to solve the multiprocessing problem in Python. (Script in extra thread, mainloop in main thread, executed and managed via PyParallel avoiding the gil) But this is only a vision/dream of mine.
I do not. But for most they want only use it as a client and the main concern for most is "I want to get this web page" and not "I will implement a web client have to do this async and get it". Regards, Wolfgang

On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
In what way? I'm thinking that the only confusion would be if you wrote "async for" instead of "for", or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want. But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful. Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager. Objects can have multiple roles, and it's not usually abused, or confusing. I'm not sure that async iterables will be any different. -- Steve

On 3 May 2015 at 16:24, Steven D'Aprano <steve@pearwood.info> wrote:
Yes. This is the same kind of confusion that this PEP is trying hard to get rid of in other parts (i.e. the confusion between 'yield' and 'yield from' in current coroutines).
Well if you've made you object both iterable and 'async iterable' then it's very likely that you're going to get something meaningful out of either kind of iteration. Just not the way you want it if you mistakenly left out (or added) the 'async' keyword in your loop.
The latter is fine, because there is no danger of mistaking one for the other, unlike iterators and 'async iterators'. But my argument is simply that there is no good reason to aim for the ability to have object conform to both protocols. So it shouldn't be a criterion when looking at the merits of a proposal. I may very well be wrong but I haven't yet seen a compelling case for an object implementing both protocols. Cheers, -- Arnaud

Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different. Moreover, the below code is a perfectly valid, infinite iterable: class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam' I'm strong -1 on this idea. Yury On 2015-05-01 3:03 PM, Stefan Behnel wrote:

On 2015-05-01 3:23 PM, Yury Selivanov wrote:
To further clarify on the example: class SomeIterable: def __iter__(self): return self async def __aiter__(self): return self async def __next__(self): print('hello') raise StopAsyncIteration If you pass this to 'async for' you will get 'hello' printed and the loop will be over. If you pass this to 'for', you will get an infinite loop, because '__next__' will return a coroutine object (that has to be also awaited, but it wouldn't, because it's a plain 'for' statement). This is something that we shouldn't let happen. Yury

Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__.
Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators? Stefan

Stefan Behnel schrieb am 02.05.2015 um 06:54:
BTW, I guess that this "AIterator", or rather "AsyncIterator", needs to be a separate protocol (and ABC) then. Implementing "__aiter__()" and "__anext__()" seems perfectly reasonable without implementing (or using) a Coroutine. That means we also need an "AsyncIterable" as a base class for it. Would Coroutine then inherit from both Iterator and AsyncIterator? Or should we try to separate the protocol hierarchy completely? The section on "Coroutine objects" seems to suggest that inheritance from Iterator is not intended. OTOH, I'm not sure if inheriting from AsyncIterator is intended for Coroutine. The latter might just be a stand-alone ABC with send/throw/close, after all. I think that in order to get a better understanding of the protocol(s) that this PEP proposes, and the terminology that it should use, it would help to implement these ABCs. That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols. Stefan

On Sun May 3 08:32:02 CEST 2015, Stefan Behnel wrote:
Call next, then stick it somewhere it be waited on. Or is that syntactically illegal, because of the separation between sync and async? The "asych for" seems to assume that you want to do the waiting right now, at each step. (At least as far as this thread of the logic goes; something else might be happening in parallel via other threads of control.)
That means we also need an "AsyncIterable" as a base class for it.
Agreed.
That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols.
I hope not; they seem more like specialized versions of functions, such as are found in math or cmath. Ideally, as much as possible of this PEP should live in asycio, rather than appearing globally. Which reminds me ... *should* the "await" keyword work with any future, or is it really intentionally restricted to use with a single library module and 3rd party replacements? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront. Stefan

On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
I think "obvious[ly] misleading" is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call "native coroutine" suboptimal (even though I proposed it myself) and I am now in favor of using "async function". -- --Guido van Rossum (python.org/~guido)

On 1 May 2015 at 16:31, Guido van Rossum <guido@python.org> wrote:
But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing. IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines. You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about "async function"? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before. For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra "callback" parameter to be called when the asynchronous work is done. I think coroutine is the name of a concept, not a specific implementation. Cheers, -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert

On Fri, May 1, 2015 at 8:55 AM, Gustavo Carneiro <gjcarneiro@gmail.com> wrote:
we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method. -- --Guido van Rossum (python.org/~guido)

On 5/1/2015 9:59 AM, Guido van Rossum wrote:
Exactly. The async function/method is an implementation technique for a specific kind/subset of coroutine functionality. So the term coroutine, qualified by a description of its best usage and limitationsof async function, can be used in defining async function, thus appealing to what people know or have heard of and vaguely understand and can read more about in the literature. A glossary entry for coroutine in the docs seems appropriate, which could point out the 16† ways to implement coroutine-style functionalities in Python, and perhaps make recommendations for different types of usage. †OK, not 16 ways, but it is 3 now, or 4?

One more thing to discuss: 7. StopAsyncIteration vs AsyncStopIteration. I don't have a strong opinion on this, I prefer the former because it reads better. There was no consensus on which one we should use. Thanks, Yury

Yury Selivanov wrote:
I'm talking about the fact that existing generator- based coroutines that aren't decorated with @coroutine won't be able to call new ones that use async def. This means that converting one body of code to the new style can force changes in other code that interacts with it. Maybe this is not considered a problem, but as long as it's true, I don't think it's accurate to claim "full backwards compatibility". -- Greg

On 2015-04-30 7:24 PM, Greg Ewing wrote:
Ah, alright. You quoted this: 3. CO_NATIVE_COROUTINE flag. This enables us to disable __iter__ and __next__ on native coroutines while maintaining full backwards compatibility. I wrote "full backwards compatibility" for that particular point #3 -- existing @asyncio.coroutines will have __iter__ and __next__ working just fine. Sorry if this was misleading.
I covered this in point #4. I also touched this in https://www.python.org/dev/peps/pep-0492/#migration-strategy I'm still waiting for feedback on this from Guido. If he decides to go with RuntimeWarnings, then it's 100% backwards compatible. If we keep TypeErrors -- then *existing code will work on 3.5*, but something *might* break during adopting new syntax. I'll update the Backwards Compatibility section. Thanks, Yury

On Thu, Apr 30, 2015 at 4:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Greg, you seem to have an odd notion of "full backwards compatibility". The term means that old code won't break. It doesn't imply that old and new code can always seamlessly interact (that would be an impossibly high bar for almost any change). That said, interoperability between old code and new code is an area of interest. But if the only thing that's standing between old code and new code is the @coroutine decorator, things are looking pretty good -- that decorator is already strongly required for coroutines intended for use with the asyncio package, and older versions of the asyncio package also define that decorator, so if there's old code out there that needs to be able to call the new coroutines (by whatever name, e.g. async functions :-), adding the @coroutine decorator to the old code doesn't look like too much of a burden. I assume there might be code out there that uses yield-from-based coroutines but does not use the asyncio package, but I doubt there is much code like that (I haven't seen much mention of yield-from outside its use in asyncio). So I think the interop problem is mostly limited to asyncio-using code that plays loose with the @coroutine decorator requirement and now wants to work with the new async functions. That's easy enough to address. -- --Guido van Rossum (python.org/~guido)

Yury Selivanov schrieb am 30.04.2015 um 03:30:
What this section does not explain, AFAICT, nor the section on design considerations, is why the iterator protocol needs to be duplicated entirely. Can't we just assume (or even validate) that any 'regular' iterator returned from "__aiter__()" (as opposed to "__iter__()") properly obeys to the new protocol? Why additionally duplicate "__next__()" and "StopIteration"? ISTM that all this really depends on is that "__next__()" returns an awaitable. Renaming the method doesn't help with that guarantee. Stefan

On Fri, May 1, 2015 at 5:39 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
This is an astute observation. I think its flaw (if any) is the situation where we want a single object to be both a regular iterator and an async iterator (say when migrating code to the new world). The __next__ method might want to return a result while __anext__ has to return an awaitable. The solution to that would be to have __aiter__ return an instance of a different class than __iter__, but that's not always convenient. Thus, aware of the choice, I would still prefer a separate __anext__ method. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum schrieb am 01.05.2015 um 17:28:
My personal gut feeling is that this case would best be handled by a generic wrapper class. Both are well defined protocols, so I don't expect people to change all of their classes and instead return a wrapped object either from __iter__() or __aiter__(), depending on which they want to optimise for, or which will eventually turn out to be easier to wrap. But that's trying to predict the [Ff]uture, obviously. It just feels like unnecessary complexity for now. And we already have a type slot for __next__ ("tp_iternext"), but not for __anext__. Stefan

Stefan, I don't like the idea of combining __next__ and __anext__. In this case explicit is better than implicit. __next__ returning coroutines is a perfectly normal thing for a normal 'for' loop (it wouldn't to anything with them), whereas 'async for' will interpret that differently, and will try to await those coroutines. Yury On 2015-05-01 1:10 PM, Stefan Behnel wrote:

Yury Selivanov schrieb am 01.05.2015 um 20:52:
Sure, but the difference is that one would have called __aiter__() first and the other __iter__(). Normally, either of the two would not exist, so using the wrong loop on an object will just fail. However, after we passed that barrier, we already know that the object that was returned is supposed to obey to the expected protocol, so it doesn't matter whether we call __next__() or name it __anext__(), except that the second requires us to duplicate an existing protocol. This has nothing to do with implicit vs. explicit. Stefan

On 1 May 2015 at 21:27, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I having been arguing for merging two different protocols. I'm saying that allowing an object to be both normal and async iterable is not an argument for having separate protocols because it's not a good thing. Cheers, -- Arnaud

Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang

Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have "native coroutines" and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with "a". Easy to confuse with "a" for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a "sync" script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them "oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5". It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called "async". Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang

Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote:
I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not.
asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations.
"networking stuff", and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury

On 5 May 2015 at 19:25, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of "trust us, it makes sense if you need it, everyone else can ignore it". Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being "make coroutines and asyncio more accessible", If the actual goals are different, my conclusion is invalid.
Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes "something that the average user can't use". Which in turn further entrenches it as a specialist-only tool. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that?
Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur. Paul

On Tue, May 5, 2015 at 3:14 PM Paul Moore <p.f.moore@gmail.com> wrote:
Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly).
I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib.
Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get "turn the wheel left to make the car go left".
I think the goal is "make coroutines easier to use" and does not directly relate to asyncio.
Yep.
I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that
You forgot to append "... yet" to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there.
Co-routine-based asynchronous programming is new to Python, so as a community we don't have it as something everyone learns over time. If you don't come from an area that supports it then it will be foreign to you and not make sense without someone giving you a good tutorial on it. But considering C#, Dart, and Ecmascript 6 (will) have co-routine support -- and those are just the languages I can name off the top of my head -- using the exact same keywords suggests to me that it isn't *that* difficult of a topic to teach people. This is just one of those PEPs where you have to trust the people with experience in the area are making good design decisions for those of us who aren't in a position to contribute directly without more experience in the domain. -Brett

Jumping in to correct one fact. On Tue, May 5, 2015 at 12:44 PM, Brett Cannon <brett@python.org> wrote:
The run_in_executor call is not callback-based -- the confusion probably stems from the name of the function argument ('callback'). It actually returns a Future representing the result (or error) of an operation, where the operation is represented by the function argument. So if you have e.g. a function def factorial(n): return 1 if n <= 0 else n*factorial(n-1) you can run it in an executor from your async(io) code like this: loop = asyncio.get_event_loop() result = yield from loop.run_in_executor(factorial, 100) (In a PEP 492 coroutine substitute await for yield from.) -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 21:38, Guido van Rossum <guido@python.org> wrote:
Thanks, that's an important correction. Given that, run_in_executor is the link to blocking calls that I was searching for. And yes, the "callback" terminology does make this far from obvious, unfortunately. As does the point at which it's introduced (before futures have been described) and the fact that it says "this method is a coroutine" rather than "this method returns a Future"[1]. Paul [1] I'm still struggling to understand the terminology, so if those two statements are equivalent, that's not yet obvious to me.

On Tue, May 5, 2015 at 1:44 PM, Paul Moore <p.f.moore@gmail.com> wrote:
I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for "either of the above" so we end up using one or the other as a shorthand. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). In the PEP 492 world, these concepts map as follows: - Future translates to "something with an __await__ method" (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - "either of the above" maps to "awaitable". -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 22:12, Guido van Rossum <guido@python.org> wrote:
Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen.
OK, that makes a lot of sense.
Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example, https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-...). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned?
OK. Although "future" is a nicer term than "something with an __await__ method" and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is "awaitable", it's still a net improvement. So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using "Future" for the generic term for "something with an __await__ method" at best, and quite possibly as equivalent to "awaitable", unfortunately). Paul

On Tue, May 5, 2015 at 2:29 PM, Paul Moore <p.f.moore@gmail.com> wrote:
No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running. Such complexity doesn't belong in such a simple example though.
I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as "complete" (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield from and await that will explain this.) It's likely that you could write a much simpler event loop by assuming just coroutines (either the kind implemented by generators using yield from or the PEP 492 kind). The reason asyncio uses callbacks at the lower levels is the hope of fostering interoperability with Twisted and Tornado (and even gevent, which also has an event loop at the bottom of everything). -- --Guido van Rossum (python.org/~guido)

On May 5, 2015 2:14 PM, "Guido van Rossum" <guido@python.org> wrote:
In the PEP 492 world, these concepts map as follows:
- Future translates to "something with an __await__ method" (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__);
- "asyncio coroutine" maps to "PEP 492 coroutine object" (either defined
with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter);
- "either of the above" maps to "awaitable".
Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. -n

On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith <njs@pobox.com> wrote: On May 5, 2015 2:14 PM, "Guido van Rossum" <guido@python.org> wrote:
The current PEP 492 proposal does endow the object returned by calling an async function (let's call it a coroutine object) with an __await__ method. And there's a good reason for this -- the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, which are used to allow send() and throw() calls on the outer generator to be passed into the inner generator with minimal fuss. (This is the reason why "yield from X" is *not* equivalent to "for x in X: yield x".) @Yury: I have a feeling the PEP could use more clarity here -- perhaps the section "Await Expression" should explain what the interepreter does for each type of awaitable? -- --Guido van Rossum (python.org/~guido)

(Yury gave similar responses, so (a) I'll just respond here, and (b) it's encouraging that you both responded so quickly with the same message) On 5 May 2015 at 20:44, Brett Cannon <brett@python.org> wrote:
I'll watch that - it should be fun. But I have seen things like that before, and I've got an idea how to write an event loop. You're right that it's easy to lose track of the fundamentally simple idea in all the complex discussions. To me that feels like a peculiar failure of the abstraction, in that in some circumstances it makes things feel *more* complex than they are :-)
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate "what is an event loop mechanism" from "what is asyncio specific". For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented "basic event loop" interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I guess the other canonical event loop use case is GUI system message dispatchers.
Fair point. If only I could avoid driving into walls :-)
OK. But in that case, some examples using a non-asyncio toy "just schedule tasks" event loop might help.
... and so you can't use it with async/await?
Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a "shared source" async and synchronous implementation of requests. So the typical "please support Python 3" issue that motivates projects to move forward doesn't exist in the same way. It's not to say that there won't be async versions of important libraries, it's just hard to see how the dynamics will work. I can't see myself raising an issue on cx_Oracle saying "please add asyncio support", and I don't know who else I would ask...
That's also a fair point, and it seems to me that there *is* reasonably general feeling that the experts can be trusted on the basic principles. There's also a huge amount of bikeshedding, but that's pretty much inevitable :-) But I do think that unless someone does something to offer some non-asyncio examples of coroutine-based asynchronous programming in Python, the link in people's minds between async and asyncio will become more and more entrenched. While asyncio is the only real event loop implementation, saying "async can be used for things other than asyncio" is a rather theoretical point. Is there anyone who feels they could write a stripped down but working example of a valid Python event loop *without* the asyncio aspects? Or is that what David Beazley's talk does? (I got the impression from what you said that he was aiming at async IO rather than just a non-IO event loop). Can asyncio.Future and asyncio.Task be reused with such an event loop, or would those need to be reimplemented as well? Writing your own event loop seems like a plausible exercise. Writing your own version of the whole task/future/coroutine/queue/synchronisation mechanisms seems like a lot to expect. And the event loop policy mechanism says that it works with loops that implement asyncio.BaseEventLoop (which as noted includes things like create_connection, etc). Paul

On Tue, May 5, 2015 at 1:39 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about.
What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... -- --Guido van Rossum (python.org/~guido)

On 5 May 2015 at 21:57, Guido van Rossum <guido@python.org> wrote:
That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). And I don't see where the PEP 492 additions would fit in (OK, "replace yield from with await" is part of it, but I don't see the rest). We may be talking at cross purposes here. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to "coroutine programming in Python" because they are referenced in any discussion of coroutines (you yield from a future, for example). If you see them as purely asyncio related (and not directly usable from outside of an asyncio context) then that may explain some of my confusion (but at the cost of reducing the coroutine concept to something pretty trivial in the absence of a library that independently implements these concepts). In some ways I wish there had been an "asyncio" library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just "async", maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. But that's water under the bridge now. Paul

Paul Moore wrote:
Because a minimal event loop doesn't *need* those. In my little scheduler, a "task" is nothing more than a yield-frommable object sitting on a queue of things to be run. There is no need to wrap it in another object. And there's really no need for the concept of a "future" at all, except maybe at the boundary between generator-based async code and other things that are based on callbacks. Even then, a "future" is really just "an object that can be passed to yield-from". There is no need for a concrete Future class, it's just a protocol.
That's really all there is to it. The rest is concerned with catching certain kinds of mistakes, and providing convenient syntax for some patterns of using 'await'.
Only because they've been elevated to prominence by asyncio and its documentation, which I regard as unfortunate. When Guido was designing asyncio, I tried very hard to dissuade him from giving futures such a central place in the model. I saw them as an unnecessary concept that would only clutter up people's thinking. Seeing all the confusion now, I'm more convinced than ever that I was right. :-(
As I said before, I don't think it's really possible to factor an event loop into those kinds of parts. You may be able to factor the *API* that way, but any given implementation has to address all the parts at once. -- Greg

On 6 May 2015 at 09:20, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
It doesn't *need* them, but as abstractions they allow easier building of reusable higher-level libraries. You can write an event loop with nothing but coroutines, but to build reusable libraries on top of it, you need some common interfaces.
Agreed, you don't need a Future class, all you need is to agree what reusable code is allowed to do with the core objects you are passing around - that's how duck typing works. The objects *I* can see are futures (in a PEP 492 world, "awaitables" which may or may not be equivalent in terms of the operations you'd want to focus on) and the event loop itself. In your example, the event loop is implicit (as it's a singleton, you use global functions rather than methods on the loop object) but that's a minor detail.
So, "things you can wait on" have one operation - "wait for a result". That's OK. You can create such things as coroutines, which is also fine. You may want to create such things explicitly (equivalent to generators vs __iter__) - maybe that's where __aiter__ comes in in PEP 492 and the Future class in asyncio. Again, all fine. You also need operations like "schedule a thing to run", which is the event loop "interface". Your sample has the following basic event loop methods that I can see: run, schedule, unschedule, and expire_timeslice (that last one may be an implementation detail, but the other 3 seem pretty basic). PEP 492 has nothing to say on the event loop side of things (something that became clear to me during this discussion).
Futures seem to me to be (modulo a few details) what "awaitables" are in PEP 492. I can't see how you can meaningfully talk about event loops in a Python context without having *some* term for "things you wait for". Maybe Future wasn't a good name, and maybe the parallel with concurrent.futures.Future wasn't helpful (I think both things were fine, but you may not) but we had to have *some* way of talking about them, and of constructing standalone awaitables. PEP 492 has new, and hopefully better, ways, but I think that awaitables *have* to be central to any model where you wait for things... By the way, it feels to me like I'm now arguing in favour of PEP 492 with a reasonable understanding of what it "means". Assuming what I said above isn't complete rubbish, thanks to everyone who's helped me get to this point of understanding through this thread! (And if I haven't understood, that's my fault, and still thanks to everyone for their efforts :-)) Paul

Paul Moore wrote:
Take a look at the example I developed when working on the yield-from pep: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Exa... The first version of the event loop presented there does exactly that, just schedules tasks in a round- robin fashion. Then I gradually add more features to it.
I don't it's possible to answer that question, because there isn't a single answer. The minimal set of features that an event loop needs depends on what you want to achieve with it. Even the notion of "just schedules tasks" is ambiguous. What does "schedule" mean? Does it just mean round-robin switching between them, or should they be able to synchronise with each other in some way? Should it be possible for a task to suspend itself for an interval of real world time, or does that come under the heading of I/O (since you're waiting for an external event, i.e. the computer's clock reaching some time)? Etc.
And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?)
I don't think *any* particular event loop features are needed. You can't pick out any of these features as being "core". For example, it would be possible to have an event loop that handled socket I/O but *didn't* do round-robin scheduling -- it could just keep on running the same task, even if it yielded, until it blocked waiting for an external event. Such a scheduler would probably be quite adequate for many use cases. It seems to me that the idea of "generator-based tasks managed by an event loop" is more of a design pattern than something you can write a detailed API specification for. Another problem with the "core" idea is that you can't start with an event loop that "just does scheduling" and then add on other features such as I/O *from the outside*. There has to be some point at which everything comes together, which means choosing something like select() or poll() or I/O completion queues, and build that into the heart of your event loop. At that point it's no longer something with a simple core. -- Greg

On Tue, May 5, 2015 at 1:39 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Twisted has a pretty good taxonomy of event loop methods, in the interfaces at the bottom of this page: http://twistedmatrix.com/documents/15.1.0/core/howto/reactor-basics.html and the comparison matrix at http://twistedmatrix.com/documents/15.1.0/core/howto/choosing-reactor.html The asyncio event loops implement most of these (not the exact interfaces, but the same functionality). Tornado implements FDSet, Time, and part of Threads in the IOLoop itself, with the rest of the functionality coming from separate classes. (You may wonder then why Twisted and asyncio put everything in the core event loop? It's necessary to abstract over certain platform differences, which is one big reason why Tornado has poor support for Windows). -Ben

On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote:
Another reason people don't realize it is that the PEP goes out of its way to avoid saying so. I understand that you (and Yuri) don't want to tie the PEP too tightly to the specific event loop implementation in asyncio.events.AbstractEventLoop, but ... that particular conflation isn't really what people are confused about. "coroutines" often brings up thoughts of independent tasks. Yuri may well know that "(Python has asymmetric coroutines, that's it)", but others have posted that this was a surprise -- and the people posting here have far more python experience than most readers will. Anyone deeply involved enough to recognize that this PEP is only about (1) a particular type of co-routine -- a subset even of prior python usage (2) used for a particular purpose (3) coordinated via an external scheduler will already know that they can substitute other event loops. Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses "yield" as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

Jim, On 2015-05-05 5:09 PM, Jim J. Jewett wrote:
Thank you for this suggestion. I've added it to the PEP: https://hg.python.org/peps/rev/7ac132b24f1f Yury

Paul, On 2015-05-05 3:14 PM, Paul Moore wrote:
I agree. We have to improve asyncio docs in this area.
Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions.
There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user.
You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes.
There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient
It's chicken and egg problem. Right now, current coroutines via generators approach is cumbersome, it's harder to write async code than it should be. It stops the innovation in this area. Some languages like Go were specifically designed to make network programming easier, and they now steal users from Python. There is no absence of libraries for Go (and it's a new language!), btw. Give people the right tools and they will build what they need. Yury [1] https://github.com/KeepSafe/aiohttp

Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote:
Not just "How?", but "Why?". Why do they *need* to be disambiguated? With the benefit of having recently read all that discussion (as opposed to just the PEP), my answer is ... uh ... that generators vs "async def" is NOT an important distinction. What matters (as best I can tell) is: "something using yield (or yield from) to mark execution context switches" vs "other kinds of callables, including those using yield to make an iterator" I'm not quite sure that the actual proposal even really separates them effectively, in part because the terminology keeps suggesting other distinctions instead. (The glossary does help; just not enough.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

On 2015-05-05 5:31 PM, Jim J. Jewett wrote:
To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury

Hi Yury, On 05.05.2015 20:25, Yury Selivanov wrote:
I think monkeypatching and the gevent way is wrong. And explicit is better than implicit. I have to clear this. I meant how can we make this async stuff more accessible to the average sync user. Sometime even without writing or knowing how to write coroutines or other async stuff. Let me explain it with a deeper example (Some of it is related to Python 2 and twisted): I had the same problem for a server application using twisted. Provide easy interfaces to my users most not aware of async stuff. My solution was to write my own decorator similar to twisted's @inlineCallbacks. On top of it I added one more level to the decorator and distinguish if it was called from the main thread (also running the mainloop) and other threads. This object usable also as decorator is called "task" and has some utility methods. And returns a "deferred". (in asyncio this would be a Future) Resulting in code like: @task def myAsyncFunction(webaddress): result = yield fetch(webaddress) # only to show sleep example yield task.sleep(0.1) task.Return(result) Usable in a sync context (extra script thread): def do(): content = myAsyncFunction("http://xyz") or async context: @task def ado(): content = yield myAsyncFunction("http://xyz") The task decorator has functionality to check if something is called from the main thread (-> also a task and async) or it is called from another thread (-> sync or script). So this async interface is usable from both worlds. If someone operates async he/she must only know the task decorator and when to yield. If he/she uses it in sync mode nothing special has to be done. To allow all this the server starts the async main loop in the main thread and executes the script in an extra script thread. The user has every time his own thread, also for rpc stuff. The only way to switch into the main loop is to decorate a function as @task, every task is a coroutine and executed in the main thread (also thread of main loop). Benefit of all this: - Easy to write a async task it is marked as one and special stuff belongs to the task object. (task.Return is needed because we are in Python 2) - The normal user executes his stuff in his own thread and he/she can program in sync mode. No problem it is an extra thread and the main loop does not block. - A network client or other stuff has to be written only once, most time this can be a @task in the async world. But this should not care the end user. We don't have to implement all twice once for async and once for the sync world. -> Less overhead This is what I mean if I say we must address the bridging problem between the worlds. It think it is the wrong way to divide it in async and sync stuff and duplicate all networking libraries in the sync and async ones. For me the answer is to write one async netowrk library and use it in both, a sync script and in an async main loop. With an easy interface and not forcing the user to know this is an async library I have to do something special. And in future go one step further and use all this combined with PyParallel to solve the multiprocessing problem in Python. (Script in extra thread, mainloop in main thread, executed and managed via PyParallel avoiding the gil) But this is only a vision/dream of mine.
I do not. But for most they want only use it as a client and the main concern for most is "I want to get this web page" and not "I will implement a web client have to do this async and get it". Regards, Wolfgang

On Fri, May 01, 2015 at 09:24:47PM +0100, Arnaud Delobelle wrote:
I'm not convinced that allowing an object to be both a normal and an async iterator is a good thing. It could be a recipe for confusion.
In what way? I'm thinking that the only confusion would be if you wrote "async for" instead of "for", or vice versa, and instead of getting an exception you got the (a)syncronous behaviour you didn't want. But I have no intuition for how likely it is that you could write an asyncronous for loop, leave out the async, and still have the code do something meaningful. Other than that, I think it would be fine to have an object be both a syncronous and asyncronous iterator. You specify the behaviour you want by how you use it. We can already do that, e.g. unittest's assertRaises is both a test assertion and a context manager. Objects can have multiple roles, and it's not usually abused, or confusing. I'm not sure that async iterables will be any different. -- Steve

On 3 May 2015 at 16:24, Steven D'Aprano <steve@pearwood.info> wrote:
Yes. This is the same kind of confusion that this PEP is trying hard to get rid of in other parts (i.e. the confusion between 'yield' and 'yield from' in current coroutines).
Well if you've made you object both iterable and 'async iterable' then it's very likely that you're going to get something meaningful out of either kind of iteration. Just not the way you want it if you mistakenly left out (or added) the 'async' keyword in your loop.
The latter is fine, because there is no danger of mistaking one for the other, unlike iterators and 'async iterators'. But my argument is simply that there is no good reason to aim for the ability to have object conform to both protocols. So it shouldn't be a criterion when looking at the merits of a proposal. I may very well be wrong but I haven't yet seen a compelling case for an object implementing both protocols. Cheers, -- Arnaud

Let's say it this way: I want to know what I am looking at when I browse through the code -- an asynchronous iterator, or a normal iterator. I want an explicit difference between these protocols, because they are different. Moreover, the below code is a perfectly valid, infinite iterable: class SomeIterable: def __iter__(self): return self async def __next__(self): return 'spam' I'm strong -1 on this idea. Yury On 2015-05-01 3:03 PM, Stefan Behnel wrote:

On 2015-05-01 3:23 PM, Yury Selivanov wrote:
To further clarify on the example: class SomeIterable: def __iter__(self): return self async def __aiter__(self): return self async def __next__(self): print('hello') raise StopAsyncIteration If you pass this to 'async for' you will get 'hello' printed and the loop will be over. If you pass this to 'for', you will get an infinite loop, because '__next__' will return a coroutine object (that has to be also awaited, but it wouldn't, because it's a plain 'for' statement). This is something that we shouldn't let happen. Yury

Yury Selivanov schrieb am 01.05.2015 um 20:52:
I don't like the idea of combining __next__ and __anext__.
Ok, fair enough. So, how would you use this new protocol manually then? Say, I already know that I won't need to await the next item that the iterator will return. For normal iterators, I could just call next() on it and continue the for-loop. How would I do it for AIterators? Stefan

Stefan Behnel schrieb am 02.05.2015 um 06:54:
BTW, I guess that this "AIterator", or rather "AsyncIterator", needs to be a separate protocol (and ABC) then. Implementing "__aiter__()" and "__anext__()" seems perfectly reasonable without implementing (or using) a Coroutine. That means we also need an "AsyncIterable" as a base class for it. Would Coroutine then inherit from both Iterator and AsyncIterator? Or should we try to separate the protocol hierarchy completely? The section on "Coroutine objects" seems to suggest that inheritance from Iterator is not intended. OTOH, I'm not sure if inheriting from AsyncIterator is intended for Coroutine. The latter might just be a stand-alone ABC with send/throw/close, after all. I think that in order to get a better understanding of the protocol(s) that this PEP proposes, and the terminology that it should use, it would help to implement these ABCs. That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols. Stefan

On Sun May 3 08:32:02 CEST 2015, Stefan Behnel wrote:
Call next, then stick it somewhere it be waited on. Or is that syntactically illegal, because of the separation between sync and async? The "asych for" seems to assume that you want to do the waiting right now, at each step. (At least as far as this thread of the logic goes; something else might be happening in parallel via other threads of control.)
That means we also need an "AsyncIterable" as a base class for it.
Agreed.
That might even help us to decide if we need new builtins (or helpers) aiter() and anext() in order to deal with these protocols.
I hope not; they seem more like specialized versions of functions, such as are found in math or cmath. Ideally, as much as possible of this PEP should live in asycio, rather than appearing globally. Which reminds me ... *should* the "await" keyword work with any future, or is it really intentionally restricted to use with a single library module and 3rd party replacements? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

Yury Selivanov schrieb am 30.04.2015 um 03:30:
1. Terminology: - *native coroutine* term is used for "async def" functions.
When I read "native", I think of native (binary) code. So "native coroutine" sounds like it's implemented in some compiled low-level language. That might be the case (certainly in the CPython world), but it's not related to this PEP nor its set of examples.
We should discuss how we will name new 'async def' coroutines in Python Documentation if the PEP is accepted.
Well, it doesn't hurt to avoid obvious misleading terminology upfront. Stefan

On Fri, May 1, 2015 at 5:50 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
I think "obvious[ly] misleading" is too strong, nobody is trying to mislead anybody, we just have different associations with the same word. Given the feedback I'd call "native coroutine" suboptimal (even though I proposed it myself) and I am now in favor of using "async function". -- --Guido van Rossum (python.org/~guido)

On 1 May 2015 at 16:31, Guido van Rossum <guido@python.org> wrote:
But what if you have async methods? I know, a method is almost a function, but still, sounds slightly confusing. IMHO, these are really classical coroutines. If gevent calls them coroutines, I don't think asyncio has any less right to call them coroutines. You have to ask yourself this: a new programmer, when he sees mentions of coroutines, how likely is he to understand what he is dealing with? What about "async function"? The former is a well known concept, since decades ago, while the latter is something he probably (at least me) never heard of before. For me, an async function is just as likely to be an API that is asynchronous in the sense that it takes an extra "callback" parameter to be called when the asynchronous work is done. I think coroutine is the name of a concept, not a specific implementation. Cheers, -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert

On Fri, May 1, 2015 at 8:55 AM, Gustavo Carneiro <gjcarneiro@gmail.com> wrote:
we have used this term ever since PEP 342. But when we're talking specifics and trying to distinguish e.g. a function declared with 'async def' from a regular function or from a regular generator function, using 'async function' sounds right. And 'async method' if it's a method. -- --Guido van Rossum (python.org/~guido)

On 5/1/2015 9:59 AM, Guido van Rossum wrote:
Exactly. The async function/method is an implementation technique for a specific kind/subset of coroutine functionality. So the term coroutine, qualified by a description of its best usage and limitationsof async function, can be used in defining async function, thus appealing to what people know or have heard of and vaguely understand and can read more about in the literature. A glossary entry for coroutine in the docs seems appropriate, which could point out the 16† ways to implement coroutine-style functionalities in Python, and perhaps make recommendations for different types of usage. †OK, not 16 ways, but it is 3 now, or 4?
participants (18)
-
Arnaud Delobelle
-
Ben Darnell
-
Brett Cannon
-
Ethan Furman
-
Glenn Linderman
-
Greg Ewing
-
Guido van Rossum
-
Gustavo Carneiro
-
Jim J. Jewett
-
Koos Zevenhoven
-
Nathaniel Smith
-
Paul Moore
-
Stefan Behnel
-
Steven D'Aprano
-
tds333@gmail.com
-
Wolfgang
-
Wolfgang Langner
-
Yury Selivanov