PEP 525: Asynchronous Generators
Hi, This is a new PEP to add asynchronous generators to Python 3.6. The PEP is also available at [1]. There is a reference implementation [2] that supports everything that the PEP proposes to add. [1] https://www.python.org/dev/peps/pep-0525/ [2] https://github.com/1st1/cpython/tree/async_gen Thank you! PEP: 525 Title: Asynchronous Generators Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury@magic.io> Discussions-To: <python-dev@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6 Post-History: 02-Aug-2016 Abstract ======== PEP 492 introduced support for native coroutines and ``async``/``await`` syntax to Python 3.5. It is proposed here to extend Python's asynchronous capabilities by adding support for *asynchronous generators*. Rationale and Goals =================== Regular generators (introduced in PEP 255) enabled an elegant way of writing complex *data producers* and have them behave like an iterator. However, currently there is no equivalent concept for the *asynchronous iteration protocol* (``async for``). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements ``__aiter__`` and ``__anext__`` to be able to use it in an ``async for`` statement. Essentially, the goals and rationale for PEP 255, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are **2x** faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated:: class Ticker: """Yield numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i The same can be implemented as a much simpler asynchronous generator:: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) Specification ============= This proposal introduces the concept of *asynchronous generators* to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). Asynchronous Generators ----------------------- A Python *generator* is any function containing one or more ``yield`` expressions:: def func(): # a function return def genfunc(): # a generator function yield We propose to use the same approach to define *asynchronous generators*:: async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield 42 The result of calling an *asynchronous generator function* is an *asynchronous generator object*, which implements the asynchronous iteration protocol defined in PEP 492. It is a ``SyntaxError`` to have a non-empty ``return`` statement in an asynchronous generator. Support for Asynchronous Iteration Protocol ------------------------------------------- The protocol requires two special methods to be implemented: 1. An ``__aiter__`` method returning an *asynchronous iterator*. 2. An ``__anext__`` method returning an *awaitable* object, which uses ``StopIteration`` exception to "yield" values, and ``StopAsyncIteration`` exception to signal the end of the iteration. Asynchronous generators define both of these methods. Let's manually iterate over a simple asynchronous generator:: async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 await gen.__anext__() # This line will raise StopAsyncIteration. Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. 3. Add two new methods to the ``sys`` module: ``set_asyncgen_finalizer()`` and ``get_asyncgen_finalizer()``. The idea behind ``sys.set_asyncgen_finalizer()`` is to allow event loops to handle generators finalization, so that the end user does not need to care about the finalization problem, and it just works. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_finalizer(old_finalizer) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) ``sys.set_asyncgen_finalizer()`` is thread-specific, so several event loops running in parallel threads can use it safely. Asynchronous Generator Object ----------------------------- The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: 1. ``agen.__aiter__()``: Returns ``agen``. 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one asynchronous generator iteration when awaited. 3. ``agen.asend(val)``: Returns an *awaitable*, that pushes the ``val`` object in the ``agen`` generator. When the ``agen`` has not yet been iterated, ``val`` must be ``None``. Example:: async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.2) g = gen() await g.asend(None) # Will return 42 after sleeping # for 0.1 seconds. await g.asend('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.2 seconds.) 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that throws an exception into the ``agen`` generator. Example:: async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after # sleeping for 0.1 seconds. v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after $ sleeping 0.2 seconds. 5. ``agen.aclose()``: Returns an *awaitable*, that throws a ``GeneratorExit`` exception into the generator. The *awaitable* can either return a yielded value, if ``agen`` handled the exception, or ``agen`` will be closed and the exception will propagate back to the caller. 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable name and qualified name attributes. 7. ``agen.ag_await``: The object that ``agen`` is currently *awaiting* on, or ``None``. This is similar to the currently available ``gi_yieldfrom`` for generators and ``cr_await`` for coroutines. 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: defined in the same way as similar attributes of standard generators. ``StopIteration`` and ``StopAsyncIteration`` are not propagated out of asynchronous generators, and are replaced with a ``RuntimeError``. Implementation Details ---------------------- Asynchronous generator object (``PyAsyncGenObject``) shares the struct layout with ``PyGenObject``. In addition to that, the reference implementation introduces three new objects: 1. ``PyAsyncGenASend``: the awaitable object that implements ``__anext__`` and ``asend()`` methods. 2. ``PyAsyncGenAThrow``: the awaitable object that implements ``athrow()`` and ``aclose()`` methods. 3. ``_PyAsyncGenWrappedValue``: every directly yielded object from an asynchronous generator is implicitly boxed into this structure. This is how the generator implementation can separate objects that are yielded using regular iteration protocol from objects that are yielded using asynchronous iteration protocol. ``PyAsyncGenASend`` and ``PyAsyncGenAThrow`` are awaitables (they have ``__await__`` methods returning ``self``) and are coroutine-like objects (implementing ``__iter__``, ``__next__``, ``send()`` and ``throw()`` methods). Essentially, they control how asynchronous generators are iterated: .. image:: pep-0525-1.png :align: center :width: 80% PyAsyncGenASend and PyAsyncGenAThrow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``PyAsyncGenASend`` is a coroutine-like object that drives ``__anext__`` and ``asend()`` methods and implements the asynchronous iteration protocol. ``agen.asend(val)`` and ``agen.__anext__()`` return instances of ``PyAsyncGenASend`` (which hold references back to the parent ``agen`` object.) The data flow is defined as follows: 1. When ``PyAsyncGenASend.send(val)`` is called for the first time, ``val`` is pushed to the parent ``agen`` object (using existing facilities of ``PyGenObject``.) Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 2. When ``PyAsyncGenASend.throw(*exc)`` is called for the first time, ``*exc`` is throwed into the parent ``agen`` object. Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 3. ``return`` statements in asynchronous generators raise ``StopAsyncIteration`` exception, which is propagated through ``PyAsyncGenASend.send()`` and ``PyAsyncGenASend.throw()`` methods. ``PyAsyncGenAThrow`` is very similar to ``PyAsyncGenASend``. The only difference is that ``PyAsyncGenAThrow.send()``, when called first time, throws an exception into the parent ``agen`` object (instead of pushing a value into it.) New Standard Library Functions and Types ---------------------------------------- 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator object. 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` methods to set up asynchronous generators finalizers in event loops. 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` introspection functions. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` function with a ``yield`` expression inside, therefore it's safe to introduce asynchronous generators in 3.6. Performance =========== Regular Generators ------------------ There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators:: def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) Improvements over asynchronous iterators ---------------------------------------- The following micro-benchmark shows that asynchronous generators are about **2.3x faster** than asynchronous iterators implemented in pure Python:: N = 10 ** 7 async def agen(): for i in range(N): yield i class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i Design Considerations ===================== ``aiter()`` and ``anext()`` builtins ------------------------------------ Originally, PEP 492 defined ``__aiter__`` as a method that should return an *awaitable* object, resulting in an asynchronous iterator. However, in CPython 3.5.2, ``__aiter__`` was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: ``__aiter__`` can still return an *awaitable* with a ``DeprecationWarning`` being issued. Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot add a synchronous implementation of ``aiter()`` built-in. Therefore, it is proposed to wait until Python 3.7. Asynchronous list/dict/set comprehensions ----------------------------------------- Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. Asynchronous ``yield from`` --------------------------- While it is theoretically possible to implement ``yield from`` support for asynchronous generators, it would require a serious redesign of the generators implementation. ``yield from`` is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. And to compose asynchronous generators a simple ``async for`` loop can be used:: async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v Why the ``asend()`` and ``athrow()`` methods are necessary ---------------------------------------------------------- They make it possible to implement concepts similar to ``contextlib.contextmanager`` using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern:: @async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from ``__anext__`` object, but it is hard to do that correctly. Adding explicit ``asend()`` and ``athrow()`` will pave a safe way to accomplish that. In terms of implementation, ``asend()`` is a slightly more generic version of ``__anext__``, and ``athrow()`` is very similar to ``aclose()``. Therefore having these methods defined for asynchronous generators does not add any extra complexity. Example ======= A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay):: async def ticker(delay, to): for i in range(to): yield i await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() Implementation ============== The complete reference implementation is available at [1]_. References ========== .. [1] https://github.com/1st1/cpython/tree/async_gen Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Hi! I'm all for this. I've run into it so many times already while implementing the async/await support in Cython, it's really a totally obvious extension to what there is currently, and it's practically a requirement for any serious usage of async iterators. Some comments below. Yury Selivanov schrieb am 03.08.2016 um 00:31:
Well, or *something* that uses them in the same way as an event loop would. Doesn't have to be an event loop.
I don't see a motivation for adding an "aclose()" method in addition to the normal "close()" method. Similar for send/throw. Could you elaborate on that?
Phew, this adds quite some complexity and magic. That is a problem. For one, this uses a global setup, so There Can Only Be One of these finalizers. ISTM that if special cleanup is required, either the asyncgen itself should know how to do that, or we should provide some explicit API that does something when *initialising* the asyncgen. That seems better than doing something global behind the users' back. Have you considered providing some kind of factory in asyncio that wraps asyncgens or so? I won't go into the implementation details for now, but thanks for the PEP and for working on this so actively. Stefan
Hi Stefan! On 2016-08-03 2:45 AM, Stefan Behnel wrote:
Sure, I'm just using the same terminology PEP 492 was defined with. We can say "coroutine runner" instead of "event loop".
There will be no "close", "send" and "throw" defined for asynchronous generators. Only their asynchronous equivalents. This topic is actually quite complex, so bear with me. 1. It is important to understand that asynchronous iteration protocol is multiplexed into normal iteration protocol. For example: @types.coroutine def foobar(): yield 'spam' async def agen(): await foobar() yield 123 The 'agen' generator, on the lowest level of generators implementation will yield two things -- 'spam', and a wrapped 123 value. Because 123 is wrapped, the async generators machinery can distinguish async yields from normal yields. The idea behind __anext__ coroutine is that it yields through all "normal" yields, and raises a StopIteration when it encounters a "wrapped" yield (same idea behind aclose(), athrow(), and asend()) 2. Now let's look at two generators (sync and async): def gen(): async def agen(): try: try: ... ... finally: finally: fin1() await afin1() fin2() await afin2() yield 123 yield 123 * If we call 'gen().close()' when gen() is suspended somewhere in its try block, a GeneratorExit exception will be thrown in it. Then, fin1() and fin2() calls will be executed. Then, a 'yield 123' line will happen, which will cause a RuntimeError('generator yielded while closing'). The reason for the RuntimeError is that the interpreter does not want generators to yield in their finally statements. It wants them to synchronously finalize themselves. Yielding while closing doesn't make any sense. * Now, if we would just reuse the synchronous 'close()' method for 'agen()' -- awaiting on 'afin1()' would simply result in a RuntimeError('generator yielded while closing'). So the close() implementation for agen() must allow some yields, to make 'await' expressions possible in the finally block. This is something that is absolutely required, because instead of try..finally you could have 'async with' block in agen() -- so the ability to call asynchronous code in the 'finally' block is very important. Therefore, it's necessary to introduce a new close semantics for asynchronous generators: - It is OK to 'await' on anything in finally blocks in async generators. - Trying to 'yield' in finally blocks will result in a RuntimeError('async generator yielded while closing') -- similarly to sync generators. - Because we have to allow awaits in generator's finally blocks, the new 'close' method has to be a coroutine-like object. Since all this is quite different from sync generators' close method, it was decided to have a different name for this method for async generators: aclose. aclose() is a coroutine-like object, you can await on it, and you can even throw a CancelledError into it; so it's possible to write 'await asyncio.wait_for(agen.aclose(), timeout=1)'. 3. asend() and athrow(). This is very similar to aclose(). Both have to be coroutines because async yields are multiplexed into normal yields that awaitables use behind the scenes. async def foo(): await asyncio.sleep(1) yield 123 If we had a synchronous send() method defined for foo(), you'd see something like this: gen = foo() gen.send(None) -> <asyncio.Future> Instead, what you really want is this: gen = foo() await gen.asend(None) -> 123 4. I really recommend you to play with the reference implementation. You can emulate synchronous "send" and "throw" buy doing this trick: gen = foo() gen.__anext__().send() gen.__anext__().throw()
set_asyncgen_finalizer is thread-specific, so you can have one finalizer set up per thread. The reference implementation actually integrates this all into asyncio. The idea is to setup loop async gens finalizer just before the loop starts, and reset the finalizer to the previous one (usually it's None) just before it stops. The finalizer is attached to a generator when it is yielding for the first time -- this guarantees that every generators will have a correct finalizer attached to it. It's not right to attach the finalizer (or wrap the generator) when the generator is initialized. Consider this code: async def foo(): async with smth(): yield async def coro(gen): async for i in foo(): ... loop.run_until_complete(coro(foo())) ^^ In the above example, when the 'foo()' is instantiated, there is no loop or finalizers set up at all. BUT since a loop (or coroutine wrapper) is required to iterate async generators, there is a strong guarantee that it *will* present on the first iteration. Regarding "async gen itself should know how to cleanup" -- that's not possible. async gen could just have an async with block and then GCed (after being partially consumed). Users won't expect to do anything besides using try..finally or async with, so it's the responsibility of the coroutine runner to cleanup async gen. Hence 'aclose' is a coroutine, and hence this set_asyncgen_finalizer API for coroutine runners. This is indeed the most magical part of the proposal. Although it's important to understand that the regular Python users will likely never encounter this in their life -- finalizers will be set up by the framework they use (asyncio, Tornado, Twisted, you name it). Thanks! Yury
Yury Selivanov schrieb am 03.08.2016 um 17:32:
Ok, why not. Different names for similar things that behave differently enough.
This is actually going to be tricky to backport for Cython (which supports Py2.6+) since it seems to depend on a globally known C implemented wrapper object type. We'd have to find a way to share that across different packages and also across different Cython versions (types are only shared within the same Cython version). I guess we'd have to store a reference to that type in some well hidden global place somewhere, and then never touch its implementation again... Is that wrapper type going to be exposed anywhere in the Python visible world, or is it purely internal? (Not that I see a use case for making it visible to Python code...) BTW, why wouldn't "async yield from" work if the only distinction point is whether a yielded object is wrapped or not? That should work at any level of delegation, shouldn't it?
Correct. And it also wouldn't help to generally extend the Async-Iterator protocol with an aclose() method because ending an (async-)for loop doesn't mean we are done with the async iterator, so this would just burden the users with unnecessary cleanup handling. That's unfortunate...
I think my main problem is that you keep speaking about event loops (of which There Can Be Only One, by design), whereas coroutines are a much more general concept and I cannot overlook all possible forms of using them in the future. What I would like to avoid is the case where we globally require setting up one finalizer handler (or context), and then prevent users from doing their own cleanup handling in some module context somewhere. It feels to me like there should be some kind of stacking for this (which in turn feels like a context manager) in order to support adapters and the like that need to do their own cleanup handling (or somehow intercept the global handling), regardless of what else is running. But I am having a hard time trying to come up with an example where the thread context really doesn't work. Even if an async generator is shared across multiple threads, then those threads would most likely be controlled by the coroutine runner, which would simply set up a proper finalizer context for each of the threads that points back to itself, so that it would not be the pure chance of first iteration to determine who gets to clean up afterwards. It's possible that such (contrieved?) cases can be handled in one way or another by changing the finalization itself, instead of changing the finalizer context. And that would be done by wrapping async iterators in a delegator that uses try-finally, be it via a decorator or an explicit wrapper. It's just difficult to come to a reasonable conclusion at this level of uncertainty. That's why I'm bouncing this consideration here, in case others have an idea how this problem could be avoided all together. Stefan
On 2016-08-06 4:29 AM, Stefan Behnel wrote:
I don't think you need to care about these details. You can implement async generators using async iteration protocol as it was defined in PEP 492. You can compile a Cython async generator into an object that implements `am_aiter` slot (which should return `self`) and `am_anext` slot (which should return a coroutine-like object, similar to what Cython compiles 'async def'). `am_anext` coroutine should "return" (i.e. raise `StopIteration`) when it reaches an "async yield" point, or raise "StopAsyncIteration" when the generator is exhausted. Essentially, because pure Python async generators work fine with 'async for' and will continue to do so in 3.6, I think there shouldn't be technical problems to add asynchronous generators in Cython.
Yes, it's a purely internal C-level thing. Python code will never see it.
I can try ;) Although when I was working on the PEP I had a feeling that this won't work without a serious refactoring of genobject.c.
Well, we have the *exact same thing* with regular (synchronous) iterators. Let's say you have an iterable object (with `__iter__` and `__next__`), which cleanups resources at the end of its iteration. If you didn't implement its __del__ properly (or at all), the resources won't be cleaned-up when it's partially iterated and then GCed.
We can call the thing that runs coroutines a "coroutine runner". I'll update the PEP. Regardless of the naming issues, I don't see any potential problem with how `set_asyncgen_finalizer` is currently defined: 1. Because a finalizer (set with `set_asyncgen_finalizer`) is assigned to generators on their first iteration -- it's guaranteed that each async generator will have a correct one attached to it. 2. It's extremely unlikely that somebody will design a system that switches coroutine runners *while async/awaiting a coroutine*. There are no good reasons to do this, and I doubt that it's even a possible thing. But even in this unlikely use case, you can easily stack finalizers following this pattern: old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(my_finalizer) try: # do my thing finally: sys.set_asyncgen_finalizer(old_finalizer) Thanks, Yiry
Yury Selivanov schrieb am 06.08.2016 um 18:39:
Hmm, I thought that at least interoperability might get in the way. I guess I'll just have to give it a try...
2. It's extremely unlikely that somebody will design a system that switches coroutine runners *while async/awaiting a coroutine*.
Yes, I guess so.
That only works for synchronous code, though, because if this is done in a coroutine, it might get suspended within the try block and would leak its own finalizer into the outer world. Stefan
On 2016-08-06 1:03 PM, Stefan Behnel wrote:
Maybe, if we add 'yield from' (which I really don't want to bother with in 3.6 at least). It'd be extremely helpful if you could prototype the PEP in Cython!
set_asyncgen_finalizer is designed to be used *only* by coroutine runners. This is a low-level API that coroutines should never touch. (At least my experience working with coroutines says so...) Thank you, Yury
Hey Yury, that's a great proposal! On 03.08.2016 00:31, Yury Selivanov wrote:
That's a great motivational example. +1 Especially when reading the venerable PEP 255 (Q/A part), this also gets rid of the "low-level detail": "raise StopAsyncIteration". This could also be worth mentioning in the motivation part for PEP 525.
From your answer to Stefan, I get the impression that the reason why we actual need all those a* methods (basically a duplication of the existing gen protocol), is the fact that normal generators can be converted to coroutines. That means, 'yield' still can be used in both ways. So, it's a technical symptom of the backwards-compatibility rather than something that cannot be avoided by design. Is this correct? If it's correct, would you think it would make sense to get rid of the a* in a later iteration of the async capabilities of Python? So, just using the normal generator protocol again? One note on all examples but the last. Reading those examples, it creates the illusion of actual working code which is not the case, right? One would always need to 1) wrap module-level statements into its own coroutine, 2) create an event-loop and 3) run it. Do you think clarifying this in the PEP makes sense? Thanks again for your hard work here. Async generators definitely completes the picture. Sven
Hi Sven, On 2016-08-04 9:20 AM, Sven R. Kunze wrote:
Thanks! [..]
async/await in Python is implemented on top of the generator protocol. Any 'await' is either awaiting on a coroutine or on a Future-like object. Future-like objects are defined by implementing the __await__ method, which should return a generator. So coroutines and generators are very intimately tied to each other, and that's *by design*. Any coroutine that iterates over an asynchronous generator uses the generator protocol behind the scenes. So we have to multiplex the async generaotor's "yields" into the generator protocol in such a way, that it stays isolated, and does not interfere with the "yields" that drive async/await.
Because async generators will contain 'await' expressions, we have to have a* methods (although we can name them without the "a" prefix, but I believe that would be confusing for many users).
I'll think about this, thanks! Maybe I can add a line in the beginning of the "Specification" section.
Thanks again for your hard work here. Async generators definitely completes the picture.
Thank you, Sven Yury
As maintainer of aiohttp (asyncio-compatible HTTP client/server) and aiopg (PostgreSQL asyncio driver) I totally support the pep. The problem is: I'm able to write async iterator class which utilizes __aiter__/__anext__. But users of my libraries are not. Writing synchronous generator functions for data processing is very common pattern. But class based approach requires saving all context not in local variables but in instance vars, you know. It's a nightmare in practice, as consequence people writes more complex and error prone code than they could write with this PEP. https://www.python.org/dev/peps/pep-0255/ for simple generators is 15 years old, I believe everybody understand why we need it. Async generators extend the same concept to asyncio world by solving the same problems but with async code. Honestly I personally don't feel a need for two-way generators with `asend()`/`athrow()` but as Yury explained he need them internally for `anext`/`aclose` anyway. On Thu, Aug 4, 2016 at 6:18 PM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
-- Thanks, Andrew Svetlov
Hi Yury! Thanks for this PEP! This is really what asyncio misses since you made async iterators. As contributor to aiohttp and maintainer of aiocouchdb and few else projects I also support this PEP. Great one! I agree with what Nikolay and Andrew already said: dancing around__aiter__/__anext__ to emulate genarator-like behaviour is quite boring and complicated. This also makes porting "old synced" code onto asyncio harder since you have to rewrite every generator in iterator fashion and wrap each with an object that provides __aiter__ / __anext__ interface. For the reference I have is ijson PR[1] that adds asyncio support. If we forget about compatibility with 3.3/3.4, with async generators the result implementation would be much more elegant and looks more closer to the original synced code, because it's all based on generators. Lack of async generators forces people to implement them on their own [2] ones[3], not very effective as the one that could be built-in. Also custom interface for each implementation makes hardly to reuse them and don't even imagine what happens if two custom async generator implementations will met in a single project. So thank you again, Yury, for pushing this really important and missing feature forward! [1]: https://github.com/isagalaev/ijson/pull/46 [2]: https://github.com/germn/aiogen [3]: https://github.com/ethanfrey/aiojson/blob/master/aiojson/utils/aiogen.py -- ,,,^..^,,,
On 2016-08-05 4:34 PM, Alexander Shorin wrote:
Thank you Alexander for the feedback! It's really great that asyncio users begin to share their experience in this thread. Indeed, the key idea is to complete async/await implementation, so that you can think in async/await terms and use simply use existing Python idioms. Want a context manager in your asyncio application? Use 'async with'. Want an iterator? Use 'async for' and async generators. Hopefully we'll get there. Thank you, Yury
On 6 August 2016 at 01:11, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Even if asend()/athrow() are cheap from an implementation perspective, they're *not* necessarily cheap from a "cognitive burden of learning the API" perspective. We know what send() and throw() on generators are for: implementing coroutines. However, we also know that layering that on top of the generator protocol turned out to be inherently confusing, hence PEP 492 and async/await. So if we don't have a concrete use case for "coroutine coroutines" (what does that even *mean*?), we shouldn't add asend()/athrow() just because it's easy to do so. If we wanted to hedge our bets, we could expose them as _asend() and _athrow() initially, on the grounds that doing so is cheap, and we're genuinely curious to see if anyone can find a non-toy use case for them. But we should be very reluctant to add them to the public API without a clear idea of why they're there. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, On 2016-08-06 4:28 AM, Nick Coghlan wrote:
One immediate use case that comes to mind is @contextlib.asynccontextmanager Essentially this: @contextlib.asynccontextmanager async def ctx(): resource = await acquire() try: yield resource finally: await release(resource) I'm also worrying that in order to do similar things people will abuse "agen.__anext__().send()" and "agen.__anext__().throw()". And this is a very error-prone thing to do. For example, in asyncio code, you will capture yielded Futures that the generator is trying to await on (those yielded futures have to be captured by the running Task object instead). "asend" and "athrow" know about iteration protocol and can see the "wrapped" yielded objects, hence they can iterate the generator correctly and reliably. I think Nathaniel Smith (CC-ed) had a few more use cases for asend/athrow.
I think we should decide if we want to expose them at all. If we're in doubt, let's not do that at all. Exposing them with a "_" prefix doesn't free us from maintaining backwards compatibility etc. You're completely right about the increased "cognitive burden of learning the API". On the other hand, send() and throw() for sync generators were designed for power users; most Python developers don't even know that they exist. I think we can view asend() and athrow() similarly. Thank you, Yury
While Nick has a point about coroutine coroutines confusion. I consider asend/athow will be useful in real world. Because they provide stable, well known and uniform interface to communicate with a coroutine. For now to have two-way communication with some coroutine you have to establish two queues: inbox and outbox, pass them around, don't eventually mess them and so on. Queues are great for message-passing communication and generally good idea to use, but it still some boilerplate code as like as __aiter__/__anext__ (which async generators will get rid) to maintain. Also queue is a bad reference to a coroutine owns it - it's hard to find own who is really reading that inbox in complex codebase. With asend/athrow there will be no need to plass queues around, but just coroutine instance and do all the communication with it as with plain object. You need to retrieve a value - do iterate over it; need to send a value back - do asend. Termination and error-passing are aslo easy to do. So while this interface looks complicated for learning curve, it could be very helpful, when you need simple communication channel between coroutines sized equals one. For something greater you have to return to queues approach, but they involve quite a lot of else bits to care about like queue overflow, which is indeed topic for advanced users. -- ,,,^..^,,, On Sat, Aug 6, 2016 at 7:51 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On 7 August 2016 at 02:51, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Good point. Let's make adding that part of the PEP, since it will be useful, makes it immediately obvious why we're including asend() and athrow(), and will also stress test the coroutine traceback machinery in an interesting way. (Note that depending on the implementation details, it may need to end up living in asyncio somewhere - that will depend on whether or not we need to use any asyncio APIs in the implementation of _CoroutineContextManager) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2016-08-07 11:53 AM, Nick Coghlan wrote:
It's already in the PEP: https://www.python.org/dev/peps/pep-0525/#why-the-asend-and-athrow-methods-a... Although it's not super visible, I agree. Should I try to make that section more prominent? Thank you, Yury
Hi Yury, Thank you for posting this PEP! As an asyncio-based libraries user, author and contributor I appreciate that topic of easy-writing of asynchronous generators is being covered now. I like how simple writing of async generators will be with this PEP, same with consequent functionality (e.g. async context wrappers). It's great that with this PEP it would be possible to write simple async data processing wrappers without a knowledge of a single special method (__aiter__, __anext__, __aenter__, __aexit__) using same patterns as regular Python user would use in synchronous code: @async_context_manager async def connection(uri): conn = await connect(uri) try: yield conn finally: conn.close() await conn.wait_closed() async def query_data(conn, q): cursor = await conn.query(q) try: row = await cursor.fetch_row() while row is not None: yield row row = await cursor.fetch_row() finally: cursor.close() await cursor.wait_closed() async with connection(uri) as conn: async for item in query_data(q): print(item) And this is as simple as in synchronous code, but made asynchronous. Can you explain details of canceling of asend and await coroutines? Consider following example: async def gen(): zero = yield 0 asyncio.sleep(10) one = yield 1 async def f(loop): g = gen() await g.asend(None) send_task = loop.create_task(g.asend('zero')) send_task.cancel() # What is the state of g now? # Can I receive one from g? Or cancel() will lead to CancelledError throw from asyncio.sleep(10)? one = g.asend('one') Will canceling of asend() task result in canceling of generator? Thanks, Vladimir Rutsky
Hi Vladimir, On 2016-08-09 1:24 PM, Vladimir Rutsky wrote:
Thanks a lot for reaching out!
Precisely this.
Cancelling `asend()` will result in an asyncio.CancelledError being raised from the 'await asyncio.sleep(0)' line inside the async generator: async def gen(): try: await asyncio.sleep(1) except asyncio.CancelledError: print('got it') raise # It's OK to not to re-raise the error too! yield 123 async def run(): g = gen() t = asyncio.ensure_future(g.asend(None)) await asyncio.sleep(0.5) t.cancel() await t # this line will fail with CancelledError ^ the above code will print "got it" and then crash with an asyncio.CancelledError raised back from 'await t'. You can handle the error in the generator and continue to iterate over it. If you don't catch that exception inside your generator, the generator will be closed (it's similar to what would happen if an exception occurs inside a sync generator, or is thrown into it with gen.throw()). Thank you, Yury
On 04.08.2016 17:17, Yury Selivanov wrote:
Yes, that is how I understand it as well. So, all this complexity stems from the intertwining of generators and coroutines. I am wondering if loosening the tie between the two could make it all simpler; even for you. (Not right now, but maybe later.)
I think that is related to Stefan's question here. I cannot speak for him but it seems to me the confusion is actually the other way around. Coming from a database background, I prefer redundant-free data representation. The "a" prefix encodes the information "that's an asynchronous generator" which is repeated several times and thus is redundant. Another example: when writing code for generators and checking for the "send" method, I would then need to check for the "asend" method as well. So, I would need to touch that code although it even might not have been necessary in the first place. I don't want to talk you out of the "a" prefix, but I get the feeling that the information "that's asynchronous" should be provided as a separate attribute. I believe it would reduce confusion, simplify duck-typing and flatten the learning curve. :)
You're welcome. :) Sven
On 2016-08-05 4:51 PM, Sven R. Kunze wrote:
Unfortunately, the only way to "loosen the tie" is to re-implement generator protocol for coroutines (make them a separate thing). Besides breaking backwards compatibility with the existing Python code that uses @coroutine decorator and 'yield from' syntax, it will also introduce a ton of work.
Thing is you can't write one piece of code that will accept any type of generator (sync or async). * send, throw, close, __next__ methods for sync generators, they are synchronous. * send, throw, close, __next__ methods for async generators, they *require* to use 'await' on them. There is no way to make them "synchronous", because you have awaits in async generators. Because of the above, duck-typing simply isn't possible. The prefix is there to make people aware that this is a completely different API, even though it looks similar. Thank you, Yury
On 06.08.2016 01:25, Yury Selivanov wrote:
You are the best to know that. :)
There are a lot of things you can do with generators (decorating, adding attributes etc.) and none of them require you to "make them synchronous".
Because of the above, duck-typing simply isn't possible.
See above.
The prefix is there to make people aware that this is a completely different API, even though it looks similar.
Sure. So, what do you think about the separate attribute to make people aware? Best, Sven
Sure. So, what do you think about the separate attribute to make people aware?
What people should be aware about? IMHO sync generators and async ones have different types. What also is needed? On Mon, Aug 8, 2016 at 2:45 PM Sven R. Kunze <srkunze@mail.de> wrote:
-- Thanks, Andrew Svetlov
On 2016-08-08 7:44 AM, Sven R. Kunze wrote:
You have to be aware of what you're decorating. Always. For instance, here's an example of a buggy code: @functools.lru_cache() def foo(): yield 123 I'm really against duck typing here. I see no point in making the API for async generators to look similar to the API of sync generators. You have to provide a solid real-world example of where it might help with asynchronous generators to convince me otherwise ;)
There is a separate attribute already -- __class__. Plus a couple of new functions in inspect module: inspect.isasyncgenfunction and inspect.isasyncgen. And the new types.AsyncGeneratorType for isinstance checks. Thanks, Yury
On 08.08.2016 19:06, Yury Selivanov wrote:
You have to be aware of what you're decorating. Always.
You have to be aware of what the decorator does first before you decide if it's relevant to be aware of what you're decorating.
And so do I to making them "almost similar" by putting an "a" in front of the well-known generator protocol. But maybe, because there is not real convincing argument on either side, it doesn't matter at all.
You have to provide a solid real-world example of where it might help with asynchronous generators to convince me otherwise ;)
Something, I learned in the past years is to be prepared. So, such solid real-world examples might emerge when async generators are out in the wild. Let's hope this does not make it harder for them to start with. :)
Got it, thanks. Just another random thought I want to share: From what I've heard in the wild, that most if not all pieces of async are mirroring existing Python features. So, building async basically builds a parallel structure in Python resembling Python. Async generators complete the picture. Some people (including me) are concerned by this because they feel that having two "almost the same pieces" is not necessarily a good thing to have. And not necessarily bad but it feels like duplicating code all over the place especially as existing functions are incompatible with async. As I understand it, one goal is to be as close to sync code as possible to make async code easier to understand. So, if dropping 'await's all over the place is the main difference between current async and sync code, I get the feeling both styles could be very much unified. (The examples of the PEP try to do it like this but it wouldn't work as we already discussed.) Maybe, it's too early for a discussion about this but I wanted to dump this thought somewhere. :) Thanks, Sven
On 9 August 2016 at 08:37, Sven R. Kunze <srkunze@mail.de> wrote:
It's a known problem that applies to programming language design in general rather than being Python specific: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ The async/await model used in C# and Python 3.5+ aims to at least make it clear whether you're dealing with "red" (asynchronous) or "blue" (synchronous) code, but some use cases will still benefit from using something like gevent to bridge between the synchronous world and the asynchronous one: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... In cases where blocking for IO is acceptable, the "run_until_complete()" method on event loop implementations can be used in order to invoke asynchronous code synchronously: http://www.curiousefficiency.org/posts/2015/07/asyncio-background-calls.html
One of the big mindset shifts it encourages is to design as many support libraries as possible as computational pipelines and message-drive state machines, rather than coupling them directly to IO operations (which is the way many of them work today). Brett Cannon started the Sans IO information project to discuss this concept at http://sans-io.readthedocs.io/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Just don't oversell run_until_complete() -- some people coming from slightly different event loop paradigms expect to be able to pump for events at any point, possibly causing recursive invocations. That doesn't work here (and it's a feature it doesn't). On Mon, Aug 8, 2016 at 8:23 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 9 August 2016 at 13:27, Guido van Rossum <guido@python.org> wrote:
Ah, interesting - I'd only ever used it for the "synchronous code that just wants this *particular* operation to run asynchronously in the current thread" case, which it handles nicely :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 9, 2016 at 7:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
It's really best reserved just for the main() function of an app. Anywhere else, you run the risk that your use of the event loop gets hidden in layers of other code (example: some logging backend that writes to a database) and eventually someone will call your function from an async callback or coroutine. -- --Guido van Rossum (python.org/~guido)
On 09.08.2016 05:23, Nick Coghlan wrote:
If it's a such well-known **problem**, why does it even exist in Python in the first place? ;-) I don't buy that one couldn't have avoided it. Lately, I talked to friend of mine about async and his initial reaction was like "hmm that reminds me of early programming days, where you have to explicitly tell the scheduler to get control back". He's much older than me, so I think it was interesting for him to see that history is repeating again.
Interesting shift indeed and I like small Lego bricks I can use to build a big house. However, couldn't this be achieved without splitting the community? Sven
On Thu, Aug 11, 2016 at 3:17 AM, Sven R. Kunze <srkunze@mail.de> wrote:
Yes. One critical difference is that in the early days, one rogue program could bring the entire computer down; what we have now is preemptive switching between processes and cooperative between tasks within a process. Worst case, you can still only stall out your own program. ChrisA
On 11 August 2016 at 03:17, Sven R. Kunze <srkunze@mail.de> wrote:
Because sync vs async *isn't* primarily about parallelism - it's about different approaches to modelling a problem space. The key differences are akin to those between functional programming and object-oriented programming, rather than to those between using threads and processes for parallel code execution (when all you want is an even lighter weight alternative to threads without changing the programming model, then you want something like gevent). Glyph (of Twisted) wrote a good post about this back when asyncio was first being considered for the standard library: https://glyph.twistedmatrix.com/2014/02/unyielding.html So Python has support for both synchronous and asynchronous programming constructs for much the same reason we have both closures and classes: while technically isomorphic capabilities at a theoretical language design level, in practice, different problems lend themselves to different modelling techniques. I don't buy that one couldn't have avoided it.
You may want to check out some of the more event-driven programming oriented languages I mention in http://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-pyt... The easy way to avoid it is to just not offer one of the other, the same way that pure functional languages don't offer the ability to define objects with mutable state, and primarily object-oriented languages may not offer support for full lexical closures.
Yep, cooperative multi-threading predated pre-emptive multi-threading by *many* years (one of the big upgrades between Mac OS 9 and OS X was finally getting a pre-emptive scheduler). The difference is that where an OS is scheduling CPU access between mutually unaware applications, application developers can assume that other control flows *within* the application space are *not* actively hostile. In that latter context, enabling local reasoning about where context switches may happen can be valuable from a maintainability perspective, hence the pursuit of "best of both worlds" approaches now that the programming language design community has extensive experience with both.
You assume the current confusion around how the sync and async worlds interrelate will be permanent - it won't, any more than the confusion around the introduction of lexical closures and new-style classes was permanent back in the 2.2 era. It will be a process of convergence, first with the disparate async communities (Twisted, Tornado, gevent, etc) converging on the async/await syntactic abstraction on the asynchronous side of things, and then with standardisation of the sync/async gateways to make async implementations easy to consume from synchronous code (thread and process pools mean consumption of synchronous code from async is already relatively standardised) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Aug 11, 2016 at 1:17 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Mac OS lagged behind other operating systems in that. Back before 1990, OS/2 offered the PC world an alternative to the cooperative multitasking of Windows. The basic difference was: In Windows (that being the 3.x line at the time), an infinite loop in your program will bring the whole system down, but in OS/2, it'll only bring your program down. Windows NT then built on the OS/2 model, and Win2K, XP, and all subsequent versions of Windows have used that same preemption. Linux takes things slightly differently in the way it handles threads within a process, but still, processes are preemptively switched between. Though tongue-in-cheek, this talk shows (along the way) some of the costs of preemption, and thus some of the possibilities you'd open up if you could go back to cooperation. While I don't think the time is right for *operating systems* to go that route, it's definitely an option for tasks within one process, where (as Nick says) you can assume that it's not actively hostile. On the other hand, I've had times when I have *really* appreciated preemptive thread switching, as it's allowed me to fix problems in one of my threads by accessing another thread interactively, so maybe the best model is a collection of threads with a master. https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript ChrisA
As glyph notes in the blog post Nick linked, the reason threads are a problematic programming model is exactly because of their pre-emptive nature. This essentially gives you combinatoric expansion of the excecution-order-possibility-space: you must now confirm that your program is safe to running its instructions in almost any order, because that is what pre-emptive multithreading allows. Now, as you point out Chris, pre-emptive thread switching can help in weird situations, but those weird situations certainly should not be the average case! Most programmers should not be writing I/O code that is multithreaded for the same reason that most programmers shouldn’t be writing assembly code to speed up their Python programs: the benefit obtained from that approach is usually not outweighed by the increased engineering and maintenance costs. That’s not to say that there aren’t times where both of those ideas are good ideas: just that they’re uncommon, and shouldn’t be the first tool you pull out of your toolkit. While we’re talking about function colour, we should note that you don’t *have* to have function colour. A quick look at your average Twisted codebase that doesn’t use @inlineCallbacks will quickly show you that you can write an asynchronous program using nothing but synchronous function calls, so long as you are careful. And this has always been true: in my first job I worked on a cooperative-multitasking program written entirely in C, and as we all know C has absolutely no method for describing function colour. Again, so long as you are careful and don’t do any blocking I/O in your program, everything works just fine. But, here’s the rub. People hate callbacks, and people are bad at threads. Not everyone: I’m perfectly happy writing Twisted code, thank you, and I know loads of programmers who swear blind that threading is easy and they never have any problems, and they are almost certainly right, I’m not going to deny their experience of their discipline. But the reality is that writing good threaded code and writing good callback code are both harder than writing good code using coloured functions: they require care, and discipline, and foresight, and an understanding of code execution that is much more complex than what is rendered in any single function. Whereas writing code with coloured functions is *easier*: the space of possible execution flows is lower, the yield points are all clearly marked, and most of the care and discipline you need is reduced. The general long term solution to this is not to remove function colour, because it helps people. The long term solution is to develop a programming culture that says that 99% of your code should be normal functions, and only the last tiny 1% should make any statements about function colour. This is part of what the Sans-I/O project is about: demonstrating that you should avoid painting your functions as async until the last possible second, and that the fewer times you write async/await/@asyncio.coroutine <mailto:async/await/@asyncio.coroutine>/yield from the easier your life gets. Cory
On 11 Aug 2016 18:48, "Cory Benfield" <cory@lukasa.co.uk> wrote:
While we’re talking about function colour, we should note that you don’t
*have* to have function colour. A quick look at your average Twisted codebase that doesn’t use @inlineCallbacks will quickly show you that you can write an asynchronous program using nothing but synchronous function calls, so long as you are careful. And this has always been true: in my first job I worked on a cooperative-multitasking program written entirely in C, and as we all know C has absolutely no method for describing function colour. Again, so long as you are careful and don’t do any blocking I/O in your program, everything works just fine. Twisted callbacks are still red functions - you call them via the event loop rather than directly, and only event loop aware functions know how to make that request of the event loop. async/await just makes the function colour locally visible with a clear syntax, rather than needing to be inferred from behavioural details of the function implementation.
This part I wholeheartedly agree with, though - the vast majority of code will ideally be synchronous *non-blocking* code, suitable for use in data transformation pipelines and other data modelling and manipulation operations. Similar to Unicode handling, "event-driven or blocking?" is a decision best made at *system boundaries*, with the bulk of the application code not needing to worry about that decision either way. Cheers, Nick.
That's not really true, though its trueness depends on what you mean when you say “Twisted callbacks”. Like asyncio (which adopted this model from Twisted), Twisted has two separate things that might be called “callbacks”. The first can be called “callbacks-by-convention”: these are things like IProtocol, which define a model that Twisted will use to deliver data to your application. The second can be called “callbacks-by-implementation”, which are Deferreds. For “callbacks-by-convention”, I think the best way to describe it is that Twisted functions have a *convention* of being red, but they aren’t *actually* red. They can be called whenever you like. For example, consider the following Twisted protocol: from twisted.internet.protocol import Protocol class PrinterProtocol(Protocol): def connectionMade(self, transport): self.transport = transport self.transport.write( b'GET / HTTP/1.1\r\n' b'Host: http2bin.org\r\n' b'Connection: close\r\n' b'\r\n' ) def dataReceived(self, data): print data, def connectionLost(self, reason): print "Connection lost” There is nothing here that requires execution inside a reactor. In fact, if you change the spelling of “write” to “sendall”, you don’t need anything that the Python 2.7 standard library doesn’t give you: import socket def main(): p = PrinterProtocol() s = socket.create_connection(('http2bin.org', 80)) p.connectionMade(s) while True: data = s.recv(8192) if data: p.dataReceived(data) else: p.connectionLost(None) break s.close() What matters here is that these functions are *just functions*. Unlike Python 3’s async functions, they do not require a coroutine runner that understands their special yield values. You call them, they do a thing, they return synchronously. This is how callback code *has* to be constructed, and it’s why callbacks are the lowest-common-denominator kind of async code. Conversely, async functions as defined in Python 3 really do need a coroutine runner to execute them or nothing happens. But I presume you were really talking about the second kind of Twisted callback-function, which is the kind that uses Deferreds. And once again, there is nothing inherent in these functions that gives them a colour. Consider this bit of code: def showConnection(conn): print conn conn.close() def doStuff(): deferredConnection = makeConnection(‘http2bin.org’, 80) deferredConnection.addCallback(showConnection) return deferredConnection Your argument is that those functions are red. My argument is that they are uncoloured. For example, you can write makeConnection() like this: def makeConnection(host, port): s = socket.create_connection((host, port)) d = Deferred() d.callback(s) return d The simple fact of returning a Deferred doesn’t make your function async. This is one of Deferred’s fairly profound differences from asyncio.Future, which genuinely *does* require an event loop: Deferred can be evaluated entirely synchronously, with no recourse to an event loop. This has the effect of *uncolouring* functions that work with Deferreds. Their control flow is obscured, sure, but they are nonetheless not like async functions. In fact, it would be entirely possible to replace Twisted’s core event loop functions with ones that don’t delegate to a reactor, and then have a synchronous version of Twisted. It would be pretty gloriously useless, but is entirely do-able. Cory
Am 10.08.2016 um 19:17 schrieb Sven R. Kunze:
Up to now there is only one answer to the question "Is `await` in Python3 Cooperative Multitasking?" http://stackoverflow.com/questions/38865050/is-await-in-python3-cooperative-... The user there writes: [about await] That sounds quite like Cooperative multitasking to me. In 1996 I was a student at university and was told that preemptive multitasking is better. Since tools like http://python-rq.org/ can be implemented in Python2.7 I ask myself: why change the language? My opinion: if you want to do parallel processing, use a tool like python-rq or celery. Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
I agree. async/await doesn’t do parallel processing, it does concurrent processing. See also: https://blog.golang.org/concurrency-is-not-parallelism <https://blog.golang.org/concurrency-is-not-parallelism> Cory
Hi Yury! Thanks you for pushing this forward! As an author/contributor to several packages would love to use async generator in aiomysql/aiobotocore libraries. In particular want to share my use case for async generators. aiobotocore (https://github.com/aio-libs/aiobotocore) is wrapper around botocore library (https://github.com/boto/botocore) brings amazon services client to asyncio world. AWS api is huge, botocore itself 20k lines of code, so I bit hacked it here and there to add asyncio support. Code a bit ugly and hackish, but we have full amazon support only with few hundreds lines of code. Since protocol separated from IO, it is pretty strait forward to change requests http client with aiohttp one. But to port pagination logic, so it is possible to use async for, was very challenging. Here is original pagination https://github.com/boto/botocore/blob/bb09e88508f5593ce4393c72e7c1edbaf6d28a... which has structure like: async def f(): var = await io_fetch() while True: if var > 10: var += 1 yield var var = await io_fetch() else: var +=2 yield var If we rewrite this generator using __aiter__ code become very ugly and barely readable, since we need to track a lot of states, with async generator such problem does not exist. Moreover with this PEP, generators could be easily ported by just putting async/await in proper places, without changing logic. Thank you for this PEP!
On 2016-08-04 11:47 PM, Nickolai Novik wrote:
Thank you, Nickolai! I've written a couple of iterators that look like the one you have in botocore -- I totally feel your pain. It's really cool that you share this feedback. As I sad to Andrew in this thread -- feedback from power asyncio users is extremely useful. Thanks, Yury
I think over on async-sig my confusion over how this is for async generators and not coroutines came up. Now I think I know where my confusion stems from. I don't think this will change anything, but I wanted to get it out there as it might influence how we communicate things. Take Yury's simple example: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) In it was see `yield` and `await` in an `async def`. OK, so that signals that it's an asynchronous generator. But what happens if we take out the `yield`? async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): await asyncio.sleep(delay) According to the inspect module that's a coroutine function that creates a coroutine/awaitable (and a function w/ @types.coroutine is just an awaitable when it contains a `yield`). Now the existence of `yield` in a function causing it to be a generator is obviously not a new concept, but since coroutines come from a weird generator background thanks to `yield from` we might need to start being very clear what @types.coroutine, `async def`, and `async def` w/ `yield` are -- awaitable (but not coroutine in spite of the decorator name), coroutine, and async generator, respectively -- and not refer to @types.coroutine/`yield from` in the same breath to minimize confusion. -Brett On Tue, 2 Aug 2016 at 15:32 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hi Brett, On 2016-08-10 12:27 PM, Brett Cannon wrote:
Good suggestion. FWIW, in PEP 492, we called awaitables created with @types.coroutine as "generator-based coroutines". These days I see less and less people use @asyncio.coroutine and 'yield from'. Even less so know about @types.coroutine and how async/await tied to generators in the interpreter. This knowledge is now only required for people who implement/maintain frameworks like asyncio/twisted/curio. So I hope that we're already at the stage when people can just use async/await without really thinking how it's implemented. PEP 525 is another step in that direction -- making asynchronous iterators easy to use. Yury
Hi! I'm all for this. I've run into it so many times already while implementing the async/await support in Cython, it's really a totally obvious extension to what there is currently, and it's practically a requirement for any serious usage of async iterators. Some comments below. Yury Selivanov schrieb am 03.08.2016 um 00:31:
Well, or *something* that uses them in the same way as an event loop would. Doesn't have to be an event loop.
I don't see a motivation for adding an "aclose()" method in addition to the normal "close()" method. Similar for send/throw. Could you elaborate on that?
Phew, this adds quite some complexity and magic. That is a problem. For one, this uses a global setup, so There Can Only Be One of these finalizers. ISTM that if special cleanup is required, either the asyncgen itself should know how to do that, or we should provide some explicit API that does something when *initialising* the asyncgen. That seems better than doing something global behind the users' back. Have you considered providing some kind of factory in asyncio that wraps asyncgens or so? I won't go into the implementation details for now, but thanks for the PEP and for working on this so actively. Stefan
Hi Stefan! On 2016-08-03 2:45 AM, Stefan Behnel wrote:
Sure, I'm just using the same terminology PEP 492 was defined with. We can say "coroutine runner" instead of "event loop".
There will be no "close", "send" and "throw" defined for asynchronous generators. Only their asynchronous equivalents. This topic is actually quite complex, so bear with me. 1. It is important to understand that asynchronous iteration protocol is multiplexed into normal iteration protocol. For example: @types.coroutine def foobar(): yield 'spam' async def agen(): await foobar() yield 123 The 'agen' generator, on the lowest level of generators implementation will yield two things -- 'spam', and a wrapped 123 value. Because 123 is wrapped, the async generators machinery can distinguish async yields from normal yields. The idea behind __anext__ coroutine is that it yields through all "normal" yields, and raises a StopIteration when it encounters a "wrapped" yield (same idea behind aclose(), athrow(), and asend()) 2. Now let's look at two generators (sync and async): def gen(): async def agen(): try: try: ... ... finally: finally: fin1() await afin1() fin2() await afin2() yield 123 yield 123 * If we call 'gen().close()' when gen() is suspended somewhere in its try block, a GeneratorExit exception will be thrown in it. Then, fin1() and fin2() calls will be executed. Then, a 'yield 123' line will happen, which will cause a RuntimeError('generator yielded while closing'). The reason for the RuntimeError is that the interpreter does not want generators to yield in their finally statements. It wants them to synchronously finalize themselves. Yielding while closing doesn't make any sense. * Now, if we would just reuse the synchronous 'close()' method for 'agen()' -- awaiting on 'afin1()' would simply result in a RuntimeError('generator yielded while closing'). So the close() implementation for agen() must allow some yields, to make 'await' expressions possible in the finally block. This is something that is absolutely required, because instead of try..finally you could have 'async with' block in agen() -- so the ability to call asynchronous code in the 'finally' block is very important. Therefore, it's necessary to introduce a new close semantics for asynchronous generators: - It is OK to 'await' on anything in finally blocks in async generators. - Trying to 'yield' in finally blocks will result in a RuntimeError('async generator yielded while closing') -- similarly to sync generators. - Because we have to allow awaits in generator's finally blocks, the new 'close' method has to be a coroutine-like object. Since all this is quite different from sync generators' close method, it was decided to have a different name for this method for async generators: aclose. aclose() is a coroutine-like object, you can await on it, and you can even throw a CancelledError into it; so it's possible to write 'await asyncio.wait_for(agen.aclose(), timeout=1)'. 3. asend() and athrow(). This is very similar to aclose(). Both have to be coroutines because async yields are multiplexed into normal yields that awaitables use behind the scenes. async def foo(): await asyncio.sleep(1) yield 123 If we had a synchronous send() method defined for foo(), you'd see something like this: gen = foo() gen.send(None) -> <asyncio.Future> Instead, what you really want is this: gen = foo() await gen.asend(None) -> 123 4. I really recommend you to play with the reference implementation. You can emulate synchronous "send" and "throw" buy doing this trick: gen = foo() gen.__anext__().send() gen.__anext__().throw()
set_asyncgen_finalizer is thread-specific, so you can have one finalizer set up per thread. The reference implementation actually integrates this all into asyncio. The idea is to setup loop async gens finalizer just before the loop starts, and reset the finalizer to the previous one (usually it's None) just before it stops. The finalizer is attached to a generator when it is yielding for the first time -- this guarantees that every generators will have a correct finalizer attached to it. It's not right to attach the finalizer (or wrap the generator) when the generator is initialized. Consider this code: async def foo(): async with smth(): yield async def coro(gen): async for i in foo(): ... loop.run_until_complete(coro(foo())) ^^ In the above example, when the 'foo()' is instantiated, there is no loop or finalizers set up at all. BUT since a loop (or coroutine wrapper) is required to iterate async generators, there is a strong guarantee that it *will* present on the first iteration. Regarding "async gen itself should know how to cleanup" -- that's not possible. async gen could just have an async with block and then GCed (after being partially consumed). Users won't expect to do anything besides using try..finally or async with, so it's the responsibility of the coroutine runner to cleanup async gen. Hence 'aclose' is a coroutine, and hence this set_asyncgen_finalizer API for coroutine runners. This is indeed the most magical part of the proposal. Although it's important to understand that the regular Python users will likely never encounter this in their life -- finalizers will be set up by the framework they use (asyncio, Tornado, Twisted, you name it). Thanks! Yury
Yury Selivanov schrieb am 03.08.2016 um 17:32:
Ok, why not. Different names for similar things that behave differently enough.
This is actually going to be tricky to backport for Cython (which supports Py2.6+) since it seems to depend on a globally known C implemented wrapper object type. We'd have to find a way to share that across different packages and also across different Cython versions (types are only shared within the same Cython version). I guess we'd have to store a reference to that type in some well hidden global place somewhere, and then never touch its implementation again... Is that wrapper type going to be exposed anywhere in the Python visible world, or is it purely internal? (Not that I see a use case for making it visible to Python code...) BTW, why wouldn't "async yield from" work if the only distinction point is whether a yielded object is wrapped or not? That should work at any level of delegation, shouldn't it?
Correct. And it also wouldn't help to generally extend the Async-Iterator protocol with an aclose() method because ending an (async-)for loop doesn't mean we are done with the async iterator, so this would just burden the users with unnecessary cleanup handling. That's unfortunate...
I think my main problem is that you keep speaking about event loops (of which There Can Be Only One, by design), whereas coroutines are a much more general concept and I cannot overlook all possible forms of using them in the future. What I would like to avoid is the case where we globally require setting up one finalizer handler (or context), and then prevent users from doing their own cleanup handling in some module context somewhere. It feels to me like there should be some kind of stacking for this (which in turn feels like a context manager) in order to support adapters and the like that need to do their own cleanup handling (or somehow intercept the global handling), regardless of what else is running. But I am having a hard time trying to come up with an example where the thread context really doesn't work. Even if an async generator is shared across multiple threads, then those threads would most likely be controlled by the coroutine runner, which would simply set up a proper finalizer context for each of the threads that points back to itself, so that it would not be the pure chance of first iteration to determine who gets to clean up afterwards. It's possible that such (contrieved?) cases can be handled in one way or another by changing the finalization itself, instead of changing the finalizer context. And that would be done by wrapping async iterators in a delegator that uses try-finally, be it via a decorator or an explicit wrapper. It's just difficult to come to a reasonable conclusion at this level of uncertainty. That's why I'm bouncing this consideration here, in case others have an idea how this problem could be avoided all together. Stefan
On 2016-08-06 4:29 AM, Stefan Behnel wrote:
I don't think you need to care about these details. You can implement async generators using async iteration protocol as it was defined in PEP 492. You can compile a Cython async generator into an object that implements `am_aiter` slot (which should return `self`) and `am_anext` slot (which should return a coroutine-like object, similar to what Cython compiles 'async def'). `am_anext` coroutine should "return" (i.e. raise `StopIteration`) when it reaches an "async yield" point, or raise "StopAsyncIteration" when the generator is exhausted. Essentially, because pure Python async generators work fine with 'async for' and will continue to do so in 3.6, I think there shouldn't be technical problems to add asynchronous generators in Cython.
Yes, it's a purely internal C-level thing. Python code will never see it.
I can try ;) Although when I was working on the PEP I had a feeling that this won't work without a serious refactoring of genobject.c.
Well, we have the *exact same thing* with regular (synchronous) iterators. Let's say you have an iterable object (with `__iter__` and `__next__`), which cleanups resources at the end of its iteration. If you didn't implement its __del__ properly (or at all), the resources won't be cleaned-up when it's partially iterated and then GCed.
We can call the thing that runs coroutines a "coroutine runner". I'll update the PEP. Regardless of the naming issues, I don't see any potential problem with how `set_asyncgen_finalizer` is currently defined: 1. Because a finalizer (set with `set_asyncgen_finalizer`) is assigned to generators on their first iteration -- it's guaranteed that each async generator will have a correct one attached to it. 2. It's extremely unlikely that somebody will design a system that switches coroutine runners *while async/awaiting a coroutine*. There are no good reasons to do this, and I doubt that it's even a possible thing. But even in this unlikely use case, you can easily stack finalizers following this pattern: old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(my_finalizer) try: # do my thing finally: sys.set_asyncgen_finalizer(old_finalizer) Thanks, Yiry
Yury Selivanov schrieb am 06.08.2016 um 18:39:
Hmm, I thought that at least interoperability might get in the way. I guess I'll just have to give it a try...
2. It's extremely unlikely that somebody will design a system that switches coroutine runners *while async/awaiting a coroutine*.
Yes, I guess so.
That only works for synchronous code, though, because if this is done in a coroutine, it might get suspended within the try block and would leak its own finalizer into the outer world. Stefan
On 2016-08-06 1:03 PM, Stefan Behnel wrote:
Maybe, if we add 'yield from' (which I really don't want to bother with in 3.6 at least). It'd be extremely helpful if you could prototype the PEP in Cython!
set_asyncgen_finalizer is designed to be used *only* by coroutine runners. This is a low-level API that coroutines should never touch. (At least my experience working with coroutines says so...) Thank you, Yury
Hey Yury, that's a great proposal! On 03.08.2016 00:31, Yury Selivanov wrote:
That's a great motivational example. +1 Especially when reading the venerable PEP 255 (Q/A part), this also gets rid of the "low-level detail": "raise StopAsyncIteration". This could also be worth mentioning in the motivation part for PEP 525.
From your answer to Stefan, I get the impression that the reason why we actual need all those a* methods (basically a duplication of the existing gen protocol), is the fact that normal generators can be converted to coroutines. That means, 'yield' still can be used in both ways. So, it's a technical symptom of the backwards-compatibility rather than something that cannot be avoided by design. Is this correct? If it's correct, would you think it would make sense to get rid of the a* in a later iteration of the async capabilities of Python? So, just using the normal generator protocol again? One note on all examples but the last. Reading those examples, it creates the illusion of actual working code which is not the case, right? One would always need to 1) wrap module-level statements into its own coroutine, 2) create an event-loop and 3) run it. Do you think clarifying this in the PEP makes sense? Thanks again for your hard work here. Async generators definitely completes the picture. Sven
Hi Sven, On 2016-08-04 9:20 AM, Sven R. Kunze wrote:
Thanks! [..]
async/await in Python is implemented on top of the generator protocol. Any 'await' is either awaiting on a coroutine or on a Future-like object. Future-like objects are defined by implementing the __await__ method, which should return a generator. So coroutines and generators are very intimately tied to each other, and that's *by design*. Any coroutine that iterates over an asynchronous generator uses the generator protocol behind the scenes. So we have to multiplex the async generaotor's "yields" into the generator protocol in such a way, that it stays isolated, and does not interfere with the "yields" that drive async/await.
Because async generators will contain 'await' expressions, we have to have a* methods (although we can name them without the "a" prefix, but I believe that would be confusing for many users).
I'll think about this, thanks! Maybe I can add a line in the beginning of the "Specification" section.
Thanks again for your hard work here. Async generators definitely completes the picture.
Thank you, Sven Yury
As maintainer of aiohttp (asyncio-compatible HTTP client/server) and aiopg (PostgreSQL asyncio driver) I totally support the pep. The problem is: I'm able to write async iterator class which utilizes __aiter__/__anext__. But users of my libraries are not. Writing synchronous generator functions for data processing is very common pattern. But class based approach requires saving all context not in local variables but in instance vars, you know. It's a nightmare in practice, as consequence people writes more complex and error prone code than they could write with this PEP. https://www.python.org/dev/peps/pep-0255/ for simple generators is 15 years old, I believe everybody understand why we need it. Async generators extend the same concept to asyncio world by solving the same problems but with async code. Honestly I personally don't feel a need for two-way generators with `asend()`/`athrow()` but as Yury explained he need them internally for `anext`/`aclose` anyway. On Thu, Aug 4, 2016 at 6:18 PM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
-- Thanks, Andrew Svetlov
Hi Yury! Thanks for this PEP! This is really what asyncio misses since you made async iterators. As contributor to aiohttp and maintainer of aiocouchdb and few else projects I also support this PEP. Great one! I agree with what Nikolay and Andrew already said: dancing around__aiter__/__anext__ to emulate genarator-like behaviour is quite boring and complicated. This also makes porting "old synced" code onto asyncio harder since you have to rewrite every generator in iterator fashion and wrap each with an object that provides __aiter__ / __anext__ interface. For the reference I have is ijson PR[1] that adds asyncio support. If we forget about compatibility with 3.3/3.4, with async generators the result implementation would be much more elegant and looks more closer to the original synced code, because it's all based on generators. Lack of async generators forces people to implement them on their own [2] ones[3], not very effective as the one that could be built-in. Also custom interface for each implementation makes hardly to reuse them and don't even imagine what happens if two custom async generator implementations will met in a single project. So thank you again, Yury, for pushing this really important and missing feature forward! [1]: https://github.com/isagalaev/ijson/pull/46 [2]: https://github.com/germn/aiogen [3]: https://github.com/ethanfrey/aiojson/blob/master/aiojson/utils/aiogen.py -- ,,,^..^,,,
On 2016-08-05 4:34 PM, Alexander Shorin wrote:
Thank you Alexander for the feedback! It's really great that asyncio users begin to share their experience in this thread. Indeed, the key idea is to complete async/await implementation, so that you can think in async/await terms and use simply use existing Python idioms. Want a context manager in your asyncio application? Use 'async with'. Want an iterator? Use 'async for' and async generators. Hopefully we'll get there. Thank you, Yury
On 6 August 2016 at 01:11, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Even if asend()/athrow() are cheap from an implementation perspective, they're *not* necessarily cheap from a "cognitive burden of learning the API" perspective. We know what send() and throw() on generators are for: implementing coroutines. However, we also know that layering that on top of the generator protocol turned out to be inherently confusing, hence PEP 492 and async/await. So if we don't have a concrete use case for "coroutine coroutines" (what does that even *mean*?), we shouldn't add asend()/athrow() just because it's easy to do so. If we wanted to hedge our bets, we could expose them as _asend() and _athrow() initially, on the grounds that doing so is cheap, and we're genuinely curious to see if anyone can find a non-toy use case for them. But we should be very reluctant to add them to the public API without a clear idea of why they're there. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, On 2016-08-06 4:28 AM, Nick Coghlan wrote:
One immediate use case that comes to mind is @contextlib.asynccontextmanager Essentially this: @contextlib.asynccontextmanager async def ctx(): resource = await acquire() try: yield resource finally: await release(resource) I'm also worrying that in order to do similar things people will abuse "agen.__anext__().send()" and "agen.__anext__().throw()". And this is a very error-prone thing to do. For example, in asyncio code, you will capture yielded Futures that the generator is trying to await on (those yielded futures have to be captured by the running Task object instead). "asend" and "athrow" know about iteration protocol and can see the "wrapped" yielded objects, hence they can iterate the generator correctly and reliably. I think Nathaniel Smith (CC-ed) had a few more use cases for asend/athrow.
I think we should decide if we want to expose them at all. If we're in doubt, let's not do that at all. Exposing them with a "_" prefix doesn't free us from maintaining backwards compatibility etc. You're completely right about the increased "cognitive burden of learning the API". On the other hand, send() and throw() for sync generators were designed for power users; most Python developers don't even know that they exist. I think we can view asend() and athrow() similarly. Thank you, Yury
While Nick has a point about coroutine coroutines confusion. I consider asend/athow will be useful in real world. Because they provide stable, well known and uniform interface to communicate with a coroutine. For now to have two-way communication with some coroutine you have to establish two queues: inbox and outbox, pass them around, don't eventually mess them and so on. Queues are great for message-passing communication and generally good idea to use, but it still some boilerplate code as like as __aiter__/__anext__ (which async generators will get rid) to maintain. Also queue is a bad reference to a coroutine owns it - it's hard to find own who is really reading that inbox in complex codebase. With asend/athrow there will be no need to plass queues around, but just coroutine instance and do all the communication with it as with plain object. You need to retrieve a value - do iterate over it; need to send a value back - do asend. Termination and error-passing are aslo easy to do. So while this interface looks complicated for learning curve, it could be very helpful, when you need simple communication channel between coroutines sized equals one. For something greater you have to return to queues approach, but they involve quite a lot of else bits to care about like queue overflow, which is indeed topic for advanced users. -- ,,,^..^,,, On Sat, Aug 6, 2016 at 7:51 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On 7 August 2016 at 02:51, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Good point. Let's make adding that part of the PEP, since it will be useful, makes it immediately obvious why we're including asend() and athrow(), and will also stress test the coroutine traceback machinery in an interesting way. (Note that depending on the implementation details, it may need to end up living in asyncio somewhere - that will depend on whether or not we need to use any asyncio APIs in the implementation of _CoroutineContextManager) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2016-08-07 11:53 AM, Nick Coghlan wrote:
It's already in the PEP: https://www.python.org/dev/peps/pep-0525/#why-the-asend-and-athrow-methods-a... Although it's not super visible, I agree. Should I try to make that section more prominent? Thank you, Yury
Hi Yury, Thank you for posting this PEP! As an asyncio-based libraries user, author and contributor I appreciate that topic of easy-writing of asynchronous generators is being covered now. I like how simple writing of async generators will be with this PEP, same with consequent functionality (e.g. async context wrappers). It's great that with this PEP it would be possible to write simple async data processing wrappers without a knowledge of a single special method (__aiter__, __anext__, __aenter__, __aexit__) using same patterns as regular Python user would use in synchronous code: @async_context_manager async def connection(uri): conn = await connect(uri) try: yield conn finally: conn.close() await conn.wait_closed() async def query_data(conn, q): cursor = await conn.query(q) try: row = await cursor.fetch_row() while row is not None: yield row row = await cursor.fetch_row() finally: cursor.close() await cursor.wait_closed() async with connection(uri) as conn: async for item in query_data(q): print(item) And this is as simple as in synchronous code, but made asynchronous. Can you explain details of canceling of asend and await coroutines? Consider following example: async def gen(): zero = yield 0 asyncio.sleep(10) one = yield 1 async def f(loop): g = gen() await g.asend(None) send_task = loop.create_task(g.asend('zero')) send_task.cancel() # What is the state of g now? # Can I receive one from g? Or cancel() will lead to CancelledError throw from asyncio.sleep(10)? one = g.asend('one') Will canceling of asend() task result in canceling of generator? Thanks, Vladimir Rutsky
Hi Vladimir, On 2016-08-09 1:24 PM, Vladimir Rutsky wrote:
Thanks a lot for reaching out!
Precisely this.
Cancelling `asend()` will result in an asyncio.CancelledError being raised from the 'await asyncio.sleep(0)' line inside the async generator: async def gen(): try: await asyncio.sleep(1) except asyncio.CancelledError: print('got it') raise # It's OK to not to re-raise the error too! yield 123 async def run(): g = gen() t = asyncio.ensure_future(g.asend(None)) await asyncio.sleep(0.5) t.cancel() await t # this line will fail with CancelledError ^ the above code will print "got it" and then crash with an asyncio.CancelledError raised back from 'await t'. You can handle the error in the generator and continue to iterate over it. If you don't catch that exception inside your generator, the generator will be closed (it's similar to what would happen if an exception occurs inside a sync generator, or is thrown into it with gen.throw()). Thank you, Yury
On 04.08.2016 17:17, Yury Selivanov wrote:
Yes, that is how I understand it as well. So, all this complexity stems from the intertwining of generators and coroutines. I am wondering if loosening the tie between the two could make it all simpler; even for you. (Not right now, but maybe later.)
I think that is related to Stefan's question here. I cannot speak for him but it seems to me the confusion is actually the other way around. Coming from a database background, I prefer redundant-free data representation. The "a" prefix encodes the information "that's an asynchronous generator" which is repeated several times and thus is redundant. Another example: when writing code for generators and checking for the "send" method, I would then need to check for the "asend" method as well. So, I would need to touch that code although it even might not have been necessary in the first place. I don't want to talk you out of the "a" prefix, but I get the feeling that the information "that's asynchronous" should be provided as a separate attribute. I believe it would reduce confusion, simplify duck-typing and flatten the learning curve. :)
You're welcome. :) Sven
On 2016-08-05 4:51 PM, Sven R. Kunze wrote:
Unfortunately, the only way to "loosen the tie" is to re-implement generator protocol for coroutines (make them a separate thing). Besides breaking backwards compatibility with the existing Python code that uses @coroutine decorator and 'yield from' syntax, it will also introduce a ton of work.
Thing is you can't write one piece of code that will accept any type of generator (sync or async). * send, throw, close, __next__ methods for sync generators, they are synchronous. * send, throw, close, __next__ methods for async generators, they *require* to use 'await' on them. There is no way to make them "synchronous", because you have awaits in async generators. Because of the above, duck-typing simply isn't possible. The prefix is there to make people aware that this is a completely different API, even though it looks similar. Thank you, Yury
On 06.08.2016 01:25, Yury Selivanov wrote:
You are the best to know that. :)
There are a lot of things you can do with generators (decorating, adding attributes etc.) and none of them require you to "make them synchronous".
Because of the above, duck-typing simply isn't possible.
See above.
The prefix is there to make people aware that this is a completely different API, even though it looks similar.
Sure. So, what do you think about the separate attribute to make people aware? Best, Sven
Sure. So, what do you think about the separate attribute to make people aware?
What people should be aware about? IMHO sync generators and async ones have different types. What also is needed? On Mon, Aug 8, 2016 at 2:45 PM Sven R. Kunze <srkunze@mail.de> wrote:
-- Thanks, Andrew Svetlov
On 2016-08-08 7:44 AM, Sven R. Kunze wrote:
You have to be aware of what you're decorating. Always. For instance, here's an example of a buggy code: @functools.lru_cache() def foo(): yield 123 I'm really against duck typing here. I see no point in making the API for async generators to look similar to the API of sync generators. You have to provide a solid real-world example of where it might help with asynchronous generators to convince me otherwise ;)
There is a separate attribute already -- __class__. Plus a couple of new functions in inspect module: inspect.isasyncgenfunction and inspect.isasyncgen. And the new types.AsyncGeneratorType for isinstance checks. Thanks, Yury
On 08.08.2016 19:06, Yury Selivanov wrote:
You have to be aware of what you're decorating. Always.
You have to be aware of what the decorator does first before you decide if it's relevant to be aware of what you're decorating.
And so do I to making them "almost similar" by putting an "a" in front of the well-known generator protocol. But maybe, because there is not real convincing argument on either side, it doesn't matter at all.
You have to provide a solid real-world example of where it might help with asynchronous generators to convince me otherwise ;)
Something, I learned in the past years is to be prepared. So, such solid real-world examples might emerge when async generators are out in the wild. Let's hope this does not make it harder for them to start with. :)
Got it, thanks. Just another random thought I want to share: From what I've heard in the wild, that most if not all pieces of async are mirroring existing Python features. So, building async basically builds a parallel structure in Python resembling Python. Async generators complete the picture. Some people (including me) are concerned by this because they feel that having two "almost the same pieces" is not necessarily a good thing to have. And not necessarily bad but it feels like duplicating code all over the place especially as existing functions are incompatible with async. As I understand it, one goal is to be as close to sync code as possible to make async code easier to understand. So, if dropping 'await's all over the place is the main difference between current async and sync code, I get the feeling both styles could be very much unified. (The examples of the PEP try to do it like this but it wouldn't work as we already discussed.) Maybe, it's too early for a discussion about this but I wanted to dump this thought somewhere. :) Thanks, Sven
On 9 August 2016 at 08:37, Sven R. Kunze <srkunze@mail.de> wrote:
It's a known problem that applies to programming language design in general rather than being Python specific: http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ The async/await model used in C# and Python 3.5+ aims to at least make it clear whether you're dealing with "red" (asynchronous) or "blue" (synchronous) code, but some use cases will still benefit from using something like gevent to bridge between the synchronous world and the asynchronous one: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programm... In cases where blocking for IO is acceptable, the "run_until_complete()" method on event loop implementations can be used in order to invoke asynchronous code synchronously: http://www.curiousefficiency.org/posts/2015/07/asyncio-background-calls.html
One of the big mindset shifts it encourages is to design as many support libraries as possible as computational pipelines and message-drive state machines, rather than coupling them directly to IO operations (which is the way many of them work today). Brett Cannon started the Sans IO information project to discuss this concept at http://sans-io.readthedocs.io/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Just don't oversell run_until_complete() -- some people coming from slightly different event loop paradigms expect to be able to pump for events at any point, possibly causing recursive invocations. That doesn't work here (and it's a feature it doesn't). On Mon, Aug 8, 2016 at 8:23 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 9 August 2016 at 13:27, Guido van Rossum <guido@python.org> wrote:
Ah, interesting - I'd only ever used it for the "synchronous code that just wants this *particular* operation to run asynchronously in the current thread" case, which it handles nicely :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 9, 2016 at 7:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
It's really best reserved just for the main() function of an app. Anywhere else, you run the risk that your use of the event loop gets hidden in layers of other code (example: some logging backend that writes to a database) and eventually someone will call your function from an async callback or coroutine. -- --Guido van Rossum (python.org/~guido)
On 09.08.2016 05:23, Nick Coghlan wrote:
If it's a such well-known **problem**, why does it even exist in Python in the first place? ;-) I don't buy that one couldn't have avoided it. Lately, I talked to friend of mine about async and his initial reaction was like "hmm that reminds me of early programming days, where you have to explicitly tell the scheduler to get control back". He's much older than me, so I think it was interesting for him to see that history is repeating again.
Interesting shift indeed and I like small Lego bricks I can use to build a big house. However, couldn't this be achieved without splitting the community? Sven
On Thu, Aug 11, 2016 at 3:17 AM, Sven R. Kunze <srkunze@mail.de> wrote:
Yes. One critical difference is that in the early days, one rogue program could bring the entire computer down; what we have now is preemptive switching between processes and cooperative between tasks within a process. Worst case, you can still only stall out your own program. ChrisA
On 11 August 2016 at 03:17, Sven R. Kunze <srkunze@mail.de> wrote:
Because sync vs async *isn't* primarily about parallelism - it's about different approaches to modelling a problem space. The key differences are akin to those between functional programming and object-oriented programming, rather than to those between using threads and processes for parallel code execution (when all you want is an even lighter weight alternative to threads without changing the programming model, then you want something like gevent). Glyph (of Twisted) wrote a good post about this back when asyncio was first being considered for the standard library: https://glyph.twistedmatrix.com/2014/02/unyielding.html So Python has support for both synchronous and asynchronous programming constructs for much the same reason we have both closures and classes: while technically isomorphic capabilities at a theoretical language design level, in practice, different problems lend themselves to different modelling techniques. I don't buy that one couldn't have avoided it.
You may want to check out some of the more event-driven programming oriented languages I mention in http://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-pyt... The easy way to avoid it is to just not offer one of the other, the same way that pure functional languages don't offer the ability to define objects with mutable state, and primarily object-oriented languages may not offer support for full lexical closures.
Yep, cooperative multi-threading predated pre-emptive multi-threading by *many* years (one of the big upgrades between Mac OS 9 and OS X was finally getting a pre-emptive scheduler). The difference is that where an OS is scheduling CPU access between mutually unaware applications, application developers can assume that other control flows *within* the application space are *not* actively hostile. In that latter context, enabling local reasoning about where context switches may happen can be valuable from a maintainability perspective, hence the pursuit of "best of both worlds" approaches now that the programming language design community has extensive experience with both.
You assume the current confusion around how the sync and async worlds interrelate will be permanent - it won't, any more than the confusion around the introduction of lexical closures and new-style classes was permanent back in the 2.2 era. It will be a process of convergence, first with the disparate async communities (Twisted, Tornado, gevent, etc) converging on the async/await syntactic abstraction on the asynchronous side of things, and then with standardisation of the sync/async gateways to make async implementations easy to consume from synchronous code (thread and process pools mean consumption of synchronous code from async is already relatively standardised) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Aug 11, 2016 at 1:17 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Mac OS lagged behind other operating systems in that. Back before 1990, OS/2 offered the PC world an alternative to the cooperative multitasking of Windows. The basic difference was: In Windows (that being the 3.x line at the time), an infinite loop in your program will bring the whole system down, but in OS/2, it'll only bring your program down. Windows NT then built on the OS/2 model, and Win2K, XP, and all subsequent versions of Windows have used that same preemption. Linux takes things slightly differently in the way it handles threads within a process, but still, processes are preemptively switched between. Though tongue-in-cheek, this talk shows (along the way) some of the costs of preemption, and thus some of the possibilities you'd open up if you could go back to cooperation. While I don't think the time is right for *operating systems* to go that route, it's definitely an option for tasks within one process, where (as Nick says) you can assume that it's not actively hostile. On the other hand, I've had times when I have *really* appreciated preemptive thread switching, as it's allowed me to fix problems in one of my threads by accessing another thread interactively, so maybe the best model is a collection of threads with a master. https://www.destroyallsoftware.com/talks/the-birth-and-death-of-javascript ChrisA
As glyph notes in the blog post Nick linked, the reason threads are a problematic programming model is exactly because of their pre-emptive nature. This essentially gives you combinatoric expansion of the excecution-order-possibility-space: you must now confirm that your program is safe to running its instructions in almost any order, because that is what pre-emptive multithreading allows. Now, as you point out Chris, pre-emptive thread switching can help in weird situations, but those weird situations certainly should not be the average case! Most programmers should not be writing I/O code that is multithreaded for the same reason that most programmers shouldn’t be writing assembly code to speed up their Python programs: the benefit obtained from that approach is usually not outweighed by the increased engineering and maintenance costs. That’s not to say that there aren’t times where both of those ideas are good ideas: just that they’re uncommon, and shouldn’t be the first tool you pull out of your toolkit. While we’re talking about function colour, we should note that you don’t *have* to have function colour. A quick look at your average Twisted codebase that doesn’t use @inlineCallbacks will quickly show you that you can write an asynchronous program using nothing but synchronous function calls, so long as you are careful. And this has always been true: in my first job I worked on a cooperative-multitasking program written entirely in C, and as we all know C has absolutely no method for describing function colour. Again, so long as you are careful and don’t do any blocking I/O in your program, everything works just fine. But, here’s the rub. People hate callbacks, and people are bad at threads. Not everyone: I’m perfectly happy writing Twisted code, thank you, and I know loads of programmers who swear blind that threading is easy and they never have any problems, and they are almost certainly right, I’m not going to deny their experience of their discipline. But the reality is that writing good threaded code and writing good callback code are both harder than writing good code using coloured functions: they require care, and discipline, and foresight, and an understanding of code execution that is much more complex than what is rendered in any single function. Whereas writing code with coloured functions is *easier*: the space of possible execution flows is lower, the yield points are all clearly marked, and most of the care and discipline you need is reduced. The general long term solution to this is not to remove function colour, because it helps people. The long term solution is to develop a programming culture that says that 99% of your code should be normal functions, and only the last tiny 1% should make any statements about function colour. This is part of what the Sans-I/O project is about: demonstrating that you should avoid painting your functions as async until the last possible second, and that the fewer times you write async/await/@asyncio.coroutine <mailto:async/await/@asyncio.coroutine>/yield from the easier your life gets. Cory
On 11 Aug 2016 18:48, "Cory Benfield" <cory@lukasa.co.uk> wrote:
While we’re talking about function colour, we should note that you don’t
*have* to have function colour. A quick look at your average Twisted codebase that doesn’t use @inlineCallbacks will quickly show you that you can write an asynchronous program using nothing but synchronous function calls, so long as you are careful. And this has always been true: in my first job I worked on a cooperative-multitasking program written entirely in C, and as we all know C has absolutely no method for describing function colour. Again, so long as you are careful and don’t do any blocking I/O in your program, everything works just fine. Twisted callbacks are still red functions - you call them via the event loop rather than directly, and only event loop aware functions know how to make that request of the event loop. async/await just makes the function colour locally visible with a clear syntax, rather than needing to be inferred from behavioural details of the function implementation.
This part I wholeheartedly agree with, though - the vast majority of code will ideally be synchronous *non-blocking* code, suitable for use in data transformation pipelines and other data modelling and manipulation operations. Similar to Unicode handling, "event-driven or blocking?" is a decision best made at *system boundaries*, with the bulk of the application code not needing to worry about that decision either way. Cheers, Nick.
That's not really true, though its trueness depends on what you mean when you say “Twisted callbacks”. Like asyncio (which adopted this model from Twisted), Twisted has two separate things that might be called “callbacks”. The first can be called “callbacks-by-convention”: these are things like IProtocol, which define a model that Twisted will use to deliver data to your application. The second can be called “callbacks-by-implementation”, which are Deferreds. For “callbacks-by-convention”, I think the best way to describe it is that Twisted functions have a *convention* of being red, but they aren’t *actually* red. They can be called whenever you like. For example, consider the following Twisted protocol: from twisted.internet.protocol import Protocol class PrinterProtocol(Protocol): def connectionMade(self, transport): self.transport = transport self.transport.write( b'GET / HTTP/1.1\r\n' b'Host: http2bin.org\r\n' b'Connection: close\r\n' b'\r\n' ) def dataReceived(self, data): print data, def connectionLost(self, reason): print "Connection lost” There is nothing here that requires execution inside a reactor. In fact, if you change the spelling of “write” to “sendall”, you don’t need anything that the Python 2.7 standard library doesn’t give you: import socket def main(): p = PrinterProtocol() s = socket.create_connection(('http2bin.org', 80)) p.connectionMade(s) while True: data = s.recv(8192) if data: p.dataReceived(data) else: p.connectionLost(None) break s.close() What matters here is that these functions are *just functions*. Unlike Python 3’s async functions, they do not require a coroutine runner that understands their special yield values. You call them, they do a thing, they return synchronously. This is how callback code *has* to be constructed, and it’s why callbacks are the lowest-common-denominator kind of async code. Conversely, async functions as defined in Python 3 really do need a coroutine runner to execute them or nothing happens. But I presume you were really talking about the second kind of Twisted callback-function, which is the kind that uses Deferreds. And once again, there is nothing inherent in these functions that gives them a colour. Consider this bit of code: def showConnection(conn): print conn conn.close() def doStuff(): deferredConnection = makeConnection(‘http2bin.org’, 80) deferredConnection.addCallback(showConnection) return deferredConnection Your argument is that those functions are red. My argument is that they are uncoloured. For example, you can write makeConnection() like this: def makeConnection(host, port): s = socket.create_connection((host, port)) d = Deferred() d.callback(s) return d The simple fact of returning a Deferred doesn’t make your function async. This is one of Deferred’s fairly profound differences from asyncio.Future, which genuinely *does* require an event loop: Deferred can be evaluated entirely synchronously, with no recourse to an event loop. This has the effect of *uncolouring* functions that work with Deferreds. Their control flow is obscured, sure, but they are nonetheless not like async functions. In fact, it would be entirely possible to replace Twisted’s core event loop functions with ones that don’t delegate to a reactor, and then have a synchronous version of Twisted. It would be pretty gloriously useless, but is entirely do-able. Cory
Am 10.08.2016 um 19:17 schrieb Sven R. Kunze:
Up to now there is only one answer to the question "Is `await` in Python3 Cooperative Multitasking?" http://stackoverflow.com/questions/38865050/is-await-in-python3-cooperative-... The user there writes: [about await] That sounds quite like Cooperative multitasking to me. In 1996 I was a student at university and was told that preemptive multitasking is better. Since tools like http://python-rq.org/ can be implemented in Python2.7 I ask myself: why change the language? My opinion: if you want to do parallel processing, use a tool like python-rq or celery. Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/
I agree. async/await doesn’t do parallel processing, it does concurrent processing. See also: https://blog.golang.org/concurrency-is-not-parallelism <https://blog.golang.org/concurrency-is-not-parallelism> Cory
Hi Yury! Thanks you for pushing this forward! As an author/contributor to several packages would love to use async generator in aiomysql/aiobotocore libraries. In particular want to share my use case for async generators. aiobotocore (https://github.com/aio-libs/aiobotocore) is wrapper around botocore library (https://github.com/boto/botocore) brings amazon services client to asyncio world. AWS api is huge, botocore itself 20k lines of code, so I bit hacked it here and there to add asyncio support. Code a bit ugly and hackish, but we have full amazon support only with few hundreds lines of code. Since protocol separated from IO, it is pretty strait forward to change requests http client with aiohttp one. But to port pagination logic, so it is possible to use async for, was very challenging. Here is original pagination https://github.com/boto/botocore/blob/bb09e88508f5593ce4393c72e7c1edbaf6d28a... which has structure like: async def f(): var = await io_fetch() while True: if var > 10: var += 1 yield var var = await io_fetch() else: var +=2 yield var If we rewrite this generator using __aiter__ code become very ugly and barely readable, since we need to track a lot of states, with async generator such problem does not exist. Moreover with this PEP, generators could be easily ported by just putting async/await in proper places, without changing logic. Thank you for this PEP!
On 2016-08-04 11:47 PM, Nickolai Novik wrote:
Thank you, Nickolai! I've written a couple of iterators that look like the one you have in botocore -- I totally feel your pain. It's really cool that you share this feedback. As I sad to Andrew in this thread -- feedback from power asyncio users is extremely useful. Thanks, Yury
I think over on async-sig my confusion over how this is for async generators and not coroutines came up. Now I think I know where my confusion stems from. I don't think this will change anything, but I wanted to get it out there as it might influence how we communicate things. Take Yury's simple example: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) In it was see `yield` and `await` in an `async def`. OK, so that signals that it's an asynchronous generator. But what happens if we take out the `yield`? async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): await asyncio.sleep(delay) According to the inspect module that's a coroutine function that creates a coroutine/awaitable (and a function w/ @types.coroutine is just an awaitable when it contains a `yield`). Now the existence of `yield` in a function causing it to be a generator is obviously not a new concept, but since coroutines come from a weird generator background thanks to `yield from` we might need to start being very clear what @types.coroutine, `async def`, and `async def` w/ `yield` are -- awaitable (but not coroutine in spite of the decorator name), coroutine, and async generator, respectively -- and not refer to @types.coroutine/`yield from` in the same breath to minimize confusion. -Brett On Tue, 2 Aug 2016 at 15:32 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Hi Brett, On 2016-08-10 12:27 PM, Brett Cannon wrote:
Good suggestion. FWIW, in PEP 492, we called awaitables created with @types.coroutine as "generator-based coroutines". These days I see less and less people use @asyncio.coroutine and 'yield from'. Even less so know about @types.coroutine and how async/await tied to generators in the interpreter. This knowledge is now only required for people who implement/maintain frameworks like asyncio/twisted/curio. So I hope that we're already at the stage when people can just use async/await without really thinking how it's implemented. PEP 525 is another step in that direction -- making asynchronous iterators easy to use. Yury
participants (13)
-
Alexander Shorin -
Andrew Svetlov -
Brett Cannon -
Chris Angelico -
Cory Benfield -
Guido van Rossum -
Nick Coghlan -
Nickolai Novik -
Stefan Behnel -
Sven R. Kunze -
Thomas Güttler -
Vladimir Rutsky -
Yury Selivanov