Mailman 3 August 2016 - Python-ideas

PEP 525: Asynchronous Generators
by Yury Selivanov Aug. 11, 2016

Aug. 11, 2016

Hi, This is a new PEP to add asynchronous generators to Python 3.6. The PEP is also available at [1]. There is a reference implementation [2] that supports everything that the PEP proposes to add. [1] https://www.python.org/dev/peps/pep-0525/ [2] https://github.com/1st1/cpython/tree/async_gen Thank you! PEP: 525 Title: Asynchronous Generators Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Discussions-To: <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6 Post-History: 02-Aug-2016 Abstract ======== PEP 492 introduced support for native coroutines and ``async``/``await`` syntax to Python 3.5. It is proposed here to extend Python's asynchronous capabilities by adding support for *asynchronous generators*. Rationale and Goals =================== Regular generators (introduced in PEP 255) enabled an elegant way of writing complex *data producers* and have them behave like an iterator. However, currently there is no equivalent concept for the *asynchronous iteration protocol* (``async for``). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements ``__aiter__`` and ``__anext__`` to be able to use it in an ``async for`` statement. Essentially, the goals and rationale for PEP 255, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are **2x** faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated:: class Ticker: """Yield numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i The same can be implemented as a much simpler asynchronous generator:: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) Specification ============= This proposal introduces the concept of *asynchronous generators* to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). Asynchronous Generators ----------------------- A Python *generator* is any function containing one or more ``yield`` expressions:: def func(): # a function return def genfunc(): # a generator function yield We propose to use the same approach to define *asynchronous generators*:: async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield 42 The result of calling an *asynchronous generator function* is an *asynchronous generator object*, which implements the asynchronous iteration protocol defined in PEP 492. It is a ``SyntaxError`` to have a non-empty ``return`` statement in an asynchronous generator. Support for Asynchronous Iteration Protocol ------------------------------------------- The protocol requires two special methods to be implemented: 1. An ``__aiter__`` method returning an *asynchronous iterator*. 2. An ``__anext__`` method returning an *awaitable* object, which uses ``StopIteration`` exception to "yield" values, and ``StopAsyncIteration`` exception to signal the end of the iteration. Asynchronous generators define both of these methods. Let's manually iterate over a simple asynchronous generator:: async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 await gen.__anext__() # This line will raise StopAsyncIteration. Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. 3. Add two new methods to the ``sys`` module: ``set_asyncgen_finalizer()`` and ``get_asyncgen_finalizer()``. The idea behind ``sys.set_asyncgen_finalizer()`` is to allow event loops to handle generators finalization, so that the end user does not need to care about the finalization problem, and it just works. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_finalizer(old_finalizer) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) ``sys.set_asyncgen_finalizer()`` is thread-specific, so several event loops running in parallel threads can use it safely. Asynchronous Generator Object ----------------------------- The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: 1. ``agen.__aiter__()``: Returns ``agen``. 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one asynchronous generator iteration when awaited. 3. ``agen.asend(val)``: Returns an *awaitable*, that pushes the ``val`` object in the ``agen`` generator. When the ``agen`` has not yet been iterated, ``val`` must be ``None``. Example:: async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.2) g = gen() await g.asend(None) # Will return 42 after sleeping # for 0.1 seconds. await g.asend('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.2 seconds.) 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that throws an exception into the ``agen`` generator. Example:: async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after # sleeping for 0.1 seconds. v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after $ sleeping 0.2 seconds. 5. ``agen.aclose()``: Returns an *awaitable*, that throws a ``GeneratorExit`` exception into the generator. The *awaitable* can either return a yielded value, if ``agen`` handled the exception, or ``agen`` will be closed and the exception will propagate back to the caller. 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable name and qualified name attributes. 7. ``agen.ag_await``: The object that ``agen`` is currently *awaiting* on, or ``None``. This is similar to the currently available ``gi_yieldfrom`` for generators and ``cr_await`` for coroutines. 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: defined in the same way as similar attributes of standard generators. ``StopIteration`` and ``StopAsyncIteration`` are not propagated out of asynchronous generators, and are replaced with a ``RuntimeError``. Implementation Details ---------------------- Asynchronous generator object (``PyAsyncGenObject``) shares the struct layout with ``PyGenObject``. In addition to that, the reference implementation introduces three new objects: 1. ``PyAsyncGenASend``: the awaitable object that implements ``__anext__`` and ``asend()`` methods. 2. ``PyAsyncGenAThrow``: the awaitable object that implements ``athrow()`` and ``aclose()`` methods. 3. ``_PyAsyncGenWrappedValue``: every directly yielded object from an asynchronous generator is implicitly boxed into this structure. This is how the generator implementation can separate objects that are yielded using regular iteration protocol from objects that are yielded using asynchronous iteration protocol. ``PyAsyncGenASend`` and ``PyAsyncGenAThrow`` are awaitables (they have ``__await__`` methods returning ``self``) and are coroutine-like objects (implementing ``__iter__``, ``__next__``, ``send()`` and ``throw()`` methods). Essentially, they control how asynchronous generators are iterated: .. image:: pep-0525-1.png :align: center :width: 80% PyAsyncGenASend and PyAsyncGenAThrow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``PyAsyncGenASend`` is a coroutine-like object that drives ``__anext__`` and ``asend()`` methods and implements the asynchronous iteration protocol. ``agen.asend(val)`` and ``agen.__anext__()`` return instances of ``PyAsyncGenASend`` (which hold references back to the parent ``agen`` object.) The data flow is defined as follows: 1. When ``PyAsyncGenASend.send(val)`` is called for the first time, ``val`` is pushed to the parent ``agen`` object (using existing facilities of ``PyGenObject``.) Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 2. When ``PyAsyncGenASend.throw(*exc)`` is called for the first time, ``*exc`` is throwed into the parent ``agen`` object. Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 3. ``return`` statements in asynchronous generators raise ``StopAsyncIteration`` exception, which is propagated through ``PyAsyncGenASend.send()`` and ``PyAsyncGenASend.throw()`` methods. ``PyAsyncGenAThrow`` is very similar to ``PyAsyncGenASend``. The only difference is that ``PyAsyncGenAThrow.send()``, when called first time, throws an exception into the parent ``agen`` object (instead of pushing a value into it.) New Standard Library Functions and Types ---------------------------------------- 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator object. 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` methods to set up asynchronous generators finalizers in event loops. 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` introspection functions. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` function with a ``yield`` expression inside, therefore it's safe to introduce asynchronous generators in 3.6. Performance =========== Regular Generators ------------------ There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators:: def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) Improvements over asynchronous iterators ---------------------------------------- The following micro-benchmark shows that asynchronous generators are about **2.3x faster** than asynchronous iterators implemented in pure Python:: N = 10 ** 7 async def agen(): for i in range(N): yield i class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i Design Considerations ===================== ``aiter()`` and ``anext()`` builtins ------------------------------------ Originally, PEP 492 defined ``__aiter__`` as a method that should return an *awaitable* object, resulting in an asynchronous iterator. However, in CPython 3.5.2, ``__aiter__`` was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: ``__aiter__`` can still return an *awaitable* with a ``DeprecationWarning`` being issued. Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot add a synchronous implementation of ``aiter()`` built-in. Therefore, it is proposed to wait until Python 3.7. Asynchronous list/dict/set comprehensions ----------------------------------------- Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. Asynchronous ``yield from`` --------------------------- While it is theoretically possible to implement ``yield from`` support for asynchronous generators, it would require a serious redesign of the generators implementation. ``yield from`` is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. And to compose asynchronous generators a simple ``async for`` loop can be used:: async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v Why the ``asend()`` and ``athrow()`` methods are necessary ---------------------------------------------------------- They make it possible to implement concepts similar to ``contextlib.contextmanager`` using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern:: @async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from ``__anext__`` object, but it is hard to do that correctly. Adding explicit ``asend()`` and ``athrow()`` will pave a safe way to accomplish that. In terms of implementation, ``asend()`` is a slightly more generic version of ``__anext__``, and ``athrow()`` is very similar to ``aclose()``. Therefore having these methods defined for asynchronous generators does not add any extra complexity. Example ======= A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay):: async def ticker(delay, to): for i in range(to): yield i await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() Implementation ============== The complete reference implementation is available at [1]_. References ========== .. [1] https://github.com/1st1/cpython/tree/async_gen Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

13 43

Re: [Python-ideas] PEP 525: Asynchronous Generators
by Adam Bartoš Aug. 9, 2016

Aug. 9, 2016

Hello, when I was reading the PEP, it surprised me that return is not allowed in an async generator. You can transparently decompose an ordinary function, and yield from was added to be able to do the same with generators, so it seems natural to be able to decompose async generators the same way – using (await) yield from and return. I found the explanation in the section about async yield from, but maybe a link to that section might be added as explanation why return produces a SyntaxError. In ideal world it seems that the transition from ordinary functions to generators and the transition from ordinary functions to async functions (coroutines) are orthogonal, and async gnerators are the natural combination of both transitions. I thing this is true despite the fact that async functions are *implemented* on top of generators. I imagine the situation as follows: being a generator adds an additional output channel – the one leading to generator consumer, and being an async function adds an additional output channel for communication with the scheduler. An async generator somehow adds both these channels. I understand that having the concepts truly ortogonal would mean tons of work, I just think it's pity that something natural doesn't work only because of implementation reasons. However, introducing async generators is an important step in the right way. Thank you for that. Regards, Adam Bartoš

2 1

Looking for a co-author for a PEP about variable declarations
by Guido van Rossum Aug. 8, 2016

Aug. 8, 2016

I've written up a wall of text at https://github.com/python/typing/issues/258 and would like to develop it into a PEP at an accelerated schedule. Is anyone interested in helping out? This *could* be a great mentoring opportunity. -- --Guido van Rossum (python.org/~guido)

4 3

Looking for a co-author for a PEP about variable declarations
by Ivan Levkivskyi Aug. 8, 2016

Aug. 8, 2016

> I've written up a wall of text at > https://github.com/python/typing/issues/258 and would like to develop it > into a PEP at an accelerated schedule. Is anyone interested in helping out? > This *could* be a great mentoring opportunity. I think this is a great idea and I will be glad to help. -- Ivan

1 0

Make "raise" an expression
by אלעזר Aug. 6, 2016

Aug. 6, 2016

The raise statement cannot be used as an expression, even though there is no theoretical reason for it not to. I don't see any practical reason either - except the fact that it requires changing the syntax. I have found this idea mentioned by Anthony Lee in a thread from 2014, regarding "or raise", https://mail.python.org/pipermail/python-ideas/2014-November/029921.html. B ut there is nothing against the idea there. Yes, it could easily be implemented using a function, but this seems to be an argument against *augmenting* the language, not against *changing* it. Moreover, this function will have uglier name, since `raise` is taken. (It could have been the other way around - omitting the statement entirely, leaving only a built in function `raise()`. I'm not suggesting that, of course) As mentioned there, it does add a single line in the traceback - which is not immediately what you are looking for. Python's tendency towards explicit control flow is irrelevant in this case. An exception can already be raised from anywhere, at any time, and there's no way see it from the source code (unlike e.g. break/continue). Such a change will (only) break external tools that use the ast module, or tools analyzing Python code that uses this feature. This happens at every language change, so it can be done together with the next such change. Use cases: result = f(args) or raise Exception(...) result = f(x, y) if x > y else raise Exception(...) f(lambda x: max(x, 0) or raise Exception(...)) d = defaultdict(list) ... d[v] or raise KeyError(v) try: return foo() except: return try_bar() or raise To be honest, I'm not sure that this change is worthwhile. Admittedly the use cases are weak, and in some ways this is complementary of the (sadly rejected) "except expression". I would still like to "finish that thought", so - why not? ~Elazar

4 4

Trial balloon: adding variable type declarations in support of PEP 484
by Guido van Rossum Aug. 6, 2016

Aug. 6, 2016

PEP 484 doesn't change Python's syntax. Therefore it has no good syntax to offer for declaring the type of variables, and instead you have to write e.g. a = 0 # type: float b = [] # type: List[int] c = None # type: Optional[str] I'd like to address this in the future, and I think the most elegant syntax would be to let you write these as follows: a: float = 0 b: List[int] = [] c: Optional[str] = None (I've considered a 'var' keyword in the past, but there just are too many variables named 'var' in my code. :-) There are some corner cases to consider. First, to declare a variable's type without giving it an initial value, we can write this: a: float Second, when these occur in a class body, they can define either class variables or instance variables. Do we need to be able to specify which? Third, there's an annoying thing with tuples/commas here. On the one hand, in a function declaration, we may see (a: int = 0, b: str = ''). On the other hand, in an assignment, we may see a, b = 0, '' Suppose we wanted to add types to the latter. Would we write this as a, b: int, str = 0, '' or as a: int, b: str = 0, '' ??? Personally I think neither is acceptable, and we should just write it as a: int = 0 b: str = '' but this is a slight step back from a, b = 0, '' # type: (int, str) -- --Guido van Rossum (python.org/~guido)

25 101

Addition to operator module: starcaller
by Daniel Spitz Aug. 6, 2016

Aug. 6, 2016

Hello, I have run into a use case where I want to create a wrapper object that will star-unpack a sequence of arguments to pass to a function. Currently one way to construct this is to create a function: def starcaller(f): def wrapper(args): return f(*args) return wrapper Such a function feels simple enough, and specific enough to the language, that it would fit in well in the operator module of the standard library. We already have a similar representation of this functionality in itertools.starmap. Whereas the nested function implementation above will produce unpickleable objects (due to the closure), a straightforward c implementation will produce pickleable ones, making them useful for parallel applications. Thanks, Dan Spitz

7 11

Re: [Python-ideas] Consider adding clip or clamp function to math
by Steven D'Aprano Aug. 4, 2016

Aug. 4, 2016

On Wed, Aug 03, 2016 at 11:04:33AM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >In any case, clamping is based of < and > comparisons, which are > >well-specified by IEEE 754 even when NANs are included: > > Those rules are not enough to determine the behaviour of > functions such as min, max and clamp, though. IEEE-754 specifies the behaviour of min() and max() with a NAN argument. See my previous email. > >>why not say that passing NaNs as bounds will result in NaN result? > > > >Because that means that EVERY call to clamp() has to convert both bounds > >to float and see if they are NANs. > > No, it doesn't -- it only needs to check whether they are > NaN if they're *already* floats. If they're not floats, > they're obviously not NaN, so just leave them alone. That's wrong. Decimal has NANs. Other numeric types could potentially have NANs too. > (Actually, that's not quite true, since they could be a > custom type with its own notion of NaN-ness -- maybe we > could do with an __isnan__ protocol?) I would like to see a common "isnan" function or method for both floats and Decimal, because I'm sick of writing code like: try: flag = x.is_nan() # Decimal? except AttributeError: flag = math.isnan(x) # float Note that the spelling is different too, for extra sadness :-( -- Steve

2 2

Re: [Python-ideas] Trial balloon: adding variable type declarations in support of PEP 484
by אלעזר Aug. 4, 2016

Aug. 4, 2016

> > > > (FWIW, I like the <var>: <type> notation more than any alternative > > proposed so far.) > > Me too. :-) > Just wanted to note that the typo lamda: int Becomes syntactically correct. Maybe there are other similar problems. >

3 2

Re: [Python-ideas] Trial balloon: adding variable type
by אלעזר Aug. 4, 2016

Aug. 4, 2016

> > Date: Wed, 3 Aug 2016 09:11:42 -0700 > From: Guido van Rossum <guido(a)python.org> ... It would be much cleaner if you could declare the > variable before the first `if`. But picking a good initializer is > tricky, especially if you have a type that does not include None. > PEP-484 suggests ellipsis for this. ~Elazar

4 3