[Python-ideas] async/await in Python

[Yann Kaiser]

Browsing the PEP and this thread, I can't help but notice there is little
mention of the possibility (or rather, the lack thereof in the current
version of the proposal) of writing async generators(ie. usable in async
for/await for) within the newly proposed coroutines other than "yield
[from] is not allowed within an async function".

I think this should be given more thought, despite the technical
difficulties involved (namely, new-style coroutines are still implemented
as generators).

While the proposal already makes it less of a burden to write asynchronous
iterables through "async for", __aiter__ and __anext__, in synchronous
systems writing a generator function(or having one as __iter__) is most
often preferred over writing a class with __iter__ and __next__ methods.

A use case I've come across using asyncio, assuming yield could be somehow
allowed within a new-style coroutine:

A web API provides paginated content with few filtering options. You have
something that fetches those pages:

    async def fetch_pages(url, params):
        while True:
            page = await(request(url, params))
            feed = page.decode_json()
            if not feed['items']: # no items when we reach the last page
                return
            yield feed

It would be sorta okay to have this as a class with __aiter__ and __anext__
methods. It would be necessary anyway.

Now we want an iterator for every item. We won't really want to iterate on
pages anyway, we'll want to iterate on items. With yield available, this is
really easy!

    async def fetch_items(pages):
        async for page in pages:
            yield from page['items']

Without, it becomes more difficult

    class ItemFetcher:
        def __init__(self, pages):
            self.pages = pages

        async def __aiter__(self):
            self.pages_itor = await self.pages.__aiter__()
            self.items = iter(())
            return self

        async def __anext__(self):
            try:
                try:
                    return next(self.page)
                except StopIteration:
                    page = await self.pages_itor.__anext__()
                    self.page = iter(page['items'])
                    return next(self.page)
            except StopIteration as exc:
                raise StopAsyncIteration(StopIteration.value) from exc

Whoa there! This is complicated and difficult to understand. But we really
want to iterate on items so we leave it in anyway.

Here the language failed already. It made a monster out of what could have
been expressed simply.

What if we only want new items?

    async def new_items(items):
        async for item in items:
            if is_new_item(item):
                yield item

Without yield:

    class NewItems:
        def __init__(self, items):
            self.items = items

        async def __aiter__(self):
            self.itor = await self.items.__aiter__()
            return self

        async def __anext__(self):
            async for item in self.itor:
                if is_new_item(item):
                    return item
            raise StopAsyncIteration

This isn't as bad as the previous example, but I'll be the first to admit
I'll use "if not is_new_item(item): continue" in client code instead.

In bullet point form, skipping the possibility of having yield within
coroutine functions causes the following:

* To write async iterators, you have to use the full async-iterable class
form.
* Iterators written in class form can't have a for loop over their
arguments, because their behavior is spread over multiple methods.
* Iterators are unwieldy when used manually
* Async iterators even more so
=> If you want to make an async iterator, it's complicated
=> If you want to make an async iterator that iterates over an iterator,
it's more complicated
=> If you want to make an async iterator that iterates over an async
iterator, it's even more complicated

I therefore think a way to have await and yield coexist should be looked
into.

Solutions include adding new bytecodes or adding a flag to the YIELD_VALUE
and YIELD_FROM bytecodes.

-- Yann

On Mon, 20 Apr 2015 at 08:43 Guido van Rossum <guido at python.org> wrote:

> On Fri, Apr 17, 2015 at 5:21 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
>
>> Yury Selivanov wrote:
>>
>>>
>>> Here's my proposal to add async/await in Python.
>>>
>>
>> You've essentially reinvented PEP 3152 - Cofunctions.
>>
>> https://www.python.org/dev/peps/pep-3152/
>>
>
> I *thought* Yury's proposal sounded familiar! :-)
>
>
>> Here's a summary of the relationships between them:
>>
>> PEP 3152                  Nearest equivalent in Yury's PEP
>> --------                  --------------------------------
>>
>> codef f(args):            async def f(args):
>>
>> cocall f(args)            await f(args)
>>
>> __cocall__                __await__
>>
>> costart()                 async_def()
>>
>> There is currently no equivalent of "async for" and
>> "async with" in PEP 3152, but they could easily be added.
>> I would probably spell them "cofor" and "cowith".
>>
>
> Unfortunately that's a lot of new reserved words, and they aren't even
> words. :-) I like the elegance of Yury's idea of prefixing various
> statements with "async". It also happens to easy introduction, since the
> lexer can be adapted (for a few releases at least) not to treat "async" as
> a reserved word unless followed by "def" etc.
>
>
>> As the author of PEP 3152 I'm obviously biased, but I
>> think my spellings are more elegant and less disruptive
>> to reading of the code.
>>
>
> As the BDFL I'm perhaps also biased (Yury went over early stages of his
> draft with me at PyCon) but I prefer his spellings.
>
>
>> PEP 3152 is currently marked as deferred. Maybe it's
>> time to revive it? If Yury's pep is to be considered,
>> we ought to discuss the relative merits of the two.
>
>
>  I wonder if, instead of putting forward a competing proposal that is so
> similar, you could help Yury with PEP 492? I'm sure he'd happily take you
> as a co-author.
>
> I have one question about PEP 3152. IIUC, it states that cofunctions
> cannot be called except using cocall, and there seems to be syntax that
> enforces that the "argument" to cocall looks like a function call. This
> would rule out things that are normally always allowed in Python, if you
> have
> ```
> return foo(args)
> ```
> you can replace that with
> ```
> r = foo(args)
> return r
> ```
> But the way I read PEP 3152 the call has to occur syntactically in the
> cocall form, so
> ```
> cocall foo(args)
> ```
> cannot be replaced with
> ```
> x = foo(args)
> cocall x
> ```
> This ability is occasionally useful in asyncio, for example when creating
> a task, we can write
> ```
> x = Task(foo(args))
> ```
> and then someone else can write
> ```
> yield from x
> ```
> Your cocall syntax is lacking such ability, since cocall insist on call
> syntax ("cocall x" is a syntax error). It would seem that in general your
> cocall is not a substitution for yield from when used with a future or
> task, and that looks like a real deficiency to me.
>
> --
> --Guido van Rossum (python.org/~guido)
>  _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150420/b71b767a/attachment.html>