[Python-ideas] async/await in Python

[Yury Selivanov]

Hi Yann,


Thanks for the feedback! Answers below.


On 2015-04-20 5:39 PM, Yann Kaiser wrote:
> Browsing the PEP and this thread, I can't help but notice there is little
> mention of the possibility (or rather, the lack thereof in the current
> version of the proposal) of writing async generators(ie. usable in async
> for/await for) within the newly proposed coroutines other than "yield
> [from] is not allowed within an async function".
>
> I think this should be given more thought, despite the technical
> difficulties involved (namely, new-style coroutines are still implemented
> as generators).
>
> While the proposal already makes it less of a burden to write asynchronous
> iterables through "async for", __aiter__ and __anext__, in synchronous
> systems writing a generator function(or having one as __iter__) is most
> often preferred over writing a class with __iter__ and __next__ methods.
>
> A use case I've come across using asyncio, assuming yield could be somehow
> allowed within a new-style coroutine:
>
> A web API provides paginated content with few filtering options. You have
> something that fetches those pages:
>
>      async def fetch_pages(url, params):
>          while True:
>              page = await(request(url, params))
>              feed = page.decode_json()
>              if not feed['items']: # no items when we reach the last page
>                  return
>              yield feed
>
> It would be sorta okay to have this as a class with __aiter__ and __anext__
> methods. It would be necessary anyway.
>
> Now we want an iterator for every item. We won't really want to iterate on
> pages anyway, we'll want to iterate on items. With yield available, this is
> really easy!
>
>      async def fetch_items(pages):
>          async for page in pages:
>              yield from page['items']
>
> Without, it becomes more difficult
>
>      class ItemFetcher:
>          def __init__(self, pages):
>              self.pages = pages
>
>          async def __aiter__(self):
>              self.pages_itor = await self.pages.__aiter__()
>              self.items = iter(())
>              return self
>
>          async def __anext__(self):
>              try:
>                  try:
>                      return next(self.page)
>                  except StopIteration:
>                      page = await self.pages_itor.__anext__()
>                      self.page = iter(page['items'])
>                      return next(self.page)
>              except StopIteration as exc:
>                  raise StopAsyncIteration(StopIteration.value) from exc
>
> Whoa there! This is complicated and difficult to understand. But we really
> want to iterate on items so we leave it in anyway.

While I understand what you're trying to achieve here, the
code doesn't look correct.  Could you please clone the ref
implementation and make it work first?

>
> Here the language failed already. It made a monster out of what could have
> been expressed simply.
>
> What if we only want new items?
>
>      async def new_items(items):
>          async for item in items:
>              if is_new_item(item):
>                  yield item
>
> Without yield:
>
>      class NewItems:
>          def __init__(self, items):
>              self.items = items
>
>          async def __aiter__(self):
>              self.itor = await self.items.__aiter__()
>              return self
>
>          async def __anext__(self):
>              async for item in self.itor:
>                  if is_new_item(item):
>                      return item
>              raise StopAsyncIteration
>
> This isn't as bad as the previous example, but I'll be the first to admit
> I'll use "if not is_new_item(item): continue" in client code instead.
>
> In bullet point form, skipping the possibility of having yield within
> coroutine functions causes the following:
>
> * To write async iterators, you have to use the full async-iterable class
> form.
> * Iterators written in class form can't have a for loop over their
> arguments, because their behavior is spread over multiple methods.
> * Iterators are unwieldy when used manually
> * Async iterators even more so
> => If you want to make an async iterator, it's complicated
> => If you want to make an async iterator that iterates over an iterator,
> it's more complicated
> => If you want to make an async iterator that iterates over an async
> iterator, it's even more complicated
>
> I therefore think a way to have await and yield coexist should be looked
> into.
>
> Solutions include adding new bytecodes or adding a flag to the YIELD_VALUE
> and YIELD_FROM bytecodes.
>
> -- Yann


All in all, I understand your case.

But also, I know that you have this case because you're
trying to think in generators and how to combine them. Which is
a good pattern, but unfortunately, this pattern had never
worked with generator-based coroutines either.

Imagine that there were no PEP 492.  That you only can use
asyncio and 'yield from'.  You would design your code
in a different way then.

I spent a great deal of time thinking if it's possible to
combine yields and awaits in one coroutine function.

Unfortunately, even if it is possible, it will require horrible
hacks and will complicate the implementation tremendously.

Moreover, I think that combining 'yield' and 'yield from'
expressions with 'await' will only create confusion and contradicts
with the main intent of the PEP, which is to *remove that confusion*.

I'd prefer to make PEP 492 maximally minimal in this regard.
Since it's prohibited to use 'yield' in coroutines, we may
allow it in later Python versions.  (Although I'm certain, that
we'll need a new keyword for 'yield' in coroutines.)

At this point, PEP 492 is created *specifically* to address
existing pain-points that asyncio developers have, no more.

I don't want to create new programming patterns or new concepts
of generator-coroutine hybrid.  I think that can and should be
covered in future PEPs in future Pythons.

If someone smarter than me can figure out a way to do this
in a non-confusing way that won't require duplicating genobject.c
and fourth of ceval.c I'd be glad to update the PEP.

Thanks,
Yury