[Python-ideas] The async API of the future: yield-from

Guido van Rossum guido at python.org
Tue Oct 16 04:10:46 CEST 2012

On Mon, Oct 15, 2012 at 1:14 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>> The main primitive I personally want out of an async API is a
>> task-based equivalent to concurrent.futures.as_completed() [1]. This
>> is what I meant about iteration being a bit of a mess: the way the
>> as_completed() works, the suspend/resume channel of the iterator
>> protocol is being used to pass completed future objects back to the
>> calling iterator. That means that channel *can't* be used to talk
>> between the coroutine and the scheduler,
> I had to read this a couple of times before I figured out
> what you're talking about, but I get it now.
> This is an instance of a general problem that was noticed
> back when I was discussing my cofunctions idea: using
> generator-based coroutines, it's not possible to have a
> "suspendable iterator", because that would require "yield"
> to have two conflicting meanings: "suspend this coroutine"
> on one hand, and "provide a value to my caller" on the
> other.
> Unfortunately, I suspect that a truly elegant solution to this
> problem will require yet another language addition -- something
> like
>    yield for item in subtask():
>       ...
> which would run a slightly different version of the iterator
> protocol in which values to be yield are wrapped somehow
> (I haven't figured out all the details yet).

I think I ran into a similar issue with NDB when defining iteration
over an asynchronous query. My solution:

  q = <some query specification>
  it = q.iter()  # Fire off the query to the datastore
  while (yield it.has_next_async()):  # Block until one result
    emp = it.next()  # Get the result that was buffered on the iterator
    print emp.name, emp.age  # Use it

--Guido van Rossum (python.org/~guido)

More information about the Python-ideas mailing list