[Python-ideas] The async API of the future: yield-from

[Guido van Rossum]

On Sun, Oct 14, 2012 at 12:12 AM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> I've had some thoughts on why I'm uncomfortable
> about this kind of pattern:
>
>    data = yield sock.async_read(1024)
>
> The idea here is that sock.async_read() returns a
> Future or similar object that performs the I/O and
> waits for the result.
>
> However, reading the data isn't necessarily the point
> at which the suspension actually occurs. If you're
> using a select-style event loop, the async read
> operation breaks down into
>
>    1. Wait for data to arrive on the socket
>    2. Read the data
>
> So the implementation of sock.async_read() is going
> to have to create another Future to handle waiting
> for the socket to become ready. But then the outer
> Future is an unnecessary complication, because you
> could get the same effect by defining
>
>    def async_read(self, length):
>       yield future_to_wait_for_fd(self.fd)
>       return os.read(self.fd, length)
>
> and calling it using
>
>    data = yield from sock.async_read(1024)
>
> If Futures are to appear anywhere, they should only
> be at the very bottom layer, at the transition
> between generator and non-generator code. And the
> place where that transition occurs depend on how
> the lower levels are implemented. If you're using
> IOCP instead of select, for example, you need to
> do things the other way around:
>
>    1. Start the read operation
>    2. Wait for it to complete
>
> So I feel that all public APIs should be functions
> called using yield-from, leaving it up to the
> implementation to decide if and where Futures
> become involved.

A logical and consistent conclusion. I actually agree: in NDB, where
all I have is "yield <future>" I have a similar guideline: all public
async APIs return a Future and must be waited on using yield, and only
at the lowest level are other types primitives involved (bare App
Engine RPCs, callbacks).

-- 
--Guido van Rossum (python.org/~guido)