[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

On Mon, Oct 22, 2012 at 3:09 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I think there's an assumption behind this whole async tasks discussion
> that the tasks being scheduled are I/O bound. We're trying to overlap
> CPU activity with I/O, and different I/O activities with each other.
> We're *not* trying to achieve concurrency of CPU-bound tasks -- the
> GIL prevents that anyway for pure Python code.

Right. Of course.

> The whole Windows IOCP thing, on the other hand, seems to be geared
> towards having a pool of threads, any of which can handle any I/O
> operation. That's not helpful for us; when one of our tasks blocks
> waiting for I/O, the completion of that I/O must wake up *that particular
> task*, and it must be run using the same OS thread that was running
> it before.

The reason we can't ignore IOCP is that it is apparently the *only*
way to do async I/O in a scalable way. The only other polling
primitive available is select() which does not scale. (Or so it is
asserted by many folks; I haven't tested this, but I believe the
argument against select() scaling in general.)

> I gather that Windows provides a way of making an async I/O request
> and specifying a callback for that request. If that's the case, do
> we need to bother with an IOCP at all? Just have the callback wake
> up the associated task directly.

AFAICT the way to do that goes through IOCP...

-- 
--Guido van Rossum (python.org/~guido)