[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

On Sun, Oct 21, 2012 at 10:30 PM, Steve Dower <Steve.Dower at microsoft.com> wrote:
[Stuff about Futures and threads]

Personally, I'm interested in designing a system, including an event
loop, where you can rely on the properties of cooperative scheduling
to avoid ever touching (OS) threading locks. I think such a system
should be "pure" and all interaction with threads should be mediated
by the event loop. (It's okay if this means that the implementation of
the event loop must at some point acquire a threading lock.) The
Futures used by the tasks to coordinate amongst themselves should not
require locking -- they should themselves be able to rely on the
guarantees of the event loop not to invoke multiple callbacks in
parallel.

IIUC you can do this on Windows with IOCP too, simply by only having a
single thread reading events.

>> > > And the Futures-only-in-public-APIs rule seems to encourage less efficient solutions.
>> >
>> > Personally, I'd prefer developers to get a correct solution without having to
>> > understand how the whole thing works (the "pit of success"). I'm also sceptical
>> > of any other rule being as portable and composable - I don't think a standard
>> > library should have APIs where "you must only call this function with yield-from".
>> > ('await' in C# is not compulsory - you can take the Task returned from an async
>> > method and do whatever you like with it.)
>>
>> Surely "whatever you like" is constrained by whatever the Task type
>> defines. Maybe it looks like a Future and has a blocking method to
>> wait for the result, like .result() on concurrent.futures.Future? If
>> you want that functionality for generators you just have to call some
>> function, passing it the generator as an argument. Remember, Python
>> doesn't consider that an inferior choice of API design compared to
>> making something a method of the object itself -- witness len(),
>> repr() and many others.
>
> I'm interested that you skipped my "portable and composable" claim and went straight for my aside about another language. I'd prefer to avoid introducing top-level names, especially since this is an API with plenty of predecessors... what sort of trouble would we be having if sched or asyncore had claimed 'wait()'? Even more so because it's Python, since it is so easy to overwrite the value.

Sorry, probably just got distracted (I was reading on three different
devices while on a family outing :-).

But my answer is short: to me, the PEP 380 style is perfectly portable
and composable. If you think it isn't, please elaborate.

> (And as it happens, Task handles both the asynchrony and the callbacks, so it looks a bit like Thread and Future mixed together. Personally, I prefer to keep the concepts separate.)

Same here.

-- 
--Guido van Rossum (python.org/~guido)