[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

Thanks for the pointer to and description of libuv; it had come up in
my research yet but so far I have not looked it up actively. Now I
will. Also thanks for your reminder of the Goroutine model -- this is
definitely something to look at for inspiration as well. (Though does
Go run on Windows? Or is it part of a secret anti-Microsoft plan? :-)

--Guido

On Tue, Oct 23, 2012 at 12:19 AM, Benoit Chesneau <benoitc at gunicorn.org> wrote:
>
> On Oct 22, 2012, at 4:59 PM, Guido van Rossum <guido at python.org> wrote:
>
>> On Sun, Oct 21, 2012 at 10:30 PM, Steve Dower <Steve.Dower at microsoft.com> wrote:
>> [Stuff about Futures and threads]
>>
>> Personally, I'm interested in designing a system, including an event
>> loop, where you can rely on the properties of cooperative scheduling
>> to avoid ever touching (OS) threading locks. I think such a system
>> should be "pure" and all interaction with threads should be mediated
>> by the event loop. (It's okay if this means that the implementation of
>> the event loop must at some point acquire a threading lock.) The
>> Futures used by the tasks to coordinate amongst themselves should not
>> require locking -- they should themselves be able to rely on the
>> guarantees of the event loop not to invoke multiple callbacks in
>> parallel.
>>
>> IIUC you can do this on Windows with IOCP too, simply by only having a
>> single thread reading events.
>>
>
> Maybe it is worth to have a look on libuv and the way it mixes threads and  and event loop [1]. Libuv is one of the good event loop around able to use IOCP and other events systems on other arch (kqueue, …) and I was thinking when reading all the exchange around that it would perfectly fit in our cases. Or at least something like it:
>
> - It provides a common api for IO watchers: read, write, writelines, readable, writeable that can probably be extend over remote systems
> - Have a job queue system for threds that is working mostly like the Futures but using the event loop
>
> In any case there is a pyuv binding [2] if some want to test. Even a twisted reactor [3]
>
> I myself toying with the idea of porting the Go concurrency model to Python [4] using greenlets and pyuv. Both the scheduler and the way IOs are handled:
>
> - In Go all coroutines are independent from each others and can only communicate via channel. Which has the advantage to allows them to run on different threads when one is blocking. In normal case they are mostly working like grrenlets on a single thread and are simply scheduled in a round-robin way. (mostly like in stackless). On the difference that goroutines can be executed in parallel. When one is blocking another thread will be created to handle other goroutines in the runnable queue.
>
> - For I/Os it exists a common api to all Connections and Listeners (Conn & Listen classes) that generally ask on a poll server. This poll server has for only task to register FDs and wake up the groutines that wait on read or fd events. This this poll server is running in a blocking loop it is automatically let by the scheduler in a thread. This pol server could be likely be replaced by an event loop if someone want.
>
> In my opinion the Go concurrency & memory model [5] could perfectly fit in the Python world and I'm surprised none already spoke about it.
>
> In flower greenlets could probably be replaced by generators but i like the API proposed by any coroutine pattern. I wonder if continulets [6] couldn't be ported in cpython to handle that…
>
> - benoît
>
>
> [1] http://nikhilm.github.com/uvbook/threads.html & http://github.com/joyent/libuv
> [2] https://github.com/saghul/pyuv
> [3] https://github.com/saghul/twisted-pyuv
> [4] https://github.com/benoitc/flower
> [5] http://golang.org/ref/mem
> [6] http://doc.pypy.org/en/latest/stackless.html#continulets



-- 
--Guido van Rossum (python.org/~guido)