[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Benoit Chesneau]

On Oct 23, 2012, at 6:51 PM, Andrew Francis <andrewfr_ice at yahoo.com> wrote:

> Hi Benoit and folks:
> 
> >Message: 3
> >Date: Tue, 23 Oct 2012 09:19:59 +0200
> >From: Benoit Chesneau <benoitc at gunicorn.org>
> >To: Guido van Rossum <guido at python.org>
> >Cc: Python-Ideas <python-ideas at python.org>
> >Subject: Re: [Python-ideas] yield from multiple iterables (was Re: The
>  >   async    API of the future: yield-from)
> >Message-ID: <BE74DBDE-1965-47F0-99B9-27F0C7CD574C at gunicorn.org>
> >Content-Type: text/plain; charset=windows-1252
> 
> (I learnt about this mailing list from Christian Tismer's post in the Stackless mailing list and I am catching up)
> 
> >I myself toying with the idea of porting the Go concurrency model to Python [4] using greenlets and pyuv. Both the scheduler >and the way IOs are handled:
> 
> >- In Go all coroutines are independent from each others and can only communicate via channel. Which has the advantage to >allows them to run on different threads when one is blocking. In normal case they are mostly working like grrenlets on a single >thread and are simply scheduled in a round-robin way. (mostly like in stackless). On the difference that goroutines can be >executed in parallel. When one is blocking another thread will be created to handle other goroutines in the runnable queue.
> 
> 
> What aspect of the Go concurrency model? Maybe you already know this but  Go and Stackless Python share a common ancestor: Limbo. More specifically the way channels work. 

Indeed :) I would have say Plan 9 and tasks inside but right channnels are in limbo too.

> 
> This may be tangential to the discussion but in the past, I have used the stackless.py module in conjunction with CPython and greenlets to rapidly prototype parts of Go's model that are not present in Stackless, i.e. the select (ALT) language feature. 
> Rob Pike and Russ Cox were really helpful in answering my questions. Newer stackless.py implementations use 
> continuelets so look for an older PyPy implementation. 
> 
> I have also prototyped a subset of Polyphonic C# join patterns.  After I got the prototype running, I had an interesting discussion with the authors of "Scalable Join Patterns."

Yes saw that. And actually some part of the Task code is based on stackless.py  but using greenlets, Channels have been slightly modified to be thread-safe and support buffering. Did you release your code somewhere ? It could be interesting to put the experience further.
> 
> For networking support, I run Twisted as a tasklet. There are a few tricks to make Stackless and Twisted co-operate.

I plan to release a new version of flower this week. For now i am also running a libuv eventloop in a tasklet, but since the tasklet need to be blocking for performance, i am writing some new code to run the tasklet in its proper thread when needed. Not sure how it will go.

Current implementation handle events when the scheduler come on the eventloop which isn't the more efficient way imo.

Another thing to considers is also rust. Rust is using libuv and put the eventloop in its own task thread :

http://dl.rust-lang.org/doc/0.4/std/uv_global_loop.html

I find this idea quite elegant.

Best,

- benoît
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121023/40a73ea1/attachment.html>