Re: [Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)
Hi Benoit and folks:
Message: 3 Date: Tue, 23 Oct 2012 09:19:59 +0200 From: Benoit Chesneau <benoitc@gunicorn.org> To: Guido van Rossum <guido@python.org> Cc: Python-Ideas <python-ideas@python.org> Subject: Re: [Python-ideas] yield from multiple iterables (was Re: The > async API of the future: yield-from) Message-ID: <BE74DBDE-1965-47F0-99B9-27F0C7CD574C@gunicorn.org> Content-Type: text/plain; charset=windows-1252
(I learnt about this mailing list from Christian Tismer's post in the Stackless mailing list and I am catching up)
I myself toying with the idea of porting the Go concurrency model to Python [4] using greenlets and pyuv. Both the scheduler >and the way IOs are handled:
- In Go all coroutines are independent from each others and can only communicate via channel. Which has the advantage to >allows them to run on different threads when one is blocking. In normal case they are mostly working like grrenlets on a single >thread and are simply scheduled in a round-robin way. (mostly like in stackless). On the difference that goroutines can be >executed in parallel. When one is blocking another thread will be created to handle other goroutines in the runnable queue.
What aspect of the Go concurrency model? Maybe you already know this but Go and Stackless Python share a common ancestor: Limbo. More specifically the way channels work. This may be tangential to the discussion but in the past, I have used the stackless.py module in conjunction with CPython and greenlets to rapidly prototype parts of Go's model that are not present in Stackless, i.e. the select (ALT) language feature. Rob Pike and Russ Cox were really helpful in answering my questions. Newer stackless.py implementations use continuelets so look for an older PyPy implementation. I have also prototyped a subset of Polyphonic C# join patterns. After I got the prototype running, I had an interesting discussion with the authors of "Scalable Join Patterns." For networking support, I run Twisted as a tasklet. There are a few tricks to make Stackless and Twisted co-operate. Cheers, Andrew
On Oct 23, 2012, at 6:51 PM, Andrew Francis <andrewfr_ice@yahoo.com> wrote:
Hi Benoit and folks:
Message: 3 Date: Tue, 23 Oct 2012 09:19:59 +0200 From: Benoit Chesneau <benoitc@gunicorn.org> To: Guido van Rossum <guido@python.org> Cc: Python-Ideas <python-ideas@python.org> Subject: Re: [Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from) Message-ID: <BE74DBDE-1965-47F0-99B9-27F0C7CD574C@gunicorn.org> Content-Type: text/plain; charset=windows-1252
(I learnt about this mailing list from Christian Tismer's post in the Stackless mailing list and I am catching up)
I myself toying with the idea of porting the Go concurrency model to Python [4] using greenlets and pyuv. Both the scheduler >and the way IOs are handled:
- In Go all coroutines are independent from each others and can only communicate via channel. Which has the advantage to >allows them to run on different threads when one is blocking. In normal case they are mostly working like grrenlets on a single >thread and are simply scheduled in a round-robin way. (mostly like in stackless). On the difference that goroutines can be >executed in parallel. When one is blocking another thread will be created to handle other goroutines in the runnable queue.
What aspect of the Go concurrency model? Maybe you already know this but Go and Stackless Python share a common ancestor: Limbo. More specifically the way channels work.
Indeed :) I would have say Plan 9 and tasks inside but right channnels are in limbo too.
This may be tangential to the discussion but in the past, I have used the stackless.py module in conjunction with CPython and greenlets to rapidly prototype parts of Go's model that are not present in Stackless, i.e. the select (ALT) language feature. Rob Pike and Russ Cox were really helpful in answering my questions. Newer stackless.py implementations use continuelets so look for an older PyPy implementation.
I have also prototyped a subset of Polyphonic C# join patterns. After I got the prototype running, I had an interesting discussion with the authors of "Scalable Join Patterns."
Yes saw that. And actually some part of the Task code is based on stackless.py but using greenlets, Channels have been slightly modified to be thread-safe and support buffering. Did you release your code somewhere ? It could be interesting to put the experience further.
For networking support, I run Twisted as a tasklet. There are a few tricks to make Stackless and Twisted co-operate.
I plan to release a new version of flower this week. For now i am also running a libuv eventloop in a tasklet, but since the tasklet need to be blocking for performance, i am writing some new code to run the tasklet in its proper thread when needed. Not sure how it will go. Current implementation handle events when the scheduler come on the eventloop which isn't the more efficient way imo. Another thing to considers is also rust. Rust is using libuv and put the eventloop in its own task thread : http://dl.rust-lang.org/doc/0.4/std/uv_global_loop.html I find this idea quite elegant. Best, - benoît
Hi Benoit: ________________________________ From: Benoit Chesneau <benoitc@gunicorn.org> To: Andrew Francis <andrewfr_ice@yahoo.com> Cc: "python-ideas@python.org" <python-ideas@python.org> Sent: Tuesday, October 23, 2012 5:48 PM Subject: Re: [Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from) On Oct 23, 2012, at 6:51 PM, Andrew Francis <andrewfr_ice@yahoo.com> wrote: AF>This may be tangential to the discussion but in the past, I have used the stackless.py module in conjunction with AF>CPython and greenlets to rapidly prototype parts of Go's model that are not present in Stackless, i.e. the select (ALT) AF>language feature. Rob Pike and Russ Cox were really helpful in answering my questions. Newer stackless.py AF>implementations use continuelets so look for an older PyPy implementation. AF>I have also prototyped a subset of Polyphonic C# join patterns. After I got the prototype running, I had an interesting AF>discussion with the authors of "Scalable Join Patterns."
Yes saw that. And actually some part of the Task code is based on stackless.py but using greenlets, >Channels have been slightly modified to be thread-safe and support buffering. Did you release your code >somewhere ? It could be interesting to put the experience further.
You may be mistaking my work with someone else. I didn't add buffering but that t is relatively easy to do without altering Stackless Python's internals. However I believe that synchronous channels with buffering is a simple and powerful concurrency model. Go's implementers got it right. John Reppy (currently a NSF director) talks about synchronous channel's power in a Concurrent ML book. If you go to to the Stackless repository example page http://code.google.com/p/stacklessexamples/wiki/StacklessExamples you will find the code for a modified stackless.py that implements Go's select statement. Since I am giving a talk in Toronto soon, I will soon release a new version of the join pattern version with documentation and examples. The code is about a year old and I have learnt new things. I can mail you an archive and you are free to play with it and ask questions. Since this is somewhat off-topic, the reason I mention all this is that if you want to experiment with a Go style system, I think it easiest to work from something like stackless.py and greenlets than start from scratch. Cheers, Andrew
On Oct 24, 2012, at 7:03 PM, Andrew Francis <andrewfr_ice@yahoo.com> wrote:
Hi Benoit:
From: Benoit Chesneau <benoitc@gunicorn.org> To: Andrew Francis <andrewfr_ice@yahoo.com> Cc: "python-ideas@python.org" <python-ideas@python.org> Sent: Tuesday, October 23, 2012 5:48 PM Subject: Re: [Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)
On Oct 23, 2012, at 6:51 PM, Andrew Francis <andrewfr_ice@yahoo.com> wrote:
AF>This may be tangential to the discussion but in the past, I have used the stackless.py module in conjunction with AF>CPython and greenlets to rapidly prototype parts of Go's model that are not present in Stackless, i.e. the select (ALT) AF>language feature. Rob Pike and Russ Cox were really helpful in answering my questions. Newer stackless.py AF>implementations use continuelets so look for an older PyPy implementation. AF>I have also prototyped a subset of Polyphonic C# join patterns. After I got the prototype running, I had an interesting AF>discussion with the authors of "Scalable Join Patterns."
Yes saw that. And actually some part of the Task code is based on stackless.py but using greenlets, >Channels have been slightly modified to be thread-safe and support buffering. Did you release your code >somewhere ? It could be interesting to put the experience further.
You may be mistaking my work with someone else. Oh was just saying i made this change.
I didn't add buffering but that t is relatively easy to do without altering Stackless Python's internals. However I believe that synchronous channels with buffering is a simple and powerful concurrency model. Go's implementers got it right. John Reppy (currently a NSF director) talks about synchronous channel's power in a Concurrent ML book.
If you go to to the Stackless repository example page
http://code.google.com/p/stacklessexamples/wiki/StacklessExamples
you will find the code for a modified stackless.py that implements Go's select statement.
Thanks for the link.
Since I am giving a talk in Toronto soon, I will soon release a new version of the join pattern version with documentation and examples. The code is about a year old and I have learnt new things. I can mail you an archive and you are free to play with it and ask questions.
Since this is somewhat off-topic, the reason I mention all this is that if you want to experiment with a Go style system, I think it easiest to work from something like stackless.py and greenlets than start from scratch.
Cheers, Andrew
Right. Actually flower is working well for simle purpose. My goal is more about testing new ideas about concurrency and async handling on python. Tomorrow I will push a new branch using Futures and libuv in its own thread. - benoît
participants (2)
-
Andrew Francis -
Benoit Chesneau