gevent-like Coroutines in Python

One of Golang's advantages is that goroutines act like gevent's coroutines instead of relying on an async/await syntax. In my opinion, it makes Golang code much more readable. I feel like having the await syntax trigger by default on any awaitable invocation in a coroutine context makes much more sense. I consider async/await syntax to be too complicated for the average developer since it opens up too many abilities whereas most developers would always mean they prefer to do this: result = [await fun(x) for fun in funcs] versus: result = [fun(x) for fun in funcs] await asyncio.gather(*result) Moreso, having it become the default makes statements like this: result = [await fun(x) for fun in funcs if await smth] Look like this: result = [fun(x) for fun in funcs if smth] Therefore, my suggestion is to create a new "async" definition which basically turns every function invocation into an "await" if a generator is returned. Instead of "async def" I propose the alternative "coroutine def" syntax. However, a better solution may be to imply the word "async" in every function definition given some sort of trigger (however I assume that this won't be the preferable approach as it is not something that can be implemented at the parsing level). For me, I believe that Python should aspire to be both concise and straightforward, which means no boilerplate code just because it's more "correct" but rather assume there's some logical default behavior going on in the back - and to me this logical behavior is that every invocation can trigger an await and go back to the main loop. Our assumption that a function invocation has to tie back to instant execution unless an "await" statement has been placed should be challenged in favor of readability, conciseness, and to always aim to appeal to novice developers (which I believe is the reason Python is on such a rise these days). I do have to admit I have not thought through what is the best syntax or the implications of this because I would like to see what is the general opinion on this idea first. - Ron

Ron Reiter wrote:
I feel like having the await syntax trigger by default on any awaitable invocation in a coroutine context makes much more sense.
Guido seems to regard the requirement to use 'await' as a feature, not a bug. He says he likes to be able to see where all the potential suspension points are.
I don't think it's feasible to automatically infer both 'async' *and* 'await', even at run time. An async function needs to be treated differently from the moment it starts running, so there must be something that statically identifies it as such. Anyway, if you want to pursue these ideas further, you should take a look at PEP 3152, which was my attempt at a nicer syntax for generator-based coroutines, before async/await came along. I think it would have been better in some ways, but it was rejected. -- Greg

On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter <ron.reiter@gmail.com> wrote:
I'm not sure what you're driving at here. From your first example, I gather that (pun intended) you're expecting the 'result' list to contain all the results from the different function calls, running them all in parallel; but your second example and described suggestion seem to imply that the waitings would continue to be sequential. Unless you're asking for straight-up magic ("do everything in parallel unless they need to be serialized"), there still needs to be a clear way to differentiate between "wait for this right now and give me a result before this function continues" and "gather all these jobs together, get me the results, and then move on once you have them all". It might perhaps be nice to have an easier/more obvious syntax for gather(), but it definitely needs to have some form of spelling. If you're not asking for them to be run in parallel, you're asking for an implicit way for a function call to block its caller, and for the calling function to act sequentially. Python already has that - it's called threading :) ChrisA

You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads. As I said - I don't think we should downgrade Python's current ability to do so, my suggestion is to create something like the "codef" proposal, which will also await on every function invocation - for readability. - Ron [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter] <http://twitter.com/#!/ronreiter> [image: LinkedIn] <https://il.linkedin.com/in/ronreiter> On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico <rosuav@gmail.com> wrote:

On Tue, Oct 30, 2018 at 11:36 PM Ron Reiter <ron.reiter@gmail.com> wrote:
You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads.
Okay, so it's actually nothing to do with asyncio.gather(). Sure. So what you're looking for is JUST the removal of the "await" keywords. As Greg already said, Guido considers the explicit await markers as a feature, not a bug; these are the exact points where an intrathread context switch can occur. As to the efficiency of parallelizing on an event loop rather than using threads, that's a tradeoff; threads aren't going anywhere just because asyncio is here. When you want the extreme simplicity of "just do this stuff, okay?", the easiest way to get it is to just use threads, and pay a bit of overhead. You'll often find that the overhead isn't actually all that significant until you get to extremes of throughput - most Python apps are not trying to run tens of thousands of concurrent TCP sockets, for instance. ChrisA

Python's coroutines are designed to make suspension points visible, which enhances "local reasoning" about code. This concept has been written up very well over here: https://glyph.twistedmatrix.com/2014/02/unyielding.html On Tue, Oct 30, 2018 at 8:37 AM Ron Reiter <ron.reiter@gmail.com> wrote:

Ron Reiter wrote:
I feel like having the await syntax trigger by default on any awaitable invocation in a coroutine context makes much more sense.
Guido seems to regard the requirement to use 'await' as a feature, not a bug. He says he likes to be able to see where all the potential suspension points are.
I don't think it's feasible to automatically infer both 'async' *and* 'await', even at run time. An async function needs to be treated differently from the moment it starts running, so there must be something that statically identifies it as such. Anyway, if you want to pursue these ideas further, you should take a look at PEP 3152, which was my attempt at a nicer syntax for generator-based coroutines, before async/await came along. I think it would have been better in some ways, but it was rejected. -- Greg

On Tue, Oct 30, 2018 at 6:01 PM Ron Reiter <ron.reiter@gmail.com> wrote:
I'm not sure what you're driving at here. From your first example, I gather that (pun intended) you're expecting the 'result' list to contain all the results from the different function calls, running them all in parallel; but your second example and described suggestion seem to imply that the waitings would continue to be sequential. Unless you're asking for straight-up magic ("do everything in parallel unless they need to be serialized"), there still needs to be a clear way to differentiate between "wait for this right now and give me a result before this function continues" and "gather all these jobs together, get me the results, and then move on once you have them all". It might perhaps be nice to have an easier/more obvious syntax for gather(), but it definitely needs to have some form of spelling. If you're not asking for them to be run in parallel, you're asking for an implicit way for a function call to block its caller, and for the calling function to act sequentially. Python already has that - it's called threading :) ChrisA

You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads. As I said - I don't think we should downgrade Python's current ability to do so, my suggestion is to create something like the "codef" proposal, which will also await on every function invocation - for readability. - Ron [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter] <http://twitter.com/#!/ronreiter> [image: LinkedIn] <https://il.linkedin.com/in/ronreiter> On Tue, Oct 30, 2018 at 12:41 PM Chris Angelico <rosuav@gmail.com> wrote:

On Tue, Oct 30, 2018 at 11:36 PM Ron Reiter <ron.reiter@gmail.com> wrote:
You are right that they are different, I was actually assuming that developers by default don't try to parallelize and would rather go ahead and write code to yield one function at a time, which is fine. The need to separate "await" from the invocation is something which is rarely used. Not sure what you mean about "threading" as it is still more efficient and lightweight to parallelize workers on an event loop rather than using blocking threads.
Okay, so it's actually nothing to do with asyncio.gather(). Sure. So what you're looking for is JUST the removal of the "await" keywords. As Greg already said, Guido considers the explicit await markers as a feature, not a bug; these are the exact points where an intrathread context switch can occur. As to the efficiency of parallelizing on an event loop rather than using threads, that's a tradeoff; threads aren't going anywhere just because asyncio is here. When you want the extreme simplicity of "just do this stuff, okay?", the easiest way to get it is to just use threads, and pay a bit of overhead. You'll often find that the overhead isn't actually all that significant until you get to extremes of throughput - most Python apps are not trying to run tens of thousands of concurrent TCP sockets, for instance. ChrisA

Python's coroutines are designed to make suspension points visible, which enhances "local reasoning" about code. This concept has been written up very well over here: https://glyph.twistedmatrix.com/2014/02/unyielding.html On Tue, Oct 30, 2018 at 8:37 AM Ron Reiter <ron.reiter@gmail.com> wrote:
participants (4)
-
Chris Angelico
-
Greg Ewing
-
Mark E. Haase
-
Ron Reiter