Mailman 3 A send() built-in function to drive coroutines - Python-ideas

A send() built-in function to drive coroutines

older
Re: [Python-ideas] A way out of...

Luciano Ramalho

16 Feb 2015 16 Feb '15

5:53 a.m.

Coroutines are useless until primed by a next(my_coro) call. There are decorators to solve this by returning a primed coroutine. Such decorators add another problem: now the user may be unsure whether a specific coroutine from a third-party API should be primed or not prior to use. How about having a built-in function named send() with the signature send(coroutine, value) that does the right thing: sends a value to the coroutine, priming it if necessary. So instead of writing this:

...

...
...
next(my_coro) my_coro.send(a_value)

You could always write this:

...

...
...
send(my_coro, a_value)

At a high level, the behavior of send() would be like this: def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value) Consider the above pseudo-code. Proper handling of other generator states should be added. This would make coroutines easier to use, and would make their basic usage more consistent with the usage of plain generators using a function syntax -- eg. next(my_gen) -- instead of method call syntax. What do you think? If this has been discussed before, please send me a link (I searched but came up empty). Best, Luciano -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Show replies by thread

Luciano Ramalho

16 Feb 16 Feb

5:59 a.m.

On Mon, Feb 16, 2015 at 10:53 AM, Luciano Ramalho wrote:

...

At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

In the pseudo-code above I forgot a return statement in the last line. It should read: def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) return coroutine.send(value) Best, Luciano -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Luciano Ramalho

11:03 a.m.

My previous pseudo-code lost the result of next(coroutine) -- which may or may not be relevant. I amended the send() code and wrote a few examples of it's use in a gist: https://gist.github.com/ramalho/c1f7df10308a4bd67198 Now the user can decide whether or not to ignore the result of the priming next() call. Best, Luciano On Mon, Feb 16, 2015 at 10:59 AM, Luciano Ramalho wrote:

...

On Mon, Feb 16, 2015 at 10:53 AM, Luciano Ramalho wrote:

...
At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

In the pseudo-code above I forgot a return statement in the last line. It should read:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) return coroutine.send(value)

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr

-- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Guido van Rossum

3:30 p.m.

This is brought up from time to time. I believe it is based on a misunderstanding of coroutines. On Mon, Feb 16, 2015 at 4:53 AM, Luciano Ramalho wrote:

...

Coroutines are useless until primed by a next(my_coro) call. There are decorators to solve this by returning a primed coroutine. Such decorators add another problem: now the user may be unsure whether a specific coroutine from a third-party API should be primed or not prior to use.

How about having a built-in function named send() with the signature send(coroutine, value) that does the right thing: sends a value to the coroutine, priming it if necessary. So instead of writing this:

...
...
...
next(my_coro) my_coro.send(a_value)

You could always write this:

...
...
...
send(my_coro, a_value)

At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

Consider the above pseudo-code. Proper handling of other generator states should be added.

This would make coroutines easier to use, and would make their basic usage more consistent with the usage of plain generators using a function syntax -- eg. next(my_gen) -- instead of method call syntax.

What do you think? If this has been discussed before, please send me a link (I searched but came up empty).

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

Luciano Ramalho

11:47 p.m.

On Mon, Feb 16, 2015 at 8:30 PM, Guido van Rossum wrote:

...

This is brought up from time to time. I believe it is based on a misunderstanding of coroutines.

Thanks, Guido, if you could elaborate that would be great. Meanwhile I'll try to guess what you may be referring to, please correct me if I am wrong: the misunderstanding has to do with the fact that between the priming next() and the first send(), the client code should be doing something interesting. If that is not the case, then the client code and the coroutine are not cooperating fully. Is that it? Best, Luciano

...

On Mon, Feb 16, 2015 at 4:53 AM, Luciano Ramalho wrote:

...
Coroutines are useless until primed by a next(my_coro) call. There are decorators to solve this by returning a primed coroutine. Such decorators add another problem: now the user may be unsure whether a specific coroutine from a third-party API should be primed or not prior to use.

How about having a built-in function named send() with the signature send(coroutine, value) that does the right thing: sends a value to the coroutine, priming it if necessary. So instead of writing this:

...
...
...
next(my_coro) my_coro.send(a_value)

You could always write this:

...
...
...
send(my_coro, a_value)

At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

Consider the above pseudo-code. Proper handling of other generator states should be added.

This would make coroutines easier to use, and would make their basic usage more consistent with the usage of plain generators using a function syntax -- eg. next(my_gen) -- instead of method call syntax.

What do you think? If this has been discussed before, please send me a link (I searched but came up empty).

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

-- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Victor Stinner

23 Feb 23 Feb

3:49 a.m.

2015-02-16 13:53 GMT+01:00 Luciano Ramalho :

...

At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

It's strange to have to sometimes run one iterations of the generator, sometimes two iterations. asyncio.Task is a nice wrapper on top of coroutines, you never use coro.send() explicitly. It makes coroutines easier to use. Victor

Luciano Ramalho

4:23 a.m.

On Mon, Feb 23, 2015 at 7:49 AM, Victor Stinner wrote:

...

2015-02-16 13:53 GMT+01:00 Luciano Ramalho :

...
At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

It's strange to have to sometimes run one iterations of the generator, sometimes two iterations.

The idea is to handle generators that may or may not be primed. I updated that snippet, it now reads like this: https://gist.github.com/ramalho/c1f7df10308a4bd67198#file-send_builtin-py-L4...

...

asyncio.Task is a nice wrapper on top of coroutines, you never use coro.send() explicitly. It makes coroutines easier to use.

Yes it is, thanks! My intent was to build something that was not tied to the asyncio event loop, to make coroutines in general easier to use. Thanks for your response, Victor. Best, Luciano -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Victor Stinner

6:12 a.m.

The use case for your send() function is unclear to me. Why not using send() directly? inspect.getgeneratorstate(generator) == 'GEN_CREATED' test looks weird (why do you need it?). You should already know if the generator started or not. Victor 2015-02-23 12:23 GMT+01:00 Luciano Ramalho :

...

On Mon, Feb 23, 2015 at 7:49 AM, Victor Stinner wrote:

...
2015-02-16 13:53 GMT+01:00 Luciano Ramalho :

...
At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

It's strange to have to sometimes run one iterations of the generator, sometimes two iterations.

The idea is to handle generators that may or may not be primed. I updated that snippet, it now reads like this:

https://gist.github.com/ramalho/c1f7df10308a4bd67198#file-send_builtin-py-L4...

...
asyncio.Task is a nice wrapper on top of coroutines, you never use coro.send() explicitly. It makes coroutines easier to use.

Yes it is, thanks! My intent was to build something that was not tied to the asyncio event loop, to make coroutines in general easier to use.

Thanks for your response, Victor.

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr

Luciano Ramalho

6:22 a.m.

On Mon, Feb 23, 2015 at 10:12 AM, Victor Stinner wrote:

...

The use case for your send() function is unclear to me. Why not using send() directly?

Bacause the generator may not have started yet, so gen.send() would fail.

...

inspect.getgeneratorstate(generator) == 'GEN_CREATED' test looks weird (why do you need it?).

To know that the generator must be primed.

...

You should already know if the generator started or not.

I would, if I had written the generator myself. But my code may need to work with a generator that was created by some code that I do not control. Thanks for your interest, Victor. Best, Luciano

...

Victor

2015-02-23 12:23 GMT+01:00 Luciano Ramalho :

...
On Mon, Feb 23, 2015 at 7:49 AM, Victor Stinner wrote:

...
2015-02-16 13:53 GMT+01:00 Luciano Ramalho :

...
At a high level, the behavior of send() would be like this:

def send(coroutine, value): if inspect.getgeneratorstate() == 'GEN_CREATED': next(coroutine) coroutine.send(value)

It's strange to have to sometimes run one iterations of the generator, sometimes two iterations.

The idea is to handle generators that may or may not be primed. I updated that snippet, it now reads like this:

https://gist.github.com/ramalho/c1f7df10308a4bd67198#file-send_builtin-py-L4...

...
asyncio.Task is a nice wrapper on top of coroutines, you never use coro.send() explicitly. It makes coroutines easier to use.

Yes it is, thanks! My intent was to build something that was not tied to the asyncio event loop, to make coroutines in general easier to use.

Thanks for your response, Victor.

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr

-- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Ron Adam

8:55 a.m.

On 02/23/2015 08:22 AM, Luciano Ramalho wrote:

...

On Mon, Feb 23, 2015 at 10:12 AM, Victor Stinner wrote:

...
...
The use case for your send() function is unclear to me. Why not using send() directly? Bacause the generator may not have started yet, so gen.send() would fail.

...
...
inspect.getgeneratorstate(generator) == 'GEN_CREATED' test looks weird (why do you need it?). To know that the generator must be primed.

...
...
You should already know if the generator started or not.

...

I would, if I had written the generator myself. But my code may need to work with a generator that was created by some code that I do not control.

When using coroutines, they are generally run by a co-routine framework. That is how they become "CO"-routines rather than just generators that you can send values to. Co-routines generally use the yield to pause/continue and to communicate to the framework, and not to get and receive data. In most case I've seen co-routines look and work very much like functions except they have yield put in places so they cooperate with the frame work that is running them. That frame work handles the starting co-routine. I think what you are referring to is generators that are both providers and consumers, and not co-routines. Cheers, Ron

Luciano Ramalho

12:10 p.m.

On Mon, Feb 23, 2015 at 12:55 PM, Ron Adam wrote:

...

When using coroutines, they are generally run by a co-routine framework. That is how they become "CO"-routines rather than just generators that you can send values to. Co-routines generally use the yield to pause/continue and to communicate to the framework, and not to get and receive data. In most case I've seen co-routines look and work very much like functions except they have yield put in places so they cooperate with the frame work that is running them. That frame work handles the starting co-routine.

I think what you are referring to is generators that are both providers and consumers, and not co-routines.

Thanks, Ron, this was very helpful, and makes perfect sense to me. The framework also handles the other feature that distinguishes coroutines from consuming generators: returning values. In a sense, coroutines are not first-class citizens in Python, only generators are. Coroutines need some supporting framework to be useful. That's partly what I was trying to address with my `send()` function. I now see the issue is much deeper than I previously thought. Thanks for the discussion, folks! Best, Luciano -- Luciano Ramalho Twitter: @ramalhoorg Professor em: http://python.pro.br Twitter: @pythonprobr

Andrew Barnert

2:27 p.m.

On Feb 23, 2015, at 11:10, Luciano Ramalho wrote:

...

On Mon, Feb 23, 2015 at 12:55 PM, Ron Adam wrote:

...
When using coroutines, they are generally run by a co-routine framework. That is how they become "CO"-routines rather than just generators that you can send values to. Co-routines generally use the yield to pause/continue and to communicate to the framework, and not to get and receive data. In most case I've seen co-routines look and work very much like functions except they have yield put in places so they cooperate with the frame work that is running them. That frame work handles the starting co-routine.

I think what you are referring to is generators that are both providers and consumers, and not co-routines.

Thanks, Ron, this was very helpful, and makes perfect sense to me.

The framework also handles the other feature that distinguishes coroutines from consuming generators: returning values.

In a sense, coroutines are not first-class citizens in Python, only generators are. Coroutines need some supporting framework to be useful.

No, Python has first-class coroutines. And coroutines don't need a supporting framework to be useful. For example, a consumer can kick off a producer and the two of them can yield back and forth with no outside help. Or you can compose them into a pipeline, or build a manual state machine, or do all the other usual coroutine things with them. But coroutines _do_ need some supporting framework to be useful as general semi-implicit cooperative threads (a la greenlet servers, classic Mac/Windows/wx GUI apps built around WaitNextEvent, non-parallel actor-model designs, etc.). If each coroutine has no idea who's supposed to run next, you need a scheduler framework to keep track of that. But because Python has real coroutines, that framework can be written in pure Python. As with asyncio.

...

That's partly what I was trying to address with my `send()` function. I now see the issue is much deeper than I previously thought.

Thanks for the discussion, folks!

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Andrew Barnert

2:40 p.m.

On Feb 23, 2015, at 13:27, Andrew Barnert wrote:

...

On Feb 23, 2015, at 11:10, Luciano Ramalho wrote:

...
On Mon, Feb 23, 2015 at 12:55 PM, Ron Adam wrote:

...
When using coroutines, they are generally run by a co-routine framework. That is how they become "CO"-routines rather than just generators that you can send values to. Co-routines generally use the yield to pause/continue and to communicate to the framework, and not to get and receive data. In most case I've seen co-routines look and work very much like functions except they have yield put in places so they cooperate with the frame work that is running them. That frame work handles the starting co-routine.

I think what you are referring to is generators that are both providers and consumers, and not co-routines.

Thanks, Ron, this was very helpful, and makes perfect sense to me.

The framework also handles the other feature that distinguishes coroutines from consuming generators: returning values.

In a sense, coroutines are not first-class citizens in Python, only generators are. Coroutines need some supporting framework to be useful.

No, Python has first-class coroutines.

I just realized that you _could_ mean something different here, which would be more accurate: In a few languages, either the execution stack (e.g., most Smalltalks--and most assembly languages, I guess) or continuations (e.g., Scheme, SML) are first-class objects, which means first-class coroutines can be written purely within the language. Which means you don't have to write fully-general coroutines, you can use different limited implementations that better fit your current application (e.g., embedding a scheduling discipline under the covers that acts like a trampoline but is invisible in the app code). Python doesn't have that. But then I'm not sure how your send would be relevant to that meaning, so I still think you mean something like what I said before, based on a misunderstanding of what first-class coroutines can do.

...

And coroutines don't need a supporting framework to be useful. For example, a consumer can kick off a producer and the two of them can yield back and forth with no outside help. Or you can compose them into a pipeline, or build a manual state machine, or do all the other usual coroutine things with them.

But coroutines _do_ need some supporting framework to be useful as general semi-implicit cooperative threads (a la greenlet servers, classic Mac/Windows/wx GUI apps built around WaitNextEvent, non-parallel actor-model designs, etc.). If each coroutine has no idea who's supposed to run next, you need a scheduler framework to keep track of that.

But because Python has real coroutines, that framework can be written in pure Python. As with asyncio.

...
That's partly what I was trying to address with my `send()` function. I now see the issue is much deeper than I previously thought.

Thanks for the discussion, folks!

Best,

Luciano

-- Luciano Ramalho Twitter: @ramalhoorg

Professor em: http://python.pro.br Twitter: @pythonprobr _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

24 Feb 24 Feb

4 a.m.

On Mon, Feb 23, 2015 at 10:22:05AM -0300, Luciano Ramalho wrote:

...

On Mon, Feb 23, 2015 at 10:12 AM, Victor Stinner wrote:

...
The use case for your send() function is unclear to me. Why not using send() directly?

Bacause the generator may not have started yet, so gen.send() would fail.

...
inspect.getgeneratorstate(generator) == 'GEN_CREATED' test looks weird (why do you need it?).

To know that the generator must be primed.

...
You should already know if the generator started or not.

I would, if I had written the generator myself. But my code may need to work with a generator that was created by some code that I do not control.

I must admit I don't understand the problem here. Here's a simple coroutine: def runningsum(): total = 0 x = (yield None) while True: total += x x = (yield total) If I have a function which expects a *primed* coroutine, and you provide an unprimed one, we get an obvious error: py> def handle_primed(co): ... for i in (1, 3, 5): ... y = co.send(i) ... return y ... py> handle_primed(runningsum()) # Oops, unprimed! Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in handle_primed TypeError: can't send non-None value to a just-started generator And if I have a function which expects an *unprimed* coroutine, and you provide a primed one, we will usually get an exception: py> def handle_unprimed(co): ... next(co) ... for i in (1, 3, 5): ... y = co.send(i) ... return y ... py> rs = runningsum() py> next(rs) # Oops, primed! py> handle_unprimed(rs) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in handle_unprimed File "<stdin>", line 5, in runningsum TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType' I say "usually" because I suppose there are some coroutines which might accept being sent None and fail to cause an error. But they should be pretty rare, and in this case I think "consenting adults" should apply: if I write such a function, I can always inspect the coroutine myself. So it seems to me that this is a case for documentation, not a built-in function. Document what you expect, and then it is up to the caller to provide what you ask for. Have I missed something? Can you give an example of when you would legitimately want to accept coroutines regardless of whether they are primed or not, but don't want to explicitly inspect the coroutine yourself? -- Steve

Anthony Towns

5:56 a.m.

On 24 February 2015 at 21:00, Steven D'Aprano wrote:

...

If I have a function which expects a *primed* coroutine, and you provide an unprimed one, we get an obvious error:

py> def handle_primed(co): ... for i in (1, 3, 5): ... y = co.send(i) ... return y ... py> handle_primed(runningsum()) # Oops, unprimed! Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in handle_primed TypeError: can't send non-None value to a just-started generator

Why would you ever want to use that function "unprimed", though? Writing "next(rs)" or "rs.send(None)" just seems like busywork to me; it's not something I'd write if I were writing pseudocode, so it's something I wish python didn't force me to write in order to use that language feature... But... you could just use a decorator to make that go away, I think? As in having a decorator along the lines of: def primed_coroutine(f): @wraps(f) def fprime(*args, **kwargs): co = f(*args, **kwargs) next(co) return co return fprime so that you could write and use a coroutine just by having: @primed_coroutine def runningsum(): total = 0 while True: total += (yield total) handle_primed(runningsum()) At least, now that I've thought of it, that's how I plan to write any generator-based coroutines in future... Cheers, aj

Steven D'Aprano

8:54 a.m.

On Tue, Feb 24, 2015 at 10:56:49PM +1000, Anthony Towns wrote:

...

On 24 February 2015 at 21:00, Steven D'Aprano wrote:

...
If I have a function which expects a *primed* coroutine, and you provide an unprimed one, we get an obvious error: [...] Why would you ever want to use that function "unprimed", though?

I wouldn't. If I wrote a function that expected a coroutine as argument, I would always expect that the caller would prime the coroutine before sending it to me. If they forget, they will get an exception. But Luciano seems to wish to accept both primed and unprimed coroutines, so I'm just following his requirements as I understand them. [...]

...

But... you could just use a decorator to make that go away, I think? As in having a decorator along the lines of:

def primed_coroutine(f): @wraps(f) def fprime(*args, **kwargs): co = f(*args, **kwargs) next(co) return co return fprime

Indeed. That is exactly what I do in my code, so all my coroutines are primed as soon as they are created and I never have to worry about them being unprimed. My code is virtually the same as yours, except I call the decorator "coroutine". @coroutine def running_sum(): ... makes it clear that running_sum is a coroutine, not a regular function or generator. Although the decorator is quite simple, I think that *every* user of coroutines should be using it. Are there any use-cases for unprimed coroutines? Perhaps this decorator should become a built-in, or at least in functools? -- Steve

Steve Dower

9:25 a.m.

I believe "yield from" is the case where you need it unprimed. We went through a few rounds of this as asyncio was being prototyped, and ended up without priming because of yield from. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Steven D'Apranomailto:steve@pearwood.info Sent: ‎2/‎24/‎2015 7:55 To: python-ideas@python.orgmailto:python-ideas@python.org Subject: Re: [Python-ideas] A send() built-in function to drive coroutines On Tue, Feb 24, 2015 at 10:56:49PM +1000, Anthony Towns wrote:

...

On 24 February 2015 at 21:00, Steven D'Aprano wrote:

...
If I have a function which expects a *primed* coroutine, and you provide an unprimed one, we get an obvious error: [...] Why would you ever want to use that function "unprimed", though?

...

But... you could just use a decorator to make that go away, I think? As in having a decorator along the lines of:

def primed_coroutine(f): @wraps(f) def fprime(*args, **kwargs): co = f(*args, **kwargs) next(co) return co return fprime

3348

Age (days ago)

3356

Last active (days ago)

List overview

Download

16 comments

8 participants

participants (8)

Andrew Barnert
Anthony Towns
Guido van Rossum
Luciano Ramalho
Ron Adam
Steve Dower
Steven D'Aprano
Victor Stinner

A send() built-in function to drive coroutines

Anthony Towns

tags

participants (8)