Clarification on the removal of generator-based coroutines in 3.10
<DISCLAIMER>I am not a Python core developer, but my question relates to changes that are expected in Python 3.10, so I felt this was the best forum to ask. Please let me know if I should discuss this elsewhere or file a documentation bug.</DISCLAIMER> First of all, thank you Yuri Selivanov and everybody who contributed to the overhaul of the asyncio docs which are now much better organized. In the process of updating "Fluent Python" to cover Python 3.9, I learned that the "Generator-based coroutines" section in the asyncio docs warns that "Support for generator-based coroutines is deprecated and is scheduled for removal in Python 3.10." [1] [1] https://docs.python.org/3/library/asyncio-task.html#generator-based-coroutin... I don't understand that warning, given that PEP 492 lists as one of the awaitable types: * A generator-based coroutine object returned from a function decorated with types.coroutine(). Perhaps what is going to be removed is support for old style generator-based coroutines decorated with @asyncio.coroutine, or not decorated (which historically also worked)? I'd appreciate some clarification on exactly what is going to be removed. Maybe we also need updates to the Glossary [2] to bridge the gap between our previous use of the word "coroutine" (as in PEP 342) and the way we are using it today, when we are not being explicit about "native coroutines". In particular, the Glossary has no entry at all for "generator-based coroutine"—those marked with @types.coroutine. [2] https://docs.python.org/3/glossary.html Thank you for your time and help! Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Thu, Feb 18, 2021 at 3:57 PM Luciano Ramalho <luciano@ramalho.org> wrote:
Reading the doc section you link to, it's pretty clear that `@asyncio.coroutine` will be removed. The fact that it's mentioned in PEP 492 is irrelevant -- there is no rule that says we can't evolve Python to invalidate APIs specified in past PEPs, as long as the deprecation process is followed (which it is in this case).
Why would it require a glossary entry? It's right there in the first link you give as the section header. And it's an outmoded concept (which was clear from the moment 'async def' was invented). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Thanks for your reply, Guido. On Fri, Feb 19, 2021 at 12:07 AM Guido van Rossum <guido@python.org> wrote:
Reading the doc section you link to, it's pretty clear that `@asyncio.coroutine` will be removed.
The *Note* right at the top of [1] says "Support for generator-based coroutines is deprecated and is scheduled for removal in Python 3.10." [1] https://docs.python.org/3/library/asyncio-task.html#generator-based-coroutin... But PEP 492 [2] says: "Since, internally, coroutines are a special kind of generators, every await is suspended by a yield somewhere down the chain of await calls" [2] https://www.python.org/dev/peps/pep-0492/#await-expression If that part of PEP 492 is no longer accurate, then I have a couple of questions: 1) What Python construct is to be used at the end of a chain of await calls, if not of a generator-based coroutine decorated with `@types.coroutine` and using a `yield` expression in its body? 2) Given that the sole purpose of `@types.coroutine` is to decorate generator-based coroutines to become awaitable, will that decorator also be removed, along with "support for generator-based coroutines"? Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Thu, Feb 18, 2021 at 8:23 PM Luciano Ramalho <luciano@ramalho.org> wrote:
It looks like types.coroutine will remain (it does not contain code to warn about deprecation like asyncio.coroutine does), but I don't think it is required to end a chain of coroutines -- it may have been an oversight that we did not start deprecating it. (But in any case it won't be supported by asyncio.) At the end of the chain you can call the __await__() method which gives an iterator, and then you call next() or send() on that iterator. Each next()/send() call then represents an await step, and send() in general is used to provide an awaited result. Eventually this will raise StopIteration with a value indicating the ultimate result (the return value of the top-level async def). The code used to "drive" a chain of await calls is called a trampoline. But writing a trampoline is not easy, and the only example I know of is asyncio's Task class, in particular its __step() method, which of course is beyond complicated because it has to handle so many special cases. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Fri, Feb 19, 2021 at 1:59 AM Guido van Rossum <guido@python.org> wrote:
At the end of the chain you can call the __await__() method which gives an iterator, and then you call next() or send() on that iterator. Each next()/send() call then represents an await step, and send() in general is used to provide an awaited result. Eventually this will raise StopIteration with a value indicating the ultimate result (the return value of the top-level async def).
All right, that made sense to me. Thank you so much, Guido. Thanks for the clarification about `@types.coroutine` as well. Take care, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
Follow up question: what's the plan to replace this use of `@types.coroutine` in `asyncio/tasks.py`? [1] @types.coroutine def __sleep0(): """<docstring omitted>""" yield [1] https://github.com/python/cpython/blob/master/Lib/asyncio/tasks.py#L585 Best, Luciano On Fri, Feb 19, 2021 at 2:31 AM Luciano Ramalho <luciano@ramalho.org> wrote:
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum <guido@python.org> wrote:
Perhaps you meant this? async def __sleep0(): await None Either way, `await None` raises "TypeError: object NoneType can't be used in 'await' expression". Maybe I misunderstood your suggestion? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum <guido@python.org> wrote:
That didn't work*, but this does: async def __sleep(): return None Was that the idea? (*) TypeError: object NoneType can't be used in 'await' expression
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 6:29 AM Luciano Ramalho <luciano@ramalho.org> wrote:
async def __sleep(): return None
Sorry, I meant to write: async def __sleep0(): return None Since the idea is to replace the generator-based coroutine `__sleep0` in tasks.py [1] with a native coroutine. [1] https://github.com/python/cpython/blob/e92d67dfbb4790df37aa6a0961fb6dc7e8d2f... -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
I think that for this functionality (force the event loop to run), the trampoline just needs to define a function that returns a magic value and special-case that. The helper then becomes def __sleep0(): await <magic> The magic object may have to be something with an __await__() method. On Fri, Feb 19, 2021 at 01:34 Luciano Ramalho <luciano@ramalho.org> wrote:
-- --Guido (mobile)
Hello, On Fri, 19 Feb 2021 06:29:35 -0300 Luciano Ramalho <luciano@ramalho.org> wrote:
No. That doesn't return control to the event loop, like the original "yield" does. The purpose of all these shaman dances to obfuscate the relationship between generators and async functions is unclear. (Well, it's kinda clear - to allow "async generators", but ughh, don't let me start on that.) The best solution now indeed appears to allow an "await None" special construct, to have the same meaning as "yield", but only for async functions. Any other solution is dirty (and inefficient!) workaround.
-- Best regards, Paul mailto:pmiscml@gmail.com
On Thu, Feb 18, 2021 at 9:04 PM Guido van Rossum <guido@python.org> wrote:
I've found https://github.com/dabeaz/curio to be the simplest, full-featured Python event loop to read. -Brett
Thanks for Mr David beazley's materials online, a beginner like me was able to understand this thread from the beginning. Rarely do i get to understand what the folks in here are rumbling about. He was also teaching compiler theory at university. Don't know if he ever contributed to CPython. Here's a direct effect of him on me: https://www.pythonkitchen.com/python-generators-in-depth/ Though i consider Mr Paul's comments as near trolling, i was able to appreciate his perspective this time If ever there is a newbie like me on this list, i'd recommend to vaccum in as much as possible from mr Beazley ^^
On Thu, Feb 18, 2021 at 3:57 PM Luciano Ramalho <luciano@ramalho.org> wrote:
Reading the doc section you link to, it's pretty clear that `@asyncio.coroutine` will be removed. The fact that it's mentioned in PEP 492 is irrelevant -- there is no rule that says we can't evolve Python to invalidate APIs specified in past PEPs, as long as the deprecation process is followed (which it is in this case).
Why would it require a glossary entry? It's right there in the first link you give as the section header. And it's an outmoded concept (which was clear from the moment 'async def' was invented). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Thanks for your reply, Guido. On Fri, Feb 19, 2021 at 12:07 AM Guido van Rossum <guido@python.org> wrote:
Reading the doc section you link to, it's pretty clear that `@asyncio.coroutine` will be removed.
The *Note* right at the top of [1] says "Support for generator-based coroutines is deprecated and is scheduled for removal in Python 3.10." [1] https://docs.python.org/3/library/asyncio-task.html#generator-based-coroutin... But PEP 492 [2] says: "Since, internally, coroutines are a special kind of generators, every await is suspended by a yield somewhere down the chain of await calls" [2] https://www.python.org/dev/peps/pep-0492/#await-expression If that part of PEP 492 is no longer accurate, then I have a couple of questions: 1) What Python construct is to be used at the end of a chain of await calls, if not of a generator-based coroutine decorated with `@types.coroutine` and using a `yield` expression in its body? 2) Given that the sole purpose of `@types.coroutine` is to decorate generator-based coroutines to become awaitable, will that decorator also be removed, along with "support for generator-based coroutines"? Best, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Thu, Feb 18, 2021 at 8:23 PM Luciano Ramalho <luciano@ramalho.org> wrote:
It looks like types.coroutine will remain (it does not contain code to warn about deprecation like asyncio.coroutine does), but I don't think it is required to end a chain of coroutines -- it may have been an oversight that we did not start deprecating it. (But in any case it won't be supported by asyncio.) At the end of the chain you can call the __await__() method which gives an iterator, and then you call next() or send() on that iterator. Each next()/send() call then represents an await step, and send() in general is used to provide an awaited result. Eventually this will raise StopIteration with a value indicating the ultimate result (the return value of the top-level async def). The code used to "drive" a chain of await calls is called a trampoline. But writing a trampoline is not easy, and the only example I know of is asyncio's Task class, in particular its __step() method, which of course is beyond complicated because it has to handle so many special cases. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Fri, Feb 19, 2021 at 1:59 AM Guido van Rossum <guido@python.org> wrote:
At the end of the chain you can call the __await__() method which gives an iterator, and then you call next() or send() on that iterator. Each next()/send() call then represents an await step, and send() in general is used to provide an awaited result. Eventually this will raise StopIteration with a value indicating the ultimate result (the return value of the top-level async def).
All right, that made sense to me. Thank you so much, Guido. Thanks for the clarification about `@types.coroutine` as well. Take care, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
Follow up question: what's the plan to replace this use of `@types.coroutine` in `asyncio/tasks.py`? [1] @types.coroutine def __sleep0(): """<docstring omitted>""" yield [1] https://github.com/python/cpython/blob/master/Lib/asyncio/tasks.py#L585 Best, Luciano On Fri, Feb 19, 2021 at 2:31 AM Luciano Ramalho <luciano@ramalho.org> wrote:
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum <guido@python.org> wrote:
Perhaps you meant this? async def __sleep0(): await None Either way, `await None` raises "TypeError: object NoneType can't be used in 'await' expression". Maybe I misunderstood your suggestion? -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 4:08 AM Guido van Rossum <guido@python.org> wrote:
That didn't work*, but this does: async def __sleep(): return None Was that the idea? (*) TypeError: object NoneType can't be used in 'await' expression
-- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
On Fri, Feb 19, 2021 at 6:29 AM Luciano Ramalho <luciano@ramalho.org> wrote:
async def __sleep(): return None
Sorry, I meant to write: async def __sleep0(): return None Since the idea is to replace the generator-based coroutine `__sleep0` in tasks.py [1] with a native coroutine. [1] https://github.com/python/cpython/blob/e92d67dfbb4790df37aa6a0961fb6dc7e8d2f... -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Technical Principal at ThoughtWorks | Twitter: @ramalhoorg
I think that for this functionality (force the event loop to run), the trampoline just needs to define a function that returns a magic value and special-case that. The helper then becomes def __sleep0(): await <magic> The magic object may have to be something with an __await__() method. On Fri, Feb 19, 2021 at 01:34 Luciano Ramalho <luciano@ramalho.org> wrote:
-- --Guido (mobile)
Hello, On Fri, 19 Feb 2021 06:29:35 -0300 Luciano Ramalho <luciano@ramalho.org> wrote:
No. That doesn't return control to the event loop, like the original "yield" does. The purpose of all these shaman dances to obfuscate the relationship between generators and async functions is unclear. (Well, it's kinda clear - to allow "async generators", but ughh, don't let me start on that.) The best solution now indeed appears to allow an "await None" special construct, to have the same meaning as "yield", but only for async functions. Any other solution is dirty (and inefficient!) workaround.
-- Best regards, Paul mailto:pmiscml@gmail.com
On Thu, Feb 18, 2021 at 9:04 PM Guido van Rossum <guido@python.org> wrote:
I've found https://github.com/dabeaz/curio to be the simplest, full-featured Python event loop to read. -Brett
Thanks for Mr David beazley's materials online, a beginner like me was able to understand this thread from the beginning. Rarely do i get to understand what the folks in here are rumbling about. He was also teaching compiler theory at university. Don't know if he ever contributed to CPython. Here's a direct effect of him on me: https://www.pythonkitchen.com/python-generators-in-depth/ Though i consider Mr Paul's comments as near trolling, i was able to appreciate his perspective this time If ever there is a newbie like me on this list, i'd recommend to vaccum in as much as possible from mr Beazley ^^
participants (5)
-
Abdur-Rahmaan Janhangeer -
Brett Cannon -
Guido van Rossum -
Luciano Ramalho -
Paul Sokolovsky