Adding asyncio.run() function in Python 3.6
One of the remaining problems with the event loop in asyncio is bootstrapping/finalizing asyncio programs. Currently, there are two different scenarios: [1] Running a coroutine: async def main(): # your program loop = asyncio.get_event_loop() try: loop.run_until_complete(main()) finally: loop.close() [2] Running a server: loop = asyncio.get_event_loop() srv = loop.run_until_complete( loop.create_server(…)) try: loop.run_forever() finally: try: srv.close() loop.run_until_complete(srv.wait_closed()) finally: loop.close() Both cases are *incomplete*: they don’t do correct finalization of asynchronous generators. To do that we’ll need to add another 1-3 lines of code (extra try-finally). This manual approach is really painful: * It makes bootstrapping asyncio code unnecessarily hard. * It makes the documentation hard to follow. And we can’t restructure the docs to cover the loop only in the advanced section. * Most of the people will never know about `loop.shutdown_asyncgen` coroutine. * We don’t have a place to add debug code to let people know that their asyncio program didn’t clean all resources properly (a lot of unordered warnings will be spit out instead). In https://github.com/python/asyncio/pull/465 I propose to add a new function to asyncio in Python 3.6: asyncio.run(). The function can either accept a coroutine, which solves [1]: async def main(): # your program asyncio.run(main()) Or it can accept an asynchronous generator, which solves [2]: async def main(): srv = await loop.create_server(…)) try: yield # let the loop run forever finally: srv.close() await srv.wait_closed() asyncio.run(main()) asyncio.run() solves the following: * An easy way to start an asyncio program that properly takes care of loop instantiation and finalization. * It looks much better in the docs. With asyncio.run people don’t need to care about the loop at all, most probably will never use it. * Easier to experiment with asyncio in REPL. * The loop and asynchronous generators will be cleaned up properly. * We can add robust debug output to the function, listing the unclosed tasks, servers, connections, asynchronous generators etc, helping people with the cleanup logic. * Later, if we need to add more cleanup code to asyncio, we will have a function to add the logic to. I feel that we should add this to asyncio. One of the arguments against that, is that overloading asyncio.run to accept both coroutines and asynchronous generators makes the API more complex. If that’s really the case, we can add two functions: asyncio.run(coro) and asyncio.run_forever(async_generator). Also take a look at https://github.com/python/asyncio/pull/465. Thanks, Yury
What's the use case for the async generator version? Could the yield be replaced by 'await loop.shutting_down()'? On Nov 16, 2016 10:12 AM, "Yury Selivanov" <yselivanov@gmail.com> wrote:
One of the remaining problems with the event loop in asyncio is bootstrapping/finalizing asyncio programs.
Currently, there are two different scenarios:
[1] Running a coroutine:
async def main(): # your program loop = asyncio.get_event_loop() try: loop.run_until_complete(main()) finally: loop.close()
[2] Running a server:
loop = asyncio.get_event_loop() srv = loop.run_until_complete( loop.create_server(…)) try: loop.run_forever() finally: try: srv.close() loop.run_until_complete(srv.wait_closed()) finally: loop.close()
Both cases are *incomplete*: they don’t do correct finalization of asynchronous generators. To do that we’ll need to add another 1-3 lines of code (extra try-finally).
This manual approach is really painful:
* It makes bootstrapping asyncio code unnecessarily hard.
* It makes the documentation hard to follow. And we can’t restructure the docs to cover the loop only in the advanced section.
* Most of the people will never know about `loop.shutdown_asyncgen` coroutine.
* We don’t have a place to add debug code to let people know that their asyncio program didn’t clean all resources properly (a lot of unordered warnings will be spit out instead).
In https://github.com/python/asyncio/pull/465 I propose to add a new function to asyncio in Python 3.6: asyncio.run().
The function can either accept a coroutine, which solves [1]:
async def main(): # your program asyncio.run(main())
Or it can accept an asynchronous generator, which solves [2]:
async def main(): srv = await loop.create_server(…)) try: yield # let the loop run forever finally: srv.close() await srv.wait_closed()
asyncio.run(main())
asyncio.run() solves the following:
* An easy way to start an asyncio program that properly takes care of loop instantiation and finalization.
* It looks much better in the docs. With asyncio.run people don’t need to care about the loop at all, most probably will never use it.
* Easier to experiment with asyncio in REPL.
* The loop and asynchronous generators will be cleaned up properly.
* We can add robust debug output to the function, listing the unclosed tasks, servers, connections, asynchronous generators etc, helping people with the cleanup logic.
* Later, if we need to add more cleanup code to asyncio, we will have a function to add the logic to.
I feel that we should add this to asyncio. One of the arguments against that, is that overloading asyncio.run to accept both coroutines and asynchronous generators makes the API more complex. If that’s really the case, we can add two functions: asyncio.run(coro) and asyncio.run_forever(async_generator).
Also take a look at https://github.com/python/asyncio/pull/465.
Thanks, Yury _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Nov 16, 2016, at 3:35 PM, Nathaniel Smith <njs@pobox.com> wrote:
What's the use case for the async generator version? Could the yield be replaced by 'await loop.shutting_down()’?
Async generator version (inspired by contextlib.contextmanager decorator) is needed in cases where you want loop.run_forever. The PR originally proposed to add `asyncio.forever()` (which is the same idea as `loop.shutting_down()`), but nobody particularly liked it. A couple of thoughts/reasons: 1. Some pretty intrusive modifications are required to be made in the event loop to make it work. That means all other event loops (including uvloop) will have to be modified to support it. This is the most important reason. 2. `loop.shutting_down()` is a no go because it’s a method on the loop object. We can discuss `asyncio.shutting_down`. The whole point of this discussion is to get rid of the event loop. 3. `await forever()` and `await shutting_down()` have a naming issue - both look weird: async def main(): srv = await asyncio.start_server(…) try: await asyncio.shutting_down() # or await forever() finally: srv.close() await srv.wait_closed() In the above example, what does the second ‘await’ do? Will it be resolved when the loop is stopped with ‘loop.stop()’? Or when a KeyboardInterrupt occurs? What will happen if you await on it in parallel from 10 different coroutines? It’s just difficult to define a clear semantics of this coroutine. Using an asynchronous generator for this case is easier in terms of implementation and in terms of specifying the execution semantics. And the approach of using generators for such things isn’t new - we have contextlib.contextmanager decorator which is quite popular. Yury
Yury this looks great! Thanks for suggesting this. One of the big problems I've seen with the basic approachability of async/await in Python is the lack of interpreter support. Now this makes that a lot easier by having a one function you can use makes this a lot easier (One of the main things I use `curio` for is simply being able to `curio.run` an Awaitable in the interpreter without having to go through the normal asyncio dance). I would love to see if we could extend this even further to making `await` "Just Work" in the interpreter. Thanks, Roy On Wed, Nov 16, 2016 at 12:53 PM, Yury Selivanov <yselivanov@gmail.com> wrote:
On Nov 16, 2016, at 3:35 PM, Nathaniel Smith <njs@pobox.com> wrote:
What's the use case for the async generator version? Could the yield be replaced by 'await loop.shutting_down()’?
Async generator version (inspired by contextlib.contextmanager decorator) is needed in cases where you want loop.run_forever.
The PR originally proposed to add `asyncio.forever()` (which is the same idea as `loop.shutting_down()`), but nobody particularly liked it.
A couple of thoughts/reasons:
1. Some pretty intrusive modifications are required to be made in the event loop to make it work. That means all other event loops (including uvloop) will have to be modified to support it. This is the most important reason.
2. `loop.shutting_down()` is a no go because it’s a method on the loop object. We can discuss `asyncio.shutting_down`. The whole point of this discussion is to get rid of the event loop.
3. `await forever()` and `await shutting_down()` have a naming issue - both look weird:
async def main(): srv = await asyncio.start_server(…) try: await asyncio.shutting_down() # or await forever() finally: srv.close() await srv.wait_closed()
In the above example, what does the second ‘await’ do? Will it be resolved when the loop is stopped with ‘loop.stop()’? Or when a KeyboardInterrupt occurs? What will happen if you await on it in parallel from 10 different coroutines? It’s just difficult to define a clear semantics of this coroutine.
Using an asynchronous generator for this case is easier in terms of implementation and in terms of specifying the execution semantics. And the approach of using generators for such things isn’t new - we have contextlib.contextmanager decorator which is quite popular.
Yury _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
One of the big problems I've seen with the basic approachability of async/await in Python is the lack of interpreter support.
You might want to have a look at aioconsole [1]; it provides an asynchronous REPL that you can use to interact with asyncio servers. You can try it out using the apython script: $ apython Python 3.5.0 (default, Sep 7 2015, 14:12:03) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. --- This console is running in an asyncio event loop. It allows you to wait for coroutines using the 'await' syntax. Try: await asyncio.sleep(1, result=3) ---
await asyncio.sleep(1, result=3) # Wait one second... 3
There is also two ipython extensions: - ipython-yf [2] - asyncio-ipython-magic [3] [1] https://github.com/vxgmichel/aioconsole [2] https://github.com/tecki/ipython-yf [3] https://github.com/Gr1N/asyncio-ipython-magic Cheers, /Vincent
participants (4)
-
Nathaniel Smith -
Roy Williams -
Vincent Michel -
Yury Selivanov