using asyncio in synchronous applications
I have a two-part question. If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap? Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other? Thanks, --Chris [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers
Creating and destroying event loops should be pretty cheap. I suspect the biggest cost is creation of the self-pipe. (But if you really want to know, time it first.) Multiple threads can each have their own independent event loop (accessible with get_event_loop() once created), so as long as they don't need to communicate that should be simple too. But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making? On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making?
It's the websockets library: https://github.com/aaugustin/websockets All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app. Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well. --Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.) On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com
wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application. This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**: @contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop) The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following: /usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False> It's like the cycle is slightly out of phase. In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed. I can think of three use cases for the context manager above: 1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation. This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()). --Chris On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Why do you call set_event_loop() on Python 3.6 at all? On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application.
This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**:
@contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)
The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following:
/usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
It's like the cycle is slightly out of phase.
In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed.
I can think of three use cases for the context manager above:
1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation.
This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()).
--Chris
On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com
wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org>
wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to
call
functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
On Tue, Jul 11, 2017 at 10:20 AM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Why do you call set_event_loop() on Python 3.6 at all?
Calling set_event_loop() at the end resets / sets things up for the next invocation. That was part of my point. Without it, I get the following error the next time I try to use the context manager (note that I've chosen a better name for the manager here): with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo()) with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo()) Traceback (most recent call last): ... result = loop.run_until_complete(future) File "/usr/local/lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed Remember that two of the three use cases I listed involve calling the function multiple times throughout the process's lifetime. Is there a way that doesn't require calling set_event_loop()? --Chris
On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application.
This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**:
@contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)
The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following:
/usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
It's like the cycle is slightly out of phase.
In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed.
I can think of three use cases for the context manager above:
1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation.
This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()).
--Chris
On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What calls are you making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
Hmm. After rethinking I see `set_event_loop()` is required in your design. But better to have `run(coro())` API, it could be implemented like def run(coro): loop = asyncio.new_event_loop() loop.run_until_complete(coro) loop.close() The implementation doesn't touch default loop but `asyncio.get_event_loop()` call from `coro` returns a running loop instance. On Tue, Jul 11, 2017 at 10:12 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Jul 11, 2017 at 10:20 AM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Why do you call set_event_loop() on Python 3.6 at all?
Calling set_event_loop() at the end resets / sets things up for the next invocation. That was part of my point. Without it, I get the following error the next time I try to use the context manager (note that I've chosen a better name for the manager here):
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
Traceback (most recent call last): ... result = loop.run_until_complete(future) File "/usr/local/lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed
Remember that two of the three use cases I listed involve calling the function multiple times throughout the process's lifetime.
Is there a way that doesn't require calling set_event_loop()?
--Chris
On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application.
This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**:
@contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)
The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following:
/usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
It's like the cycle is slightly out of phase.
In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed.
I can think of three use cases for the context manager above:
1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation.
This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()).
--Chris
On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage
of
async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote:
But the big question is, what is that library doing for you? In the abstract it is hard to give you a good answer. What library is it? What
calls
are you making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote: > > I have a two-part question. > > If my application is single-threaded and synchronous (e.g. a web
app
> using Gunicorn with sync workers [1]), and occasionally I need to > call > functions in a library that requires an event loop, is there any > downside to creating and closing the loop on-the-fly only when I > call > the function? In other words, is creating and destroying loops > cheap? > > Second, if I were to switch to a multi-threaded model (e.g. Gunicorn > with async workers), is my only option to start the loop at the > beginning of the process, and use loop.call_soon_threadsafe()? Or > can > I do what I was asking about above and create and close loops > on-the-fly in different threads? Is either approach much more > efficient than the other? > > Thanks, > --Chris > > [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers > _______________________________________________ > Async-sig mailing list > Async-sig@python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
On Tue, Jul 11, 2017 at 1:25 PM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Hmm. After rethinking I see `set_event_loop()` is required in your design. But better to have `run(coro())` API, it could be implemented like
def run(coro): loop = asyncio.new_event_loop() loop.run_until_complete(coro) loop.close()
Hmm. This was confusing and surprising to me that it works. For example, calling asyncio.get_event_loop() inside run() returns a different loop than when calling from inside coro. Then I remembered seeing something related to this, and found this: https://github.com/python/asyncio/pull/452 ("Make get_event_loop() return the current loop if called from coroutines/callbacks") It might be good for the main get_event_loop() docs: https://docs.python.org/3/library/asyncio-eventloops.html#asyncio.get_event_... to be updated to say that get_event_loop() returns the currently running loop and not e.g. the loop last passed to set_event_loop(), which is what the function names and current docs seem to suggest. But thank you for your pattern, Andrew. I'm glad I asked. By the way, have people settled on the best practice boilerplate for starting / cleaning up servers and loops, etc (e.g. as a gist)? It's partly what I'm trying to work out. From this issue: https://github.com/python/asyncio/pull/465 it seems like there are some subtle issues that may not have been decided, or maybe the path is clear but the sticking point is just whether it should go in the standard library. Thanks, --Chris
The implementation doesn't touch default loop but `asyncio.get_event_loop()` call from `coro` returns a running loop instance.
On Tue, Jul 11, 2017 at 10:12 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Jul 11, 2017 at 10:20 AM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Why do you call set_event_loop() on Python 3.6 at all?
Calling set_event_loop() at the end resets / sets things up for the next invocation. That was part of my point. Without it, I get the following error the next time I try to use the context manager (note that I've chosen a better name for the manager here):
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
Traceback (most recent call last): ... result = loop.run_until_complete(future) File "/usr/local/lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed
Remember that two of the three use cases I listed involve calling the function multiple times throughout the process's lifetime.
Is there a way that doesn't require calling set_event_loop()?
--Chris
On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application.
This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**:
@contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)
The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following:
/usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
It's like the cycle is slightly out of phase.
In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed.
I can think of three use cases for the context manager above:
1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation.
This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()).
--Chris
On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> wrote: > But the big question is, what is that library doing for you? In > the > abstract > it is hard to give you a good answer. What library is it? What > calls > are > you > making?
It's the websockets library: https://github.com/aaugustin/websockets
All I really need to do is occasionally connect briefly to a websocket server as a client from a synchronous app.
Since I'm already using the library on the server-side, I thought I'd save myself the trouble of having to use two libraries and just use the same library on the client side as well.
--Chris
> > On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek > <chris.jerdonek@gmail.com> > wrote: >> >> I have a two-part question. >> >> If my application is single-threaded and synchronous (e.g. a web >> app >> using Gunicorn with sync workers [1]), and occasionally I need to >> call >> functions in a library that requires an event loop, is there any >> downside to creating and closing the loop on-the-fly only when I >> call >> the function? In other words, is creating and destroying loops >> cheap? >> >> Second, if I were to switch to a multi-threaded model (e.g. >> Gunicorn >> with async workers), is my only option to start the loop at the >> beginning of the process, and use loop.call_soon_threadsafe()? Or >> can >> I do what I was asking about above and create and close loops >> on-the-fly in different threads? Is either approach much more >> efficient than the other? >> >> Thanks, >> --Chris >> >> [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers >> _______________________________________________ >> Async-sig mailing list >> Async-sig@python.org >> https://mail.python.org/mailman/listinfo/async-sig >> Code of Conduct: https://www.python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
On Tue, Jul 11, 2017 at 2:35 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Jul 11, 2017 at 1:25 PM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Hmm. After rethinking I see `set_event_loop()` is required in your design. But better to have `run(coro())` API, it could be implemented like
def run(coro): loop = asyncio.new_event_loop() loop.run_until_complete(coro) loop.close()
Hmm. This was confusing and surprising to me that it works. For example, calling asyncio.get_event_loop() inside run() returns a different loop than when calling from inside coro.
Then I remembered seeing something related to this, and found this: https://github.com/python/asyncio/pull/452 ("Make get_event_loop() return the current loop if called from coroutines/callbacks")
It might be good for the main get_event_loop() docs: https://docs.python.org/3/library/asyncio-eventloops.html#asyncio.get_event_... to be updated to say that get_event_loop() returns the currently running loop and not e.g. the loop last passed to set_event_loop(), which is what the function names and current docs seem to suggest.
For the record, I filed an issue about this here: http://bugs.python.org/issue30935 --Chris
But thank you for your pattern, Andrew. I'm glad I asked.
By the way, have people settled on the best practice boilerplate for starting / cleaning up servers and loops, etc (e.g. as a gist)? It's partly what I'm trying to work out. From this issue: https://github.com/python/asyncio/pull/465 it seems like there are some subtle issues that may not have been decided, or maybe the path is clear but the sticking point is just whether it should go in the standard library.
Thanks, --Chris
The implementation doesn't touch default loop but `asyncio.get_event_loop()` call from `coro` returns a running loop instance.
On Tue, Jul 11, 2017 at 10:12 PM Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Tue, Jul 11, 2017 at 10:20 AM, Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
Why do you call set_event_loop() on Python 3.6 at all?
Calling set_event_loop() at the end resets / sets things up for the next invocation. That was part of my point. Without it, I get the following error the next time I try to use the context manager (note that I've chosen a better name for the manager here):
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
with reset_loop_after(): loop = asyncio.get_event_loop() loop.run_until_complete(foo())
Traceback (most recent call last): ... result = loop.run_until_complete(future) File "/usr/local/lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete self._check_closed() File "/usr/local/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed
Remember that two of the three use cases I listed involve calling the function multiple times throughout the process's lifetime.
Is there a way that doesn't require calling set_event_loop()?
--Chris
On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
There's something I realized about "creating and destroying" ephemeral event loops if you want to create temporary event loops over time in a synchronous application.
This wasn't clear to me at the beginning, but it's actually more natural to do the reverse and "destroy and create," and **at the end**:
@contextmanager def run_in_loop(): try: yield finally: loop = asyncio.get_event_loop() loop.close() loop = asyncio.new_event_loop() asyncio.set_event_loop(loop)
The reason is that at the beginning of an application, the event loop starts out not closed. So if you start out by creating a new loop at the beginning, you'll get a warning like the following:
/usr/local/lib/python3.6/asyncio/base_events.py:509: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
It's like the cycle is slightly out of phase.
In contrast, if you create a new loop **at the end**, you're returning the application to the neutral state it was at the beginning, namely with a non-None loop that is neither running nor closed.
I can think of three use cases for the context manager above:
1) for wrapping the "main" function of an application, 2) for calling async functions from a synchronous app (even from different threads), which is what I was originally asking about, and 3) as part of a decorator around individual unit tests to guarantee loop isolation.
This seems like a really simple thing, but I haven't seen the pattern above written down anywhere (e.g. in past discussions of asyncio.run()).
--Chris
On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido@python.org> wrote:
OK, then as long as close the connection and the loop properly it shouldn't be a problem, even multi-threaded. (You basically lose all advantage of async, but it seems you're fine with that.)
On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote: > > On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido@python.org> > wrote: > > But the big question is, what is that library doing for you? In > > the > > abstract > > it is hard to give you a good answer. What library is it? What > > calls > > are > > you > > making? > > It's the websockets library: https://github.com/aaugustin/websockets > > All I really need to do is occasionally connect briefly to a > websocket > server as a client from a synchronous app. > > Since I'm already using the library on the server-side, I thought > I'd > save myself the trouble of having to use two libraries and just use > the same library on the client side as well. > > --Chris > > > > > > > > On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek > > <chris.jerdonek@gmail.com> > > wrote: > >> > >> I have a two-part question. > >> > >> If my application is single-threaded and synchronous (e.g. a web > >> app > >> using Gunicorn with sync workers [1]), and occasionally I need to > >> call > >> functions in a library that requires an event loop, is there any > >> downside to creating and closing the loop on-the-fly only when I > >> call > >> the function? In other words, is creating and destroying loops > >> cheap? > >> > >> Second, if I were to switch to a multi-threaded model (e.g. > >> Gunicorn > >> with async workers), is my only option to start the loop at the > >> beginning of the process, and use loop.call_soon_threadsafe()? Or > >> can > >> I do what I was asking about above and create and close loops > >> on-the-fly in different threads? Is either approach much more > >> efficient than the other? > >> > >> Thanks, > >> --Chris > >> > >> [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers > >> _______________________________________________ > >> Async-sig mailing list > >> Async-sig@python.org > >> https://mail.python.org/mailman/listinfo/async-sig > >> Code of Conduct: https://www.python.org/psf/codeofconduct/ > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- Thanks, Andrew Svetlov
-- Thanks, Andrew Svetlov
Hi Chris, I don't know your technical or efficiency constraints, however, if you want to be 100% sure that you won't have any side effects between sync and async code, you might have two daemons, one for each pattern, and use microservice approach to exchanges messages. Especially if you plan to use WebSockets, it would help you. Up to you to decide the easiest approach for you. Regards. -- Ludovic Gasc (GMLudo) Lead Developer Architect at ALLOcloud https://be.linkedin.com/in/ludovicgasc 2017-07-10 5:48 GMT+02:00 Chris Jerdonek <chris.jerdonek@gmail.com>:
I have a two-part question.
If my application is single-threaded and synchronous (e.g. a web app using Gunicorn with sync workers [1]), and occasionally I need to call functions in a library that requires an event loop, is there any downside to creating and closing the loop on-the-fly only when I call the function? In other words, is creating and destroying loops cheap?
Second, if I were to switch to a multi-threaded model (e.g. Gunicorn with async workers), is my only option to start the loop at the beginning of the process, and use loop.call_soon_threadsafe()? Or can I do what I was asking about above and create and close loops on-the-fly in different threads? Is either approach much more efficient than the other?
Thanks, --Chris
[1] http://docs.gunicorn.org/en/latest/design.html#sync-workers _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
participants (4)
-
Andrew Svetlov -
Chris Jerdonek -
Guido van Rossum -
Ludovic Gasc