[Async-sig] using asyncio in synchronous applications

[Chris Jerdonek]

On Tue, Jul 11, 2017 at 2:35 PM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> On Tue, Jul 11, 2017 at 1:25 PM, Andrew Svetlov
> <andrew.svetlov at gmail.com> wrote:
>> Hmm. After rethinking I see `set_event_loop()` is required in your design.
>> But better to have `run(coro())` API, it could be implemented like
>>
>> def run(coro):
>>     loop = asyncio.new_event_loop()
>>     loop.run_until_complete(coro)
>>     loop.close()
>
> Hmm. This was confusing and surprising to me that it works. For
> example, calling asyncio.get_event_loop() inside run() returns a
> different loop than when calling from inside coro.
>
> Then I remembered seeing something related to this, and found this:
> https://github.com/python/asyncio/pull/452
> ("Make get_event_loop() return the current loop if called from
> coroutines/callbacks")
>
> It might be good for the main get_event_loop() docs:
> https://docs.python.org/3/library/asyncio-eventloops.html#asyncio.get_event_loop
> to be updated to say that get_event_loop() returns the currently
> running loop and not e.g. the loop last passed to set_event_loop(),
> which is what the function names and current docs seem to suggest.

For the record, I filed an issue about this here:
http://bugs.python.org/issue30935

--Chris

>
> But thank you for your pattern, Andrew. I'm glad I asked.
>
> By the way, have people settled on the best practice boilerplate for
> starting / cleaning up servers and loops, etc (e.g. as a gist)? It's
> partly what I'm trying to work out. From this issue:
> https://github.com/python/asyncio/pull/465
> it seems like there are some subtle issues that may not have been
> decided, or maybe the path is clear but the sticking point is just
> whether it should go in the standard library.
>
> Thanks,
> --Chris
>
>
>>
>> The implementation doesn't touch default loop but `asyncio.get_event_loop()`
>> call from `coro` returns a running loop instance.
>>
>>
>> On Tue, Jul 11, 2017 at 10:12 PM Chris Jerdonek <chris.jerdonek at gmail.com>
>> wrote:
>>>
>>> On Tue, Jul 11, 2017 at 10:20 AM, Andrew Svetlov
>>> <andrew.svetlov at gmail.com> wrote:
>>> > Why do you call set_event_loop() on Python 3.6 at all?
>>>
>>> Calling set_event_loop() at the end resets / sets things up for the
>>> next invocation. That was part of my point. Without it, I get the
>>> following error the next time I try to use the context manager (note
>>> that I've chosen a better name for the manager here):
>>>
>>>     with reset_loop_after():
>>>         loop = asyncio.get_event_loop()
>>>         loop.run_until_complete(foo())
>>>
>>>     with reset_loop_after():
>>>         loop = asyncio.get_event_loop()
>>>         loop.run_until_complete(foo())
>>>
>>>     Traceback (most recent call last):
>>>       ...
>>>         result = loop.run_until_complete(future)
>>>       File "/usr/local/lib/python3.6/asyncio/base_events.py", line
>>> 443, in run_until_complete
>>>         self._check_closed()
>>>       File "/usr/local/lib/python3.6/asyncio/base_events.py", line
>>> 357, in _check_closed
>>>         raise RuntimeError('Event loop is closed')
>>>     RuntimeError: Event loop is closed
>>>
>>> Remember that two of the three use cases I listed involve calling the
>>> function multiple times throughout the process's lifetime.
>>>
>>> Is there a way that doesn't require calling set_event_loop()?
>>>
>>> --Chris
>>>
>>>
>>> > On Tue, Jul 11, 2017, 17:56 Chris Jerdonek <chris.jerdonek at gmail.com>
>>> > wrote:
>>> >>
>>> >> There's something I realized about "creating and destroying" ephemeral
>>> >> event loops if you want to create temporary event loops over time in a
>>> >> synchronous application.
>>> >>
>>> >> This wasn't clear to me at the beginning, but it's actually more
>>> >> natural to do the reverse and "destroy and create," and **at the
>>> >> end**:
>>> >>
>>> >>     @contextmanager
>>> >>     def run_in_loop():
>>> >>         try:
>>> >>             yield
>>> >>         finally:
>>> >>             loop = asyncio.get_event_loop()
>>> >>             loop.close()
>>> >>             loop = asyncio.new_event_loop()
>>> >>             asyncio.set_event_loop(loop)
>>> >>
>>> >> The reason is that at the beginning of an application, the event loop
>>> >> starts out not closed. So if you start out by creating a new loop at
>>> >> the beginning, you'll get a warning like the following:
>>> >>
>>> >>   /usr/local/lib/python3.6/asyncio/base_events.py:509:
>>> >> ResourceWarning: unclosed event loop <_UnixSelectorEventLoop
>>> >> running=False closed=False debug=False>
>>> >>
>>> >> It's like the cycle is slightly out of phase.
>>> >>
>>> >> In contrast, if you create a new loop **at the end**, you're returning
>>> >> the application to the neutral state it was at the beginning, namely
>>> >> with a non-None loop that is neither running nor closed.
>>> >>
>>> >> I can think of three use cases for the context manager above:
>>> >>
>>> >> 1) for wrapping the "main" function of an application,
>>> >> 2) for calling async functions from a synchronous app (even from
>>> >> different threads), which is what I was originally asking about, and
>>> >> 3) as part of a decorator around individual unit tests to guarantee
>>> >> loop isolation.
>>> >>
>>> >> This seems like a really simple thing, but I haven't seen the pattern
>>> >> above written down anywhere (e.g. in past discussions of
>>> >> asyncio.run()).
>>> >>
>>> >> --Chris
>>> >>
>>> >>
>>> >> On Mon, Jul 10, 2017 at 7:46 AM, Guido van Rossum <guido at python.org>
>>> >> wrote:
>>> >> > OK, then as long as close the connection and the loop properly it
>>> >> > shouldn't
>>> >> > be a problem, even multi-threaded. (You basically lose all advantage
>>> >> > of
>>> >> > async, but it seems you're fine with that.)
>>> >> >
>>> >> > On Sun, Jul 9, 2017 at 9:07 PM, Chris Jerdonek
>>> >> > <chris.jerdonek at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> On Sun, Jul 9, 2017 at 9:00 PM, Guido van Rossum <guido at python.org>
>>> >> >> wrote:
>>> >> >> > But the big question is, what is that library doing for you? In
>>> >> >> > the
>>> >> >> > abstract
>>> >> >> > it is hard to give you a good answer. What library is it? What
>>> >> >> > calls
>>> >> >> > are
>>> >> >> > you
>>> >> >> > making?
>>> >> >>
>>> >> >> It's the websockets library: https://github.com/aaugustin/websockets
>>> >> >>
>>> >> >> All I really need to do is occasionally connect briefly to a
>>> >> >> websocket
>>> >> >> server as a client from a synchronous app.
>>> >> >>
>>> >> >> Since I'm already using the library on the server-side, I thought
>>> >> >> I'd
>>> >> >> save myself the trouble of having to use two libraries and just use
>>> >> >> the same library on the client side as well.
>>> >> >>
>>> >> >> --Chris
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> > On Sun, Jul 9, 2017 at 8:48 PM, Chris Jerdonek
>>> >> >> > <chris.jerdonek at gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> I have a two-part question.
>>> >> >> >>
>>> >> >> >> If my application is single-threaded and synchronous (e.g. a web
>>> >> >> >> app
>>> >> >> >> using Gunicorn with sync workers [1]), and occasionally I need to
>>> >> >> >> call
>>> >> >> >> functions in a library that requires an event loop, is there any
>>> >> >> >> downside to creating and closing the loop on-the-fly only when I
>>> >> >> >> call
>>> >> >> >> the function? In other words, is creating and destroying loops
>>> >> >> >> cheap?
>>> >> >> >>
>>> >> >> >> Second, if I were to switch to a multi-threaded model (e.g.
>>> >> >> >> Gunicorn
>>> >> >> >> with async workers), is my only option to start the loop at the
>>> >> >> >> beginning of the process, and use loop.call_soon_threadsafe()? Or
>>> >> >> >> can
>>> >> >> >> I do what I was asking about above and create and close loops
>>> >> >> >> on-the-fly in different threads? Is either approach much more
>>> >> >> >> efficient than the other?
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> --Chris
>>> >> >> >>
>>> >> >> >> [1] http://docs.gunicorn.org/en/latest/design.html#sync-workers
>>> >> >> >> _______________________________________________
>>> >> >> >> Async-sig mailing list
>>> >> >> >> Async-sig at python.org
>>> >> >> >> https://mail.python.org/mailman/listinfo/async-sig
>>> >> >> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > --Guido van Rossum (python.org/~guido)
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > --Guido van Rossum (python.org/~guido)
>>> >> _______________________________________________
>>> >> Async-sig mailing list
>>> >> Async-sig at python.org
>>> >> https://mail.python.org/mailman/listinfo/async-sig
>>> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>> >
>>> > --
>>> > Thanks,
>>> > Andrew Svetlov
>>
>> --
>> Thanks,
>> Andrew Svetlov