I would like to clarify that asyncio *does* most certainly provide support for programs that use multiple event loops and interact with a number of different threads. So, asyncio can definitely be used outside of a purely "single-process, single-threaded, single-event-loop approach". However, a key point that should be followed is that __async objects__ created in a single event loop are intended to be used only in that event loop. If you try to go outside of the bounds of that, you'll likely find yourself in the territory of undefined behavior (which is a part of why we're deprecating the loop argument throughout the high-level API, to discourage that). The asyncio part of your program can use a thread to run a function and get the result without blocking the event loop (using loop.run_in_executor(None, func, *args) or the upcoming asyncio.to_thread() in 3.9), schedule callbacks from another thread with loop.call_soon_threadsafe(), or submit a coroutine to be run in a specific event loop with asyncio.run_coroutine_threadsafe() from a different thread. But it does require some thinking about the architecture of your program, and can't just be used to replace an existing threaded program without any considerations. Fundamentally, OS threads and coroutines are two entirely different models of concurrency; though, we do have interoperability tools in place and are actively working on making them easier to utilize. Also, with adequate arguments for specific real-world use cases, those interoperability tools can be expanded upon as needed. However, these should be separate from existing parts of asyncio rather than changing methods like loop.run_until_complete(), IMO. There's something to be said about having too much purpose crammed into a single API member -- it tends to make the behavior needlessly complex and hard to follow. On Tue, Jun 9, 2020 at 2:37 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Fortunately, Guido has jumped in. He's the authority on asyncio. I recommend you consider his advice very carefully. Here are a couple more commments that I had already written when I saw his post.
Celelibi writes:
But that doesn't mean we can't try to design asyncio to be more thread-friendly.
True, lack of an example to emulate doesn't mean we can't try ... but the fact that asyncio and threads are based on completely different models of concurrency likely does, in combination with that lack. Strong suggestion that "here be dragons".
I'm working on switching out a thread-based client protocol lib for an asyncio-based lib instead.
Have you tried to do this with an alternative to asyncio such as curio (by David Beazley, IIRC) or trio (by Nathaniel Smith)? They provide simpler APIs, and Nathaniel claims that they also provide a concurrency model that's far easier to understand. They're still async def/await-based, but they have a property Nathaniel calls "respect for causality" that seems very useful in understanding programs.
But if it did, I'd end up calling call_soon_threadsafe from the thread that runs the event loop and wait for the result. Which would just lock up.
If I understand correctly, this is the kind of thing that "respect for causality" helps to identify and to some extent prevent.
Another issue I found is that the client protocol lib class isn't always instanciated by the same thread, and its constructor indirectly instanciate asyncio objects (like Event).
The relations among the threading library, the asyncio library, and the client protocol library in your software are quite unclear to me. But this sounds very bad for your "divide and conquer" strategy. Asyncio is fundamentally a single-process, single-threaded, single-event-loop approach to handling concurrency. If the asyncio part of your program can ignore what other threads are doing, you might be OK, but this kind of coupling means you can't, and will need to do things like:
In the end, its start method will start the event loop in its own thread. With the deprecation of the loop argument, Event() finds the loop it should use by calling get_event_loop() in its constructor. I therefore had to add a call set_event_loop(), even though this event loop will never run in this thread. That's a pretty ugly hack.
But it's intended to be a temporary one, no? This is the kind of thing you need to do when you have multiple incompatible models of computation running in the same system.
The deprecation of the loop arguments looks like an incentive to create the asyncio objects in the threads that use them.
It is. A major point (perhaps the whole point) of asyncio is to provide that "single-process, single-threaded, single-event-loop approach" which nevertheless is extremely fast and scalable for I/O-bound programs.
If you can't get to the point where all of your interactions with the threaded code can be done in a single thread, asyncio is very unlikely to be a good model for your system.
Which seems pretty crazy to me as a whole.
You should think carefully about that feeling. If it's just because you're temporarily in concurrency model hell, you'll get over that once the port to asyncio is done. But if you really need multiple threads because some code exposes them and you can't port that code to asyncio, you need to consider whether asyncio can do what you need.
Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CFSVWP... Code of Conduct: http://python.org/psf/codeofconduct/