I would like to clarify that asyncio *does* most certainly provide support for programs that use multiple event loops and interact with a number of different threads. So, asyncio can definitely be used outside of a purely "single-process, single-threaded, single-event-loop
approach". However, a key point that should be followed is that __async objects__ created in a single event loop are intended to be used only in that event loop. If you try to go outside of the bounds of that, you'll likely find yourself in the territory of undefined behavior (which is a part of why we're deprecating the loop argument throughout the high-level API, to discourage that).

The asyncio part of your program can use a thread to run a function and get the result without blocking the event loop (using loop.run_in_executor(None, func, *args) or the upcoming asyncio.to_thread() in 3.9), schedule callbacks from another thread with loop.call_soon_threadsafe(), or submit a coroutine to be run in a specific event loop with asyncio.run_coroutine_threadsafe() from a different thread. But it does require some thinking about the architecture of your program, and can't just be used to replace an existing threaded program without any considerations.

Fundamentally, OS threads and coroutines are two entirely different models of concurrency; though, we do have interoperability tools in place and are actively working on making them easier to utilize. Also, with adequate arguments for specific real-world use cases, those interoperability tools can be expanded upon as needed. However, these should be separate from existing parts of asyncio rather than changing methods like loop.run_until_complete(), IMO. There's something to be said about having too much purpose crammed into a single API member -- it tends to make the behavior needlessly complex and hard to follow.

On Tue, Jun 9, 2020 at 2:37 AM Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Fortunately, Guido has jumped in.  He's the authority on asyncio.  I
recommend you consider his advice very carefully.  Here are a couple
more commments that I had already written when I saw his post.

Celelibi writes:

 > But that doesn't mean we can't try to design asyncio to be more
 > thread-friendly.

True, lack of an example to emulate doesn't mean we can't try ... but
the fact that asyncio and threads are based on completely different
models of concurrency likely does, in combination with that lack.
Strong suggestion that "here be dragons".

 > I'm working on switching out a thread-based client protocol lib for
 > an asyncio-based lib instead.

Have you tried to do this with an alternative to asyncio such as curio
(by David Beazley, IIRC) or trio (by Nathaniel Smith)?  They provide
simpler APIs, and Nathaniel claims that they also provide a
concurrency model that's far easier to understand.  They're still
async def/await-based, but they have a property Nathaniel calls
"respect for causality" that seems very useful in understanding

 > But if it did, I'd end up calling call_soon_threadsafe from the thread
 > that runs the event loop and wait for the result. Which would just lock
 > up.

If I understand correctly, this is the kind of thing that "respect for
causality" helps to identify and to some extent prevent.

 > Another issue I found is that the client protocol lib class isn't always
 > instanciated by the same thread, and its constructor indirectly
 > instanciate asyncio objects (like Event).

The relations among the threading library, the asyncio library, and
the client protocol library in your software are quite unclear to me.
But this sounds very bad for your "divide and conquer" strategy.
Asyncio is fundamentally a single-process, single-threaded,
single-event-loop approach to handling concurrency.  If the asyncio
part of your program can ignore what other threads are doing, you
might be OK, but this kind of coupling means you can't, and will need
to do things like:

 > In the end, its start method will start the event loop in its own
 > thread.  With the deprecation of the loop argument, Event() finds
 > the loop it should use by calling get_event_loop() in its
 > constructor. I therefore had to add a call set_event_loop(), even
 > though this event loop will never run in this thread. That's a
 > pretty ugly hack.

But it's intended to be a temporary one, no?  This is the kind of
thing you need to do when you have multiple incompatible models of
computation running in the same system.

 > The deprecation of the loop arguments looks like an incentive to create
 > the asyncio objects in the threads that use them.

It is.  A major point (perhaps the whole point) of asyncio is to
provide that "single-process, single-threaded, single-event-loop
approach" which nevertheless is extremely fast and scalable for
I/O-bound programs.

If you can't get to the point where all of your interactions with the
threaded code can be done in a single thread, asyncio is very unlikely
to be a good model for your system.

 > Which seems pretty crazy to me as a whole.

You should think carefully about that feeling.  If it's just because
you're temporarily in concurrency model hell, you'll get over that
once the port to asyncio is done.  But if you really need multiple
threads because some code exposes them and you can't port that code to
asyncio, you need to consider whether asyncio can do what you need.

Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CFSVWPMXSNP6ABLGBSOVVKZMBKNB2ER3/
Code of Conduct: http://python.org/psf/codeofconduct/