I agree that is counter to the basic philosophy, though there are potential use cases where it may be unavoidable to use threads.  It seems the best solutions when it is then are to use run_in_executor, which automatically handles the waking up of the loop.  While the Queue object doesn't work with threads, though there does seem to be a library janus which provides a thread-capbable queue object which could be another solution. -----Original Message----- From: Guido van Rossum To: Brian Allen Vanderburg II Cc: Python-Ideas Sent: Thu, Feb 13, 2020 1:18 am Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe Sharing futures between threads like that goes counter to asyncio's basic philosophy (which is not to use threads :-). You already showed the solution: future._loop.call_soon_threadsafe(future.set_result, ...). If that's unacceptable for you, maybe you can wrap the future in a wrapper class that calls call_soon_threadsafe. On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via Python-ideas wrote: Currently asyncio.futures.Future.set_result will result in any callbacks being scheduled using loop.call_soon instead of loop.call_soon_threadsafe. However in situations where the future's result is set from a different thread, the loop might not wake up as a result of this if it is currently sleeping. def compute_something(...):     loop = asyncio.get_event_loop()     future = loop.create_future()     manager.run_computation(future, ...) # add the computation to a thread that is running it     return future async def compute_stuff():     result = await compute_something(...) loop = asyncio.get_event_loop() loop.run_until_complete(compute stuff()) loop.close() The reasoning behind it is after yielding the Future object, the ready list of the event loop is empty so when waiting for selectors the timeout value is None and it just waits. When the other thread calls set_result, the loop ready list will get updated, but it's still waiting on selectors.  The other thread could call future._loop.call_soon_threadsafe(future.set_result, ...), which writes some bytes to a socket in order to wake the even loop. I'm aware there are other ways to do things like this (queues, executors, etc). It just seems that, regardless of any method used, the code holding the future with the purpose of setting a result on it should be able to be blind to other implementation details.  future.set_result should be all it needs to know to call even if from another thread, or if not, a method like future.set_result_threadsafe, which would hide the details of the loop.call_soon/call_soon_threadsafe _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PLKYKF... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido)Pronouns: he/him (why is my pronoun here?)

Oops, I forgot to start the thread in the above example for compute_something(), ``th.start()`` should go right after initializing the thread but before the try-finally block On Thu, Feb 13, 2020 at 6:15 PM Kyle Stanley wrote:

...

I agree that is counter to the basic philosophy, though there are potential use cases where it may be unavoidable to use threads. It seems the best solutions when it is then are to use run_in_executor, which automatically handles the waking up of the loop.

While this doesn't provide an immediate solution, we're currently working on implementing an asyncio.ThreadPool for Python 3.9, which is made to be more self-contained (initialized and finalized within an async context manager using ``async with``) and with a higher level API compared to loop.run_in_executor(). In general, I'd recommend making use of either of those two compared to directly using threads in asyncio.

That being said, if you need more fine-grained control, there's also one other alternative low-level construct/design pattern you can use for working with threads in asyncio that we use in the implementation of some coroutine methods, such as one that I recently added to 3.9 called loop.shutdown_default_executor(). Using a similar format to your above example:

``` def _do_complete_something(fut, loop, ...): try: # perform some action, optionally get a result to set to the future loop.call_soon_threadsafe(future.set_result, None) # replace None with actual result if needed except Exception as ex: loop.call_soon_threadsafe(future.exception, ex)

async def compute_something(...): # In general, I'd recommend using get_running_loop() to get the event loop within a coroutine function. # It has more predictable behavior compared to get_event_loop(). (Python 3.7+) loop = asyncio.get_running_loop() fut = loop.create_future() th = threading.Thread(target=_do_compute_something, args=(fut,loop, ...)) try: # use "return await future" if the value is desired await future finally: th.join()

async def compute_stuff(): result = await compute_something(...)

asyncio.run(compute_stuff()) ```

Hopefully that helps a bit, or at least gives you some some ideas.

On Thu, Feb 13, 2020 at 9:31 AM Brianvanderburg2 via Python-ideas < python-ideas@python.org> wrote:

...

I agree that is counter to the basic philosophy, though there are potential use cases where it may be unavoidable to use threads. It seems the best solutions when it is then are to use run_in_executor, which automatically handles the waking up of the loop. While the Queue object doesn't work with threads, though there does seem to be a library janus which provides a thread-capbable queue object which could be another solution.

-----Original Message----- From: Guido van Rossum To: Brian Allen Vanderburg II Cc: Python-Ideas Sent: Thu, Feb 13, 2020 1:18 am Subject: Re: [Python-ideas] Asyncio Future.set_result_threadsafe

Sharing futures between threads like that goes counter to asyncio's basic philosophy (which is not to use threads :-).

You already showed the solution: future._loop.call_soon_threadsafe(future.set_result, ...). If that's unacceptable for you, maybe you can wrap the future in a wrapper class that calls call_soon_threadsafe.

On Wed, Feb 12, 2020 at 8:51 PM Brian Allen Vanderburg II via Python-ideas wrote:

Currently asyncio.futures.Future.set_result will result in any callbacks being scheduled using loop.call_soon instead of loop.call_soon_threadsafe. However in situations where the future's result is set from a different thread, the loop might not wake up as a result of this if it is currently sleeping.

def compute_something(...): loop = asyncio.get_event_loop() future = loop.create_future() manager.run_computation(future, ...) # add the computation to a thread that is running it return future

async def compute_stuff(): result = await compute_something(...)

loop = asyncio.get_event_loop() loop.run_until_complete(compute stuff()) loop.close()

The reasoning behind it is after yielding the Future object, the ready list of the event loop is empty so when waiting for selectors the timeout value is None and it just waits. When the other thread calls set_result, the loop ready list will get updated, but it's still waiting on selectors. The other thread could call future._loop.call_soon_threadsafe(future.set_result, ...), which writes some bytes to a socket in order to wake the even loop.

I'm aware there are other ways to do things like this (queues, executors, etc). It just seems that, regardless of any method used, the code holding the future with the purpose of setting a result on it should be able to be blind to other implementation details. future.set_result should be all it needs to know to call even if from another thread, or if not, a method like future.set_result_threadsafe, which would hide the details of the loop.call_soon/call_soon_threadsafe

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PLKYKF... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CWV6UQ... Code of Conduct: http://python.org/psf/codeofconduct/