Hi Vincent,

I've read your write-up with interest. You're right that it's a bit awkward to make calls from the threaded world into the asyncio world. Interestingly, there's much better support for passing work off from the asyncio event loop to a thread (run_in_executor()). Perhaps that's because the use case there was obvious from the start: some things that may block for I/O just don't have an async interface yet, so in order to use them from an asyncio task they must be off-loaded to a separate thread or else the entire event loop is blocked. (This is used for calling getaddrinfo(), for example.)

I'm curious where you have encountered the opposite use case?

I think if I had to do this myself I would go for a more minimalist interface: something like your submit() method but without the call to asyncio.coroutine(fn). Having the caller pass in the already-called coroutine object might simplify the signature even further. I'm not sure I see the advantage of trying to make this an executor -- but perhaps I'm missing something?

--Guido

On Sat, Sep 26, 2015 at 7:29 AM, Vincent Michel <vxgmichel@gmail.com> wrote:

Hi,

I noticed there is currently no standard solution to submit a job from a thread to an asyncio event loop.

Here's what the asyncio documentation says about concurrency and multithreading:

> To schedule a callback from a different thread, the BaseEventLoop.call_soon_threadsafe() method should be used.
> Example to schedule a coroutine from a different thread:
>     loop.call_soon_threadsafe(asyncio.async, coro_func())

The issue with this method is the loss of the coroutine result.

One way to deal with this issue is to connect the asyncio.Future returned by async (or ensure_future) to a concurrent.futures.Future. It is then possible to use a subclass of concurrent.futures.Executor to submit a callback to an asyncio event loop. Such an executor can also be used to set up communication between two event loops using run_in_executor.

I posted an implementation called LoopExecutor on GitHub:
https://github.com/vxgmichel/asyncio-loopexecutor
The repo contains the loopexecutor module along with tests for several use cases. The README describes the whole thing (context, examples, issues, implementation).

It is interesting to note that this executor is a bit different than ThreadPoolExecutor and ProcessPoolExecutor since it can also submit a coroutine function. Example:

with LoopExecutor(loop) as executor:
    future = executor.submit(operator.add, 1, 2)
    assert future.result() == 3
    future = executor.submit(asyncio.sleep, 0.1, result=3)
    assert future.result() == 3

This works in both cases because submit always cast the given function to a coroutine. That means it would also work with a function that returns a Future.

Here's a few topic related to the current implementation that might be interesting to discuss:

- possible drawback of casting the callback to a coroutine
- possible drawback of concurrent.future.Future using asyncio.Future._copy_state
- does LoopExecutor need to implement the shutdown method?
- removing the limitation in run_in_executor (can't submit a coroutine function)
- adding a generic Future connection function in asyncio
- reimplementing wrap_future with the generic connection
- adding LoopExecutor to asyncio (or concurrent.futures)

At the moment, the interaction between asyncio and concurrent.futures only goes one way. It would be nice to have a standard solution (LoopExecutor or something else) to make it bidirectional.

Thanks,

Vincent

_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

--Guido van Rossum (python.org/~guido)