[Python-ideas] Submitting a job to an asyncio event loop

[Guido van Rossum]

Hi Vincent,

I've read your write-up with interest. You're right that it's a bit awkward
to make calls from the threaded world into the asyncio world.
Interestingly, there's much better support for passing work off from the
asyncio event loop to a thread (run_in_executor()). Perhaps that's because
the use case there was obvious from the start: some things that may block
for I/O just don't have an async interface yet, so in order to use them
from an asyncio task they must be off-loaded to a separate thread or else
the entire event loop is blocked. (This is used for calling getaddrinfo(),
for example.)

I'm curious where you have encountered the opposite use case?

I think if I had to do this myself I would go for a more minimalist
interface: something like your submit() method but without the call to
asyncio.coroutine(fn). Having the caller pass in the already-called
coroutine object might simplify the signature even further. I'm not sure I
see the advantage of trying to make this an executor -- but perhaps I'm
missing something?

--Guido



On Sat, Sep 26, 2015 at 7:29 AM, Vincent Michel <vxgmichel at gmail.com> wrote:

> Hi,
>
> I noticed there is currently no standard solution to submit a job from a
> thread to an asyncio event loop.
>
> Here's what the asyncio documentation says about concurrency and
> multithreading:
>
> > To schedule a callback from a different thread, the
> BaseEventLoop.call_soon_threadsafe() method should be used.
> > Example to schedule a coroutine from a different thread:
> >     loop.call_soon_threadsafe(asyncio.async, coro_func())
>
> The issue with this method is the loss of the coroutine result.
>
> One way to deal with this issue is to connect the asyncio.Future returned
> by async (or ensure_future) to a concurrent.futures.Future. It is then
> possible to use a subclass of concurrent.futures.Executor to submit a
> callback to an asyncio event loop. Such an executor can also be used to set
> up communication between two event loops using run_in_executor.
>
> I posted an implementation called LoopExecutor on GitHub:
> https://github.com/vxgmichel/asyncio-loopexecutor
> The repo contains the loopexecutor module along with tests for several use
> cases. The README describes the whole thing (context, examples, issues,
> implementation).
>
> It is interesting to note that this executor is a bit different than
> ThreadPoolExecutor and ProcessPoolExecutor since it can also submit a
> coroutine function. Example:
>
> with LoopExecutor(loop) as executor:
>     future = executor.submit(operator.add, 1, 2)
>     assert future.result() == 3
>     future = executor.submit(asyncio.sleep, 0.1, result=3)
>     assert future.result() == 3
>
> This works in both cases because submit always cast the given function to
> a coroutine. That means it would also work with a function that returns a
> Future.
>
> Here's a few topic related to the current implementation that might be
> interesting to discuss:
>
> - possible drawback of casting the callback to a coroutine
> - possible drawback of concurrent.future.Future using
> asyncio.Future._copy_state
> - does LoopExecutor need to implement the shutdown method?
> - removing the limitation in run_in_executor (can't submit a coroutine
> function)
> - adding a generic Future connection function in asyncio
> - reimplementing wrap_future with the generic connection
> - adding LoopExecutor to asyncio (or concurrent.futures)
>
> At the moment, the interaction between asyncio and concurrent.futures only
> goes one way. It would be nice to have a standard solution (LoopExecutor or
> something else) to make it bidirectional.
>
> Thanks,
>
> Vincent
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150926/eb2977e7/attachment.html>