[Python-ideas] Submitting a job to an asyncio event loop

[Guido van Rossum]

OK, I think I understand your primary use case -- the C++ library calls
callbacks in their own threads but you want the callback code to run in
your event loop, where presumably it is structured as a coroutine and may
use `yield from` or `await` to wait for other coroutines, tasks or futures.
Then when that coroutine is done it returns a value which your machinery
passes back as the result of a concurrent.futures.Future on which the
callback thread is waiting.

I don't think the use case involving multiple event loops in different
threads is as clear. I am still waiting for someone who is actually trying
to use this. It might be useful on a system where there is a system event
loop that must be used for UI events (assuming this event loop can somehow
be wrapped in a custom asyncio loop) and where an app might want to have a
standard asyncio event loop for network I/O. Come to think of it, the
ProactorEventLoop on Windows has both advantages and disadvantages, and
some app might need to use both that and SelectorEventLoop. But this is a
real pain (because you can't share any mutable state between event loops).



On Sun, Sep 27, 2015 at 6:36 AM, Vincent Michel <vxgmichel at gmail.com> wrote:

> Hi Guido,
>
> Thanks for your interest,
>
> I work for a synchrotron and we use the distributed control system
> TANGO. The main implementation is in C++, but we use a python binding
> called PyTango. The current server implementation (on the C++ side)
> does not feature an event loop but instead create a different thread
> for each client.
>
> TANGO: http://www.tango-controls.org/
> PyTango:
> http://www.esrf.eu/computing/cs/tango/tango_doc/kernel_doc/pytango/latest/index.html
>
> I wanted to add asyncio support to the library, so that we can benefit
> from single-threaded asynchronous programming. The problem is that
> client callbacks run in different threads and there is not much we can
> do about it until a pure python implementation is developed (and it's
> a lot of work). Instead, it is possible to use an asyncio event loop,
> run the server through run_in_executor (juste like you mentioned in
> your mail), and redirect all the client callbacks to the event loop.
> That's the part where job submission from a different thread comes in
> handy.
>
> A very similar solution has been developed using gevent, but I like
> explicit coroutines better :p
>
> Another use case is the communication between two event loops. From
> what I've seen, the current context (get/set event loop) is only
> related to the current thread. It makes it easy to run different event
> loops in different threads. Even though I'm not sure what the use case
> is, I suppose it's been done intentionally. Then the executor
> interface is useful to run things like:
>
> executor = LoopExecutor(other_loop)
> result = await my_loop.run_in_executor(executor, coro_func, *args)
>
> There is working example in the test directory:
>
> https://github.com/vxgmichel/asyncio-loopexecutor/blob/master/test/test_multi_loop.py
>
> ***
>
> The coroutine(fn) cast only makes sense if a subclass of Executor is
> used, in order to be consistent with the Executor.submit signature.
> Otherwise, passing an already-called coroutine is perfectly fine. I
> think it is a good idea to define a simple submit function like you
> recommended:
>
> def submit_to_loop(loop, coro):
>     future = concurrent.futures.Future()
>     callback = partial(schedule, coro, destination=future)
>     loop.call_soon_threadsafe(callback)
>     return future
>
> And then use the executor interface if we realize it is actually
> useful. It's really not a lot of code anyway:
>
> class LoopExecutor(concurrent.futures.Executor):
>
>     def __init__(self, loop=None):
>         self.loop = loop or asyncio.get_event_loop()
>
>     def submit(self, fn, *args, **kwargs):
>         coro = asyncio.coroutine(fn)(*args, **kwargs)
>         return submit_to_loop(self.loop, coro)
>
> I'll update the repository.
>
> Cheers,
>
> Vincent
>
> 2015-09-27 4:52 GMT+02:00 Guido van Rossum <guido at python.org>:
> >
> > Hi Vincent,
> >
> > I've read your write-up with interest. You're right that it's a bit
> awkward to make calls from the threaded world into the asyncio world.
> Interestingly, there's much better support for passing work off from the
> asyncio event loop to a thread (run_in_executor()). Perhaps that's because
> the use case there was obvious from the start: some things that may block
> for I/O just don't have an async interface yet, so in order to use them
> from an asyncio task they must be off-loaded to a separate thread or else
> the entire event loop is blocked. (This is used for calling getaddrinfo(),
> for example.)
> >
> > I'm curious where you have encountered the opposite use case?
> >
> > I think if I had to do this myself I would go for a more minimalist
> interface: something like your submit() method but without the call to
> asyncio.coroutine(fn). Having the caller pass in the already-called
> coroutine object might simplify the signature even further. I'm not sure I
> see the advantage of trying to make this an executor -- but perhaps I'm
> missing something?
> >
> > --Guido
> >
> >
> >
> > On Sat, Sep 26, 2015 at 7:29 AM, Vincent Michel <vxgmichel at gmail.com>
> wrote:
> >>
> >> Hi,
> >>
> >> I noticed there is currently no standard solution to submit a job from
> a thread to an asyncio event loop.
> >>
> >> Here's what the asyncio documentation says about concurrency and
> multithreading:
> >>
> >> > To schedule a callback from a different thread, the
> BaseEventLoop.call_soon_threadsafe() method should be used.
> >> > Example to schedule a coroutine from a different thread:
> >> >     loop.call_soon_threadsafe(asyncio.async, coro_func())
> >>
> >> The issue with this method is the loss of the coroutine result.
> >>
> >> One way to deal with this issue is to connect the asyncio.Future
> returned by async (or ensure_future) to a concurrent.futures.Future. It is
> then possible to use a subclass of concurrent.futures.Executor to submit a
> callback to an asyncio event loop. Such an executor can also be used to set
> up communication between two event loops using run_in_executor.
> >>
> >> I posted an implementation called LoopExecutor on GitHub:
> >> https://github.com/vxgmichel/asyncio-loopexecutor
> >> The repo contains the loopexecutor module along with tests for several
> use cases. The README describes the whole thing (context, examples, issues,
> implementation).
> >>
> >> It is interesting to note that this executor is a bit different than
> ThreadPoolExecutor and ProcessPoolExecutor since it can also submit a
> coroutine function. Example:
> >>
> >> with LoopExecutor(loop) as executor:
> >>     future = executor.submit(operator.add, 1, 2)
> >>     assert future.result() == 3
> >>     future = executor.submit(asyncio.sleep, 0.1, result=3)
> >>     assert future.result() == 3
> >>
> >> This works in both cases because submit always cast the given function
> to a coroutine. That means it would also work with a function that returns
> a Future.
> >>
> >> Here's a few topic related to the current implementation that might be
> interesting to discuss:
> >>
> >> - possible drawback of casting the callback to a coroutine
> >> - possible drawback of concurrent.future.Future using
> asyncio.Future._copy_state
> >> - does LoopExecutor need to implement the shutdown method?
> >> - removing the limitation in run_in_executor (can't submit a coroutine
> function)
> >> - adding a generic Future connection function in asyncio
> >> - reimplementing wrap_future with the generic connection
> >> - adding LoopExecutor to asyncio (or concurrent.futures)
> >>
> >> At the moment, the interaction between asyncio and concurrent.futures
> only goes one way. It would be nice to have a standard solution
> (LoopExecutor or something else) to make it bidirectional.
> >>
> >> Thanks,
> >>
> >> Vincent
> >>
> >>
> >> _______________________________________________
> >> Python-ideas mailing list
> >> Python-ideas at python.org
> >> https://mail.python.org/mailman/listinfo/python-ideas
> >> Code of Conduct: http://python.org/psf/codeofconduct/
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150927/ed3f1bc6/attachment-0001.html>