[Async-sig] async executors

[Christoph Groth]

Hi,

A few days ago I posted a message [1] on python-ideas to which no 
one replied except Brett who suggested re-posting it here, which I 
do now.  The following is a complete rewrite/update of the 
original message.  I promise: I will stop the spamming if no one 
replies this time!

Async programming with executors seems to be a useful approach for 
number-crunching problems where the main algorithm (the one 
launching the futures) is quite complicated and possibly 
concurrent, but light enough to run on a single core (the heavy 
lifting is done “inside” the futures).

This use case is addressed by asyncio's run_in_executor that uses 
the executors provided by the concurrent.futures package.  There 
is, however, a lot of duplication of functionality between 
concurrent.futures and asyncio.  (The main difference is that 
concurrent.futures relies on threading and locks.)  If one is 
going to use concurrent.futures only from coroutines that run 
under the control of the asyncio event loop, it should be possible 
to replace the concurrent.futures executors by async versions of 
them.

Why could such an async executor be interesting?

• It can be a lot simpler (and thus easier to understand and 
  extend) because it builds on top of asyncio.  There have been 
  already discussions on unifying the Future type from 
  concurrent.futures with that of asyncio [2].

• Performance is better, especially when many worker processes are 
  involved, because of the simpler code that uses less locking 
  (the only locking that remains is inside the multiprocessing 
  module).

As far as I can see the original 
concurrent.futures.ProcessPoolExecutor has no advantage when used 
in asyncio-based programs except when some coroutine blocks for 
long enough for the call queue to become empty.  (But hey, async 
programming is about non-blocking coroutines!)

Are there other possible advantages or disadvantages?

Based on concurrent.futures.ProcessPoolExecutor, I’ve made a 
proof-of-concept implementation [3] that demonstrates that the 
idea works.  (There have been some improvements compared to the 
version that I posted on python-ideas.)

Christoph


PS.

I would be grateful if any asyncio experts could have a look at 
the part of the main loop of the process management coroutine 
where the coroutine awaits new data [4].  Currently, I am using an 
asyncio.Event that is set by a callback via asyncio’s 
add_reader().  Is this the most natural way to do it currently? 
I’m particularly puzzled by the fact that reader.poll(0) is not 
always true after the event has been set.

If there is no better way to combine asyncio and 
multiprocessing.Process, perhaps this is a thing that could be 
improved in a future version of the Python stdlib?


[1] http://thread.gmane.org/gmane.comp.python.ideas/40709
[2] 
http://thread.gmane.org/gmane.comp.python.ideas/24551/focus=24557
[3] https://gitlab.kwant-project.org/cwg/aexecutor
[4] 
https://gitlab.kwant-project.org/cwg/aexecutor/blob/9290504e779c543d9ee93adcb1de36390a69eaad/aexecutor/process.py#L298
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/5e88d6b4/attachment-0001.sig>