[Python-Dev] Fwd: shutdown concurrent.futures.ProcessPoolExecutor

[Giampaolo Rodola']

This is a follow up of:
https://groups.google.com/forum/#!topic/python-tulip/91NCCqV4SFs
According to the information I collected so far it seems it's not possible
(or at least very hard) to cleanly shutdown a process pool and all its
workers in case of KeyboardInterrupt / SIGINT.
Literally, what I would like to do is this:

- intercept SIGINT form the main process and/or from each worker process
- send SIGINT to all workers
- wait for them to finish with a timeout
- finally exit

Whereas it exists a solution based on multiprocessing:
http://noswap.com/blog/python-multiprocessing-keyboardinterrupt
...concurrent.futures.ProcessPoolExecutor does not expose all the necessary
bits to emulate it.

By opening this thread I hope either to find a solution and document it or
if it turns out that it is simply not possible to do this via
concurrent.futures then discuss whether it's the case to expose more
ProcessPoolExecutor APIs, because right now the interface it offers is
pretty limited compared to multiprocessing.Pool:
http://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool
In particular, it won't let you pass an initializer function to pass to
multiprocessing.Pool nor terminate() the process pool (only wait() for it).



---------- Forwarded message ----------
From: Guido van Rossum <guido at python.org>
Date: Sat, Mar 29, 2014 at 12:05 AM
Subject: Re: [python-tulip] How to cleanly shutdown the IO loop when using
run_in_executor()?
To: Giampaolo Rodola' <g.rodola at gmail.com>
Cc: python-tulip <python-tulip at googlegroups.com>


You're going to have to move the discussion to python-dev or the python
issue tracker then.


On Fri, Mar 28, 2014 at 4:02 PM, Giampaolo Rodola' <g.rodola at gmail.com>wrote:

>
> On Wed, Mar 26, 2014 at 7:35 PM, Guido van Rossum <guido at python.org>wrote:
>
>> Another thing to investigate might be how the executor creates the
>> processes, and if anything happens to signals there.
>
>
> After some further digging this seems to be related with the problem at
> hand.
> According to this:
> http://noswap.com/blog/python-multiprocessing-keyboardinterrupt
> ...it appears a feasible solution is to prevent workers to ever receive
> KeyboardInterrupt and have the main process shutdown the pool via
> pool.terminate() and finally pool.join().
> Unfortunately concurrent.futures.ProcessPoolExecutor does not expose all
> the necessary bits to do that (multiprocessing.Pool(initializer=...)
> argument and terminate() method).
>
> I also suspect that in order to emulate the suggested solution the
> underlying Process instance should be daemonized, similarly to what
> multiprocessing.Pool does:
>
> http://hg.python.org/cpython/file/3567d7ebd382/Lib/multiprocessing/pool.py#l222
>
> I wonder whether concurrent.futures.ProcessPoolExecutor would be better
> off exposing more APIs in order to facilitate tasks like these.
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
>


-- 
--Guido van Rossum (python.org/~guido)



-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140329/23e490a0/attachment.html>