[issue29759] Deadlock in multiprocessing.pool.Pool on terminate

[Michael]

New submission from Michael:

Following code snippet causes a deadlock on Linux:

"""
import multiprocessing.pool
import signal


def signal_handler(signum, frame):
    pass

if __name__ == '__main__':
    signal.signal(signal.SIGTERM, signal_handler)
    pool = multiprocessing.pool.Pool(processes=1)
    pool.terminate() # alternatively - raise Exception("EXCEPTION")
"""

The reason is that the termination code starts before the worker processes being fully initialized.

Here, parent process acquires a forever-lock:

"""
    @staticmethod
    def _help_stuff_finish(inqueue, task_handler, size):
        # task_handler may be blocked trying to put items on inqueue
        util.debug('removing tasks from inqueue until task handler finished')
        inqueue._rlock.acquire()          < -----------------
        while task_handler.is_alive() and inqueue._reader.poll():
            inqueue._reader.recv()
            time.sleep(0)
"""

And then the worker processes are getting stuck here:

"""
def worker(...):

    while maxtasks is None or (maxtasks and completed < maxtasks):
        try:
            task = get()                  < ----------------- trying to acquire the same lock
        except (EOFError, OSError):
            util.debug('worker got EOFError or OSError -- exiting')
            break


"""

Whats going on then? As far as the default process start method is set to 'fork', worker subprocesses inherit parent's signal handler. Trying to terminate workers from _terminate_pool() doesn't have any effect. Finally, processes enter into a deadlock when parent join()-s workers.

----------
components: Library (Lib)
messages: 289248
nosy: mapozyan
priority: normal
severity: normal
status: open
title: Deadlock in multiprocessing.pool.Pool on terminate
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29759>
_______________________________________