[New-bugs-announce] [issue31886] Multiprocessing.Pool hangs after re-spawning several worker process.
report at bugs.python.org
Fri Oct 27 11:10:27 EDT 2017
New submission from olarn <bo.bantukulolarn at gmail.com>:
Multiprocessing's pool apparently attempts to repopulate the pool in an event of sub-process worker crash. However the pool seems to hangs after about ~ 4*(number of worker) process re-spawns.
I've tracked the issue down to queue.get() stalling at multiprocessing.pool, line 102
Is this a known issue? Are there any known workaround?
To reproduce this issue:
multiprocessing.util._logger = multiprocessing.util.log_to_stderr(logging.DEBUG)
print("attempting to crash the interpreter in ", multiprocessing.current_process())
i = ctypes.c_char('a'.encode())
j = ctypes.pointer(i)
c = 0
j[c] = 'a'.encode()
c += 1
print("test_fn in ", multiprocessing.current_process().name, x)
if __name__ == '__main__':
# pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
pool = multiprocessing.Pool(processes=1)
args_queue = [n for n in range(20)]
# subprocess quits
# subprocess crashes
components: Library (Lib)
title: Multiprocessing.Pool hangs after re-spawning several worker process.
versions: Python 2.7, Python 3.6
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce