[New-bugs-announce] [issue31886] Multiprocessing.Pool hangs after re-spawning several worker process.

[olarn]

New submission from olarn <bo.bantukulolarn at gmail.com>:

Multiprocessing's pool apparently attempts to repopulate the pool in an event of sub-process worker crash. However the pool seems to hangs after about ~ 4*(number of worker) process re-spawns.

I've tracked the issue down to queue.get() stalling at multiprocessing.pool, line 102

Is this a known issue? Are there any known workaround?

To reproduce this issue:

import multiprocessing
import multiprocessing.util
import logging

multiprocessing.util._logger = multiprocessing.util.log_to_stderr(logging.DEBUG)
import time
import ctypes


def crash_py_interpreter():
    print("attempting to crash the interpreter in ", multiprocessing.current_process())
    i = ctypes.c_char('a'.encode())
    j = ctypes.pointer(i)
    c = 0
    while True:
        j[c] = 'a'.encode()
        c += 1
    j


def test_fn(x):
    print("test_fn in ", multiprocessing.current_process().name, x)
    exit(0)

    time.sleep(0.1)


if __name__ == '__main__':

    # pool = multiprocessing.Pool(processes=multiprocessing.cpu_count())
    pool = multiprocessing.Pool(processes=1)

    args_queue = [n for n in range(20)]

    # subprocess quits
    pool.map(test_fn, args_queue)

    # subprocess crashes
    # pool.map(test_fn,queue)

----------
components: Library (Lib)
messages: 305124
nosy: olarn
priority: normal
severity: normal
status: open
title: Multiprocessing.Pool hangs after re-spawning several worker process.
type: behavior
versions: Python 2.7, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31886>
_______________________________________