[New-bugs-announce] [issue38501] multiprocessing.Pool hangs atexit (and garbage collection sometimes)
Eric Larson
report at bugs.python.org
Wed Oct 16 14:31:14 EDT 2019
New submission from Eric Larson <larson.eric.d at gmail.com>:
The following code hangs on Python 3.8.0.rc0 on Ubuntu 19.10 when exiting the interpreter:
from multiprocessing import Pool
class A(object):
def __init__(self):
self.pool = Pool(processes=2)
solver = A()
When you eventually do ctrl-C, the traceback is:
^CProcess ForkPoolWorker-2:
Error in atexit._run_exitfuncs:
Process ForkPoolWorker-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/util.py", line 277, in _run_finalizers
finalizer()
File "/usr/lib/python3.8/multiprocessing/util.py", line 201, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 689, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/usr/lib/python3.8/multiprocessing/pool.py", line 674, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/lib/python3.8/multiprocessing/queues.py", line 355, in get
with self._rlock:
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.8/multiprocessing/pool.py", line 114, in worker
task = get()
File "/usr/lib/python3.8/multiprocessing/queues.py", line 356, in get
res = self._reader.recv_bytes()
File "/usr/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
A similar problem seems to occur when garbage collecting when it's part of a class, and that class has not yet called `self.pool.terminate()` and `self.pool.close()`. Hopefully fixing the `atexit` behavior will similarly fix the `gc.collect()` behavior.
Cross-ref is in SciPy, https://github.com/scipy/scipy/issues/10927. It appears to also cause hangs on Travis 3.8-dev builds: https://travis-ci.org/scipy/scipy/jobs/598786785
----------
components: Library (Lib)
messages: 354813
nosy: Eric Larson
priority: normal
severity: normal
status: open
title: multiprocessing.Pool hangs atexit (and garbage collection sometimes)
type: behavior
versions: Python 3.8
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38501>
_______________________________________
More information about the New-bugs-announce
mailing list