[New-bugs-announce] [issue17273] multiprocessing.pool.Pool task/worker handlers are not fork safe
Arun Babu Neelicattu
report at bugs.python.org
Fri Feb 22 07:20:53 CET 2013
New submission from Arun Babu Neelicattu:
The task/worker handler threads in the multiprocessing.pool.Pool class are (in accordance to posix standards) not copied over when the process containing the pool is forked.
This leads to a situation where the Pool keeps receiving tasks but the tasks never get handled. This could potentially lead to deadlocks if AsyncResult.wait() is called.
Not sure if this should be considered as a bug, or an invalid use case. However, this becomes a problem when importing modules that use pools and the main code uses multiprocessing too.
Reassigning Pool._task_handler to a new instance of threading.Thread after the fork seems to work in the case highlighted in the example.
Linux 3.7.8-202.fc18.x86_64 #1 SMP Fri Feb 15 17:33:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
An example of this issue is shown below:
from multiprocessing import Pool, Process
# We expect the pool to handle this
pool = Pool()
# We assign a task to the pool
if __name__ == '__main__':
# Process() forks the main process containing the pool
components: Library (Lib)
title: multiprocessing.pool.Pool task/worker handlers are not fork safe
versions: Python 3.3
Added file: http://bugs.python.org/file29157/pool_forking.py
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce