[New-bugs-announce] [issue17273] multiprocessing.pool.Pool task/worker handlers are not fork safe

[Arun Babu Neelicattu]

New submission from Arun Babu Neelicattu:

The task/worker handler threads in the multiprocessing.pool.Pool class are (in accordance to posix standards) not copied over when the process containing the pool is forked.

This leads to a situation where the Pool keeps receiving tasks but the tasks never get handled. This could potentially lead to deadlocks if AsyncResult.wait() is called.

Not sure if this should be considered as a bug, or an invalid use case. However, this becomes a problem when importing modules that use pools and the main code uses multiprocessing too.

[BAD] Workaround:
Reassigning Pool._task_handler to a new instance of threading.Thread after the fork seems to work in the case highlighted in the example.

Environment:
Fedora 18 
Linux 3.7.8-202.fc18.x86_64 #1 SMP Fri Feb 15 17:33:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
python3-3.3.0-1.fc18.x86_64

An example of this issue is shown below:

from multiprocessing import Pool, Process

def t2():
	# We expect the pool to handle this
	print('t2: Hello!')

pool = Pool()
def t1():
	# We assign a task to the pool
	pool.apply_async(t2)
	print('t1: Hello!')

if __name__ == '__main__':
	# Process() forks the main process containing the pool
	Process(target=t1).start()

----------
components: Library (Lib)
files: pool_forking.py
messages: 182647
nosy: abn
priority: normal
severity: normal
status: open
title: multiprocessing.pool.Pool task/worker handlers are not fork safe
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file29157/pool_forking.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17273>
_______________________________________