[New-bugs-announce] [issue31782] Add a timeout to multiprocessing's Pool.join

[Will Starms]

New submission from Will Starms <vilhelmen at gmail.com>:

Pool's join function currently (3.6.3) lacks a timeout, which can cause the managing thread to sleep indefinitely when a pool worker hangs or starts misbehaving. Adding a timeout allows the owning thread to attempt a join and, after the timeout, return to other tasks, such as monitoring worker health.

In my specific situation, I have a Pool running a task on a large set of files. If any single task fails, the whole operation is ruined and the pool should be terminated. A task can communicate with the main thread through error_callback, but if the thread has already called join, it can't check until join returns, after the Pool has finished all processing.

Attached is an incredibly simple patch to the current (3.6) cpython implementation that emulates threading.thread.join's behavior.

----------
components: Library (Lib)
files: cpython_timeout.patch
keywords: patch
messages: 304350
nosy: Will Starms
priority: normal
severity: normal
status: open
title: Add a timeout to multiprocessing's Pool.join
type: enhancement
versions: Python 3.6
Added file: https://bugs.python.org/file47219/cpython_timeout.patch

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31782>
_______________________________________