[New-bugs-announce] [issue6963] Add worker process lifetime to multiprocessing.Pool - patch included

[Charles Cazabon]

New submission from Charles Cazabon <charlesc-python at pyropus.ca>:

Worker processes with multiprocessing.Pool live for the duration of the
Pool.  If the tasks they run happen to leak memory (from a C extension
module, or from creating cycles of unreachable objects, etc) or open
files or other resources, there's no easy way to clean them up.

Similarly, if one task passed to the pool allocates a large amount of
memory, but further tasks are small, that additional memory isn't
returned to the system because the process involved hasn't exited.

A common approach to this problem (as used by Apache, mod_wsgi, and
various other software) is to allow worker processes to exit (and be
replaced with fresh processes) after completing a specified amount of
work.  The attached patch (against Python 2.6.2, but applies to various
other versions with some fuzz) implements this as optional new behaviour
in multiprocessing.Pool().  An additional optional argument is specified
for the maximum number of tasks a worker process performs before it
exits and is replaced with a fresh worker process.

----------
components: Library (Lib)
files: worker-lifetime-python2.6.2.patch
keywords: patch
messages: 92971
nosy: charlesc
severity: normal
status: open
title: Add worker process lifetime to multiprocessing.Pool - patch included
type: feature request
versions: Python 2.7, Python 3.1
Added file: http://bugs.python.org/file14946/worker-lifetime-python2.6.2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6963>
_______________________________________