[Python-Dev] issue 6721 "Locks in python standard library should be sanitized on fork"

[Jesse Noller]

On Fri, Aug 26, 2011 at 3:18 AM, Nir Aides <nir at winpdb.org> wrote:
> Another face of the discussion is about whether to deprecate the mixing of
> the threading and processing modules and what to do about the
> multiprocessing module which is implemented with worker threads.

There's a bug open - http://bugs.python.org/issue8713 which would
offer non windows users the ability to avoid using fork() entirely,
which would sidestep the problem outlined in the atfork() bug. Under
windows, which has no fork() mechanism, we create a subprocess and
then use pipes for intercommunication: nothing is inherited from the
parent process except the state passed into the child.

I think that "deprecating" the use of threads w/ multiprocessing - or
at least crippling it is the wrong answer. Multiprocessing needs the
helper threads it uses internally to manage queues, etc. Removing that
ability would require a near-total rewrite, which is just a
non-starter.

I'd rather examine bug 8713 more closely, and offer this option for
all users in 3.x and document the existing issues outlined in
http://bugs.python.org/issue6721 for 2.x - the proposals in that bug
are IMHO, out of bounds for a 2.x release.

In essence; the issue here is multiprocessing's use of fork on unix
without the following exec - which is what the windows implementation
essentially does using subprocess.

Adding the option to *not* fork changes the fundamental behavior on
unix systems - but I fundamentally feel that it's a saner, and more
consistent behavior for the module as a whole.

So, I'd ask that we not talk about tearing out the ability to use MP
and threads, or threads with MP - that would be crippling, and there's
existing code in the wild (including multiprocessing itself) that uses
this mix without issue - it's stripping out functionality for what is
a surprising and painful edge case that rarely directly affects users.

I would focus on the atfork() patch more directly, ignoring
multiprocessing in the discussion, and focusing on the merits of gps'
initial proposal and patch.

jesse