[Python-Dev] issue 6721 "Locks in python standard library should be sanitized on fork"

[Ask Solem]

On 26 Aug 2011, at 16:53, Antoine Pitrou wrote:

> 
> Hi,
> 
>> I think that "deprecating" the use of threads w/ multiprocessing - or
>> at least crippling it is the wrong answer. Multiprocessing needs the
>> helper threads it uses internally to manage queues, etc. Removing that
>> ability would require a near-total rewrite, which is just a
>> non-starter.
> 
> I agree that this wouldn't actually benefit anyone.
> Besides, I don't think it's even possible to avoid threads in
> multiprocessing, given the various constraints. We would have to force
> the user to run their main thread in an event loop, and that would be
> twisted (tm).
> 
>> I would focus on the atfork() patch more directly, ignoring
>> multiprocessing in the discussion, and focusing on the merits of gps'
>> initial proposal and patch.
> 
> I think this could also be combined with Charles-François' patch.
> 
> Regards



Have to agree with Jesse and Antoine here.

Celery (celeryproject.org) uses multiprocessing, is wildly used in production,
and is regarded as stable software that have been known to run for months at a time
only to be restarted for software upgrades.

I have been investigating an issue for some time, that I'm pretty sure is caused
by this.  It occurs only rarely, so rarely I have not had any actual bug reports
about it, it's just something I have experienced during extensive testing.
The tone of the discussion on the bug tracker makes me think that I have
been very lucky :-)

Using the fork+exec approach seems like a much more realistic solution
than rewriting multiprocessing.Pool and Manager to not use threads. In fact
this is something I have been considering as a fix for the suspected
issue for for some time.
It does have implications that are annoying for sure, but we are already
used to this on the Windows platform (it could help portability even).

-- 
Ask Solem
twitter.com/asksol | +44 (0)7713357179