[New-bugs-announce] [issue6721] Locks in python standard library should be sanitized on fork

[Gregory P. Smith]

New submission from Gregory P. Smith <greg at krypto.org>:

The python logging module uses a lock to surround many operations, in 
particular.  This causes deadlocks in programs that use logging, fork 
and threading simultaneously.

1) spawn one or more threads in your program
2) have at least one of those threads make logging calls that will be 
emitted.
3) have your main thread or another thread use os.fork() to run some 
python code in a child process.
4) If the fork happened while one of your threads was within the 
logging.Handler.handle() critical section (or anywhere else where 
handler.lock is acquired), your child process will deadlock as soon as 
it tries to log anything.  It inherited a held lock.

The deadlock is more likely to happen on a highly loaded system which 
tends to widen the deadlock opportunity window due to context switching.

A demo of the problem simplified into one file is attached.

The Python standard library should not be the cause of these deadlocks.  
We need a way for all standard library locks to be cleaned up when 
forking.  By doing one of the following:

A) acquire all locks before forking, release them immediately after.
B) forceably release all standard library locks after forking in the 
child process.

Code was added to call some cleanups after forking in 
http://bugs.python.org/issue874900 but there are more things that also 
need this same sort of cleanup (logging for example).

Rather than having to manually add after fork code hooks into every file 
in the standard library that uses locks, a more general solution to 
track and manage locks across fork would be a good idea.

----------
assignee: gregory.p.smith
files: lock_fork_thread_deadlock_demo.py
messages: 91674
nosy: gregory.p.smith
priority: high
severity: normal
status: open
title: Locks in python standard library should be sanitized on fork
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2
Added file: http://bugs.python.org/file14740/lock_fork_thread_deadlock_demo.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue6721>
_______________________________________