[New-bugs-announce] [issue13778] Python should invalidate all non-owned 'thread.lock' objects when forking

[lesha]

New submission from lesha <pybug.20.lesha at xoxy.net>:

Here is a great description of the issue:

http://docs.oracle.com/cd/E19683-01/806-6867/gen-1/index.html


This enhancement proposes a way to make Python more resistant to this kind of deadlock.


Consider this program:


import threading
import subprocess
import time

l = threading.Lock() 

def f():
    l.acquire()  
    time.sleep(1)
    l.release()
 
t = threading.Thread(target=f)
t.start()

def g(l):
    l.acquire()
    l.release() 
    print 'ohai'


subprocess.Popen(['ls'], preexec_fn=lambda: g(l))



g() gets called in the forked process, which means that it's waiting on a *copy* of the lock, which can never get released.


This, in turn, means that the main thread will forever wait for the Popen to finish.



The program above incorrectly assumes that a threading lock can be shared across fork() parent and child.

I suspect adding such sharing is impractical, requiring OS support or excessive complexity. If the sharing could be had cheaply, it would be great -- programs like this would work as intended, but no other programs would break. 

Crazy idea: free the locks. Sadly, that is not safe! The ones that are currently locked by other threads might be protecting some network resource, and allowing the fork child to access them would result in a logical error.

However, it is always a bad idea for a fork() child to access a lock that is held by a thread that is not its fork() parent. That lock was locked at the time of the fork(), and will stay locked, because the child process will not get updated by the lock-holding threads.

So, it is always invalid to access that type of lock. Currently, you are guaranteed a deadlock.

Proposal: trying to acquire such a lock should crash the forked child with a nice, detailed error message (including the offending lock), rather than hang the entire program.

Sample steps to implement:

1) Store the process ID on each lock instance.
2) Acquire/release should crash if the lock does not belong to the current thread AND has a different process ID from the current one.

There are other potential implementations, such as explicitly enumerating such locks at the time of fork, and invalidating them.

This crash cannot be an exception in the child, because lock methods must not throw. However, it can and should be an exception in the fork() parent.

I think this enhancement would make it much easier to debug this kind of problem. It's an easy mistake to make, because preexec_fn or fork docs do not warn you of the danger, and locks can be acquired quite implicitly by innocent-looking code.

----------
components: Library (Lib)
messages: 151165
nosy: lesha
priority: normal
severity: normal
status: open
title: Python should invalidate all non-owned 'thread.lock' objects when forking
type: enhancement
versions: Python 2.6, Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13778>
_______________________________________