Nick Coghlan wrote:
Rotem took the time to explain the problem to me more fully in private email, and it appears that using import statements inside functions is currently fundamentally unsafe in a posix environment where both multiple threads and fork() are used. If any thread other than the one invoking fork() holds the import lock at the time of the fork then import statements in the spawned process will deadlock.
So I believe fixing this properly would indeed require assigning a new lock object in one of the two places Rotem suggested.
I still don't understand the problem fully. According to POSIX, fork does: "A process is created with a single thread. If a multi-threaded process calls fork(), the new process contains a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources." So if the the import lock was held during the fork call, the new thread will hold the import lock of the new process, and subsequent imports will block. However, then the problem is not with the execve implementation, but with the fact that the import lock was held when the process forked. Rotem should simply avoid to fork() in the toplevel code of a module. This problem should be fixed in Python 2.5, whose PyOS_AfterFork now reads void PyOS_AfterFork(void) { #ifdef WITH_THREAD PyEval_ReInitThreads(); main_thread = PyThread_get_thread_ident(); main_pid = getpid(); _PyImport_ReInitLock(); #endif } This was added in ------------------------------------------------------------------------ r39522 | gvanrossum | 2005-09-14 20:09:42 +0200 (Mi, 14 Sep 2005) | 4 lines - Changes donated by Elemental Security to make it work on AIX 5.3 with IBM's 64-bit compiler (SF patch #1284289). This also closes SF bug #105470: test_pwd fails on 64bit system (Opteron). ------------------------------------------------------------------------ Unfortunately, that fix would apply to AIX only, though: void _PyImport_ReInitLock(void) { #ifdef _AIX if (import_lock != NULL) import_lock = PyThread_allocate_lock(); #endif } If we want to support fork while an import is going on, we should release the import lock if it is held. Alternatively, the code could throw away the old import lock on all systems; that seems like a waste of resources to me, though. One should also reset import_lock_thread and import_lock_level. I'd be curious if Elemental Security added this code only to solve the very same problem, i.e. a fork that happened with the import lock held. Regards, Martin