[Python-Dev] pthreads, fork, import, and execvp

["Martin v. Löwis"]

Nick Coghlan wrote:
> Rotem took the time to explain the problem to me more fully in private email, 
> and it appears that using import statements inside functions is currently 
> fundamentally unsafe in a posix environment where both multiple threads and 
> fork() are used. If any thread other than the one invoking fork() holds the 
> import lock at the time of the fork then import statements in the spawned 
> process will deadlock.
> 
> So I believe fixing this properly would indeed require assigning a new lock 
> object in one of the two places Rotem suggested.

I still don't understand the problem fully. According to POSIX, fork does:

"A process is created with a single thread. If a multi-threaded process
calls fork(), the new process contains a replica of the calling thread
and its entire address space, possibly including the states of mutexes
and other resources."

So if the the import lock was held during the fork call, the new thread
will hold the import lock of the new process, and subsequent imports
will block.

However, then the problem is not with the execve implementation, but
with the fact that the import lock was held when the process forked.

Rotem should simply avoid to fork() in the toplevel code of a module.

This problem should be fixed in Python 2.5, whose PyOS_AfterFork now
reads

void
PyOS_AfterFork(void)
{
#ifdef WITH_THREAD
        PyEval_ReInitThreads();
        main_thread = PyThread_get_thread_ident();
        main_pid = getpid();
        _PyImport_ReInitLock();
#endif
}

This was added in

------------------------------------------------------------------------
r39522 | gvanrossum | 2005-09-14 20:09:42 +0200 (Mi, 14 Sep 2005) | 4 lines

- Changes donated by Elemental Security to make it work on AIX 5.3
  with IBM's 64-bit compiler (SF patch #1284289).  This also closes SF
  bug #105470: test_pwd fails on 64bit system (Opteron).

------------------------------------------------------------------------

Unfortunately, that fix would apply to AIX only, though:

void
_PyImport_ReInitLock(void)
{
#ifdef _AIX
        if (import_lock != NULL)
                import_lock = PyThread_allocate_lock();
#endif
}

If we want to support fork while an import is going on, we should
release the import lock if it is held. Alternatively, the code
could throw away the old import lock on all systems; that seems
like a waste of resources to me, though. One should also reset
import_lock_thread and import_lock_level.

I'd be curious if Elemental Security added this code only to solve
the very same problem, i.e. a fork that happened with the import
lock held.

Regards,
Martin