[Python-Dev] pthreads, fork, import, and execvp

[Thomas Wouters]

On Thu, May 18, 2006 at 20:02, "Martin v. Löwis" <martin at v.loewis.de> wrote:

> Nick Coghlan wrote:
> > And if I understand it correctly, it falls under the category that
> > waiting for another thread while holding the import lock is a *really*
> > bad idea from a thread safety point of view.
> >
> > The thing with the import-after-fork deadlock is that you can trigger it
> > without even doing anything that's known not to be thread-safe.
>
> Right. With some googling, I found that one solution is pthread_atexit:
> a pthread_atexit handler is a triple (before, in_parent, in_child).
> You set it to (acquire, release, release). When somebody forks,
> the pthread library will first acquire the import lock in the thread
> that wants to fork. Then the fork occurs, and the import lock gets
> then released both in the parent and in the child.
>
> I would like to see this approach implemented, but I agree with you
> that a test case should be created first.
>

Picking up a rather old discussion... We encountered this bug at Google and
I'm now "incentivized" to fix it.

For a short recap: Python has an import lock that prevents more than one
thread from doing an import at any given time. However, unlike most of the
locks we have lying around, we don't clear that lock in the child after an
os.fork(). That means that doing an os.fork() during an import means the
child process can't do any other imports. It also means that doing an
os.fork() *while another thread is doing an import* means the child process
can't do any other imports.

Since this three-year-old discussion we've added a couple of
post-fork-cleanups to CPython (the TLS, the threading module's idea of
active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do
simply discard the memory for other locks held during fork (the GIL, see
Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in
Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock,
except when the platform is AIX. I don't see any particular reason why we
aren't doing the same thing to the import lock that we do to the other
locks, on all platforms. It's a quick fix for a real problem (see
http://bugs.python.org/issue1590864 and
http://bugs.python.org/issue1404925 for two bugreports that seem to be this
very issue.)

It also seems to me, since we have two or three (depending on how you count)
uses for it, that we should either add a way to free the old locks (to the
various threading implementations) or add a way to use pthread_atexit
portably -- or at least portably across systems with fork(). I don't think
we should wait with fixing the big issue (making threads and fork() being
unnecessarily flakey) until we have a good fix for the small issue (a tiny
bit of a memory leak after a fork() -- in the child process only.)
Especially since the good fix for the small issue might require
co-ordination between all the threading implementations we have and nobody
knows about.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090716/a11ca265/attachment.htm>