[Python-Dev] help debugging thread deadlocks

[Tim Peters]

[Russ Cox]
> If this should be in python-list, say so and I'll move,

Yes, it should -- it's not about developing Python.

> but python-dev seemed a better match.

To what <wink>?

> I've got a python program that's hanging in
> PyThread_acquire_lock called from fast_cfunction,

Which version of Python, release or debug build, which OS, which C compiler
and libc (or moral equivalent), which of the many versions of Python threads
(pthreads, Windows threads, etc)?  Any extension modules loaded?

> which I assume means the python program itself is
> calling the acquire method on a python-allocated
> (as opposed to C-allocated) lock.

Hard to say but a reasonable guess.

> There are two threads and both are hung trying to acquire
> different locks.

Well, there's your problem <heh>.

> Unfortunately that's all I can tell from the
> stack trace.

I'm not sure how anyone could tell more.  There are some locks and you're
staring at a deadlock.  This isn't rare in threaded programs, alas.  Are you
by any chance spawning and executing a thread as a side-effect of doing an
import, where the spawned thread itself does an import and the spawning code
waits for the spawned thread to finish?  If so, the spawning thread would
wait forever on whatever gimmick it's using to wait for the spawned thread
to finish, and the spawned thread would wait forever for the global import
lock to get released (which can't happen, because the global import lock is
waiting for the spawning thread to release it, but the spawning thread is
waiting for the spawned thread to finish).  Or, more simply, never spawn a
thread as a side-effect of importing.

> My python program allocates three locks using threading.RLock, and
> I can't find any other python code (in the standard modules),

Most library modules use the older thread module, so you'll find more by
searching for allocate_lock.  The most commonly hit locks of this kind are
in tempfile.py and Queue.py.

> but the locks that are hanging are not those three locks.

If everything's hung, how do you know this?

> I looked in the standard module library code and can't
> find any that allocate locks.  There are some locks
> allocated in the C interpreter, of course, but I don't
> see how those would be acquirable from python code.
>
> Is there some function I can call from within
> acquire_lock or allocate_lock (from C) to dump
> a python stack trace to stdout or stderr?

I'm not clear on what you're asking here.  See the traceback module for
convenient ways to dump the Python stack.  Also, if the particular Python
thread implementation you use supports it, in a debug build you can set the
envar THREADDEBUG to 15 to get a trace of calls made to the lowest-level
thread routines.  For example, on Windows

C:\Code\python\PCbuild>set THREADDEBUG=15

C:\Code\python\PCbuild>python_d
PyThread_init_thread called
-1471359: PyThread_allocate_lock() -> 00960830
-1471359: PyThread_acquire_lock(00960830, 1) called
-1471359: PyThread_acquire_lock(00960830, 1) -> 1
-1471359: PyThread_release_lock(00960830) called
-1471359: PyThread_acquire_lock(00960830, 1) called
-1471359: PyThread_acquire_lock(00960830, 1) -> 1
-1471359: PyThread_release_lock(00960830) called
PyThread_allocate_lock called
-1471359: PyThread_allocate_lock() -> 0096B030
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
Adding parser accelerators ...
Done.
-1471359: PyThread_release_lock(0096B030) called
Python 2.3a0 (#29, Mar  1 2002, 16:33:03) [MSC 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
-1471359: PyThread_acquire_lock(0096B030, 0) called
-1471359: PyThread_acquire_lock(0096B030, 0) -> 1
-1471359: PyThread_release_lock(0096B030) called
>>>

Also run a debug build with the -v switch to get a trace of imports.
Stubbing your toe on the import lock is the only way I know of for a vanilla
(no C extension modules) Python program to get in trouble with a deadlock
involving an internal Python lock.