help debugging thread deadlocks
If this should be in python-list, say so and I'll move, but python-dev seemed a better match. I've got a python program that's hanging in PyThread_acquire_lock called from fast_cfunction, which I assume means the python program itself is calling the acquire method on a python-allocated (as opposed to C-allocated) lock. There are two threads and both are hung trying to acquire different locks. Unfortunately that's all I can tell from the stack trace. My python program allocates three locks using threading.RLock, and I can't find any other python code (in the standard modules), but the locks that are hanging are not those three locks. I looked in the standard module library code and can't find any that allocate locks. There are some locks allocated in the C interpreter, of course, but I don't see how those would be acquirable from python code. Is there some function I can call from within acquire_lock or allocate_lock (from C) to dump a python stack trace to stdout or stderr? Thanks. Russ
[Russ Cox]
If this should be in python-list, say so and I'll move,
Yes, it should -- it's not about developing Python.
but python-dev seemed a better match.
To what <wink>?
I've got a python program that's hanging in PyThread_acquire_lock called from fast_cfunction,
Which version of Python, release or debug build, which OS, which C compiler and libc (or moral equivalent), which of the many versions of Python threads (pthreads, Windows threads, etc)? Any extension modules loaded?
which I assume means the python program itself is calling the acquire method on a python-allocated (as opposed to C-allocated) lock.
Hard to say but a reasonable guess.
There are two threads and both are hung trying to acquire different locks.
Well, there's your problem <heh>.
Unfortunately that's all I can tell from the stack trace.
I'm not sure how anyone could tell more. There are some locks and you're staring at a deadlock. This isn't rare in threaded programs, alas. Are you by any chance spawning and executing a thread as a side-effect of doing an import, where the spawned thread itself does an import and the spawning code waits for the spawned thread to finish? If so, the spawning thread would wait forever on whatever gimmick it's using to wait for the spawned thread to finish, and the spawned thread would wait forever for the global import lock to get released (which can't happen, because the global import lock is waiting for the spawning thread to release it, but the spawning thread is waiting for the spawned thread to finish). Or, more simply, never spawn a thread as a side-effect of importing.
My python program allocates three locks using threading.RLock, and I can't find any other python code (in the standard modules),
Most library modules use the older thread module, so you'll find more by searching for allocate_lock. The most commonly hit locks of this kind are in tempfile.py and Queue.py.
but the locks that are hanging are not those three locks.
If everything's hung, how do you know this?
I looked in the standard module library code and can't find any that allocate locks. There are some locks allocated in the C interpreter, of course, but I don't see how those would be acquirable from python code.
Is there some function I can call from within acquire_lock or allocate_lock (from C) to dump a python stack trace to stdout or stderr?
I'm not clear on what you're asking here. See the traceback module for convenient ways to dump the Python stack. Also, if the particular Python thread implementation you use supports it, in a debug build you can set the envar THREADDEBUG to 15 to get a trace of calls made to the lowest-level thread routines. For example, on Windows C:\Code\python\PCbuild>set THREADDEBUG=15 C:\Code\python\PCbuild>python_d PyThread_init_thread called -1471359: PyThread_allocate_lock() -> 00960830 -1471359: PyThread_acquire_lock(00960830, 1) called -1471359: PyThread_acquire_lock(00960830, 1) -> 1 -1471359: PyThread_release_lock(00960830) called -1471359: PyThread_acquire_lock(00960830, 1) called -1471359: PyThread_acquire_lock(00960830, 1) -> 1 -1471359: PyThread_release_lock(00960830) called PyThread_allocate_lock called -1471359: PyThread_allocate_lock() -> 0096B030 -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 Adding parser accelerators ... Done. -1471359: PyThread_release_lock(0096B030) called Python 2.3a0 (#29, Mar 1 2002, 16:33:03) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called -1471359: PyThread_acquire_lock(0096B030, 0) called -1471359: PyThread_acquire_lock(0096B030, 0) -> 1 -1471359: PyThread_release_lock(0096B030) called
Also run a debug build with the -v switch to get a trace of imports. Stubbing your toe on the import lock is the only way I know of for a vanilla (no C extension modules) Python program to get in trouble with a deadlock involving an internal Python lock.
participants (2)
-
Russ Cox -
Tim Peters