Tim Peters wrote:
Florent's DeadlockDebugger in turn builds on an external C threadframe module:
http://www.majid.info/mylos/stories/2004/06/10/threadframe.html
Folding the functionality of that (or similar functionality) into the core would, IMO, be a valuable addition for 2.5, and would make an excellent intro project for an aspiring contributor interested in how threads work in CPython (what this module does is conceptually simple). It belongs in the core because it's not safe to chase the tstate chain without holding pystate.c's internal head_mutex lock (holding the GIL isn't enough -- it's normal practice to call PyThreadState_Delete() while not holding the GIL).
I'd do it myself (and maybe I will anyway), but this really would make a good (finite; conceptually simple) project for someone who wants to gain Python developer experience.
Since I started this, I might as well finish it. I do have some Python developer experience (hey, I even voted for comp.lang.python back when...) but not in the core interpreter itself. I suspect integrating this feature (let's call it sys._current_frames() for the sake of argument, although it probably belongs in the threads module) in the core is not going to be quite as trivial as you say, as there are potential memory leaks. If the function returns a dictionary, it should probably be a weak dict to avoid a circular reference between the frame of the thread that calls _current_frames and its locals that contain the returned dictionary. But that would also mean the references to other threads' frames are going to be weak, thus not allowing the current thread to inspect their locals and backtraces at will as those weak references may be broken. -- Fazal Majid Email: python-dev@fromemail.com Home: +1 415 359-0918 1111 Jones Apt. 1 Cell: +1 415 244-1337 San Francisco, CA 94109, USA Web: www.majid.info
At 02:42 PM 3/6/05 -0800, Fazal Majid wrote:
I suspect integrating this feature (let's call it sys._current_frames() for the sake of argument, although it probably belongs in the threads module) in the core is not going to be quite as trivial as you say, as there are potential memory leaks. If the function returns a dictionary, it should probably be a weak dict to avoid a circular reference between the frame of the thread that calls _current_frames and its locals that contain the returned dictionary.
Given the primary use case as a debugging tool, I don't think the circularity will have any significant problems. It would be simpler to just document that a caller should do this: try: framedict = sys._current_frames() # do stuff here finally: framedict = None # break the cycle, allowing GC
[Fazal Majid]
Since I started this, I might as well finish it. I do have some Python developer experience (hey, I even voted for comp.lang.python back when...) but not in the core interpreter itself.
Cool! WRT your current module, it would need changes to follow Python's C coding style, to check _every_ C API call for an error return, and to grab head_mutex while crawling over the tstate chain.
I suspect integrating this feature (let's call it sys._current_frames() for the sake of argument, although it probably belongs in the threads module)
I expect that Phillip's thought here is that the sys module already has a _getframe() function, so everyone who knows that would likely expect a new frame-retrieval function to be exposed in sys too.
in the core is not going to be quite as trivial as you say, as there are potential memory leaks.
Good news: I don't think so.
If the function returns a dictionary, it should probably be a weak dict to avoid a circular reference between the frame of the thread that calls _current_frames and its locals that contain the returned dictionary. But that would also mean the references to other threads' frames are going to be weak, thus not allowing the current thread to inspect their locals and backtraces at will as those weak references may be broken.
dicts and frames both participate in cyclic gc in recent Pythons. For example, you can run this all day in 2.4 and not see memory grow (provided you don't disable cyclic gc): def f(): import sys myframe = sys._getframe() while 1: f() Frames aren't weakly referenceable anyway:
import weakref import sys f = sys._getframe() weakref.ref(f) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: cannot create weak reference to 'frame' object
On 06 March 2005, Fazal Majid said:
Since I started this, I might as well finish it. I do have some Python developer experience (hey, I even voted for comp.lang.python back when...) but not in the core interpreter itself.
What would be *really* spiffy is to provide a way for
externally-triggered thread dumps. This is one of my top two Java
features [1]. The way this works in Java is a bit awkward --
"kill -QUIT" the Java process and it writes a traceback for every
running thread to stdout -- but it works. Something similar ought to be
possible for Python, although optional (because Python apps can handle
signals themselves, unlike Java apps).
It could be as simple as this: calling
sys.enablethreaddump(signal=signal.SIGQUIT)
from the program enables externally-triggered thread dumps via the
specified signal.
Greg
[1] The other is compiler recognition of "@deprecated" in doc comments.
--
Greg Ward
At 08:23 PM 3/7/05 -0500, Greg Ward wrote:
On 06 March 2005, Fazal Majid said:
Since I started this, I might as well finish it. I do have some Python developer experience (hey, I even voted for comp.lang.python back when...) but not in the core interpreter itself.
What would be *really* spiffy is to provide a way for externally-triggered thread dumps. This is one of my top two Java features [1]. The way this works in Java is a bit awkward -- "kill -QUIT" the Java process and it writes a traceback for every running thread to stdout -- but it works. Something similar ought to be possible for Python, although optional (because Python apps can handle signals themselves, unlike Java apps).
It could be as simple as this: calling
sys.enablethreaddump(signal=signal.SIGQUIT)
from the program enables externally-triggered thread dumps via the specified signal.
Couldn't this just be done with traceback.print_stack(), given the _current_frames() facility? About the only real problem with it is that the other threads can keep running while you're trying to print the stack dumps. I suppose you could set the check interval really high and then restore it afterwards as a sneaky way of creating a critical section. Unfortunately, there's no getcheckinterval(). Which reminds me, btw, it would be nice while we're adding more execution control functions to have a way to get the current trace hook and profiling hook, not to mention ways to set them on non-current threads. You can do these things from C of course; I mean accessible as part of the Python API.
[Greg Ward]
What would be *really* spiffy is to provide a way for externally-triggered thread dumps. This is one of my top two Java features [1]. The way this works in Java is a bit awkward -- "kill -QUIT" the Java process and it writes a traceback for every running thread to stdout -- but it works. Something similar ought to be possible for Python, although optional (because Python apps can handle signals themselves, unlike Java apps).
It could be as simple as this: calling
sys.enablethreaddump(signal=signal.SIGQUIT)
from the program enables externally-triggered thread dumps via the specified signal.
See the link in my original post to Florent's Zope deadlock debugger. Things like the above are easy enough to create _given_ a bit of C code in the core to build on. [Phillip J. Eby]
Couldn't this just be done with traceback.print_stack(), given the _current_frames() facility?
Right.
About the only real problem with it is that the other threads can keep running while you're trying to print the stack dumps.
I don't know that it matters. If you're debugging a deadlocked thread, its stack isn't going to change. If you're trying to find out where unexpected time is getting swallowed, statements in the offending loop(s) are still going to show up in the stack trace.
I suppose you could set the check interval really high and then restore it afterwards as a sneaky way of creating a critical section. Unfortunately, there's no getcheckinterval().
sys.getcheckinterval() was added in Python 2.3.
Which reminds me, btw, it would be nice while we're adding more execution control functions to have a way to get the current trace hook and profiling hook, not to mention ways to set them on non-current threads. You can do these things from C of course; I mean accessible as part of the Python API.
Huh. It didn't remind me of those at all <wink>.
"Phillip J. Eby"
Which reminds me, btw, it would be nice while we're adding more execution control functions to have a way to get the current trace hook and profiling hook,
Well, there's the f_trace member of frame objects, but I honestly can't remember what it's for...
not to mention ways to set them on non-current threads. You can do these things from C of course; I mean accessible as part of the Python API.
Again, given sys._current_frames(), this shouldn't be very hard. Cheers, mwh -- And then the character-only displays went away (leading to increasingly silly graphical effects and finally to ads on web pages). -- John W. Baxter, comp.lang.python
At 10:03 AM 3/8/05 +0000, Michael Hudson wrote:
"Phillip J. Eby"
writes: Which reminds me, btw, it would be nice while we're adding more execution control functions to have a way to get the current trace hook and profiling hook,
Well, there's the f_trace member of frame objects, but I honestly can't remember what it's for...
It sets the *local* trace function, which is only active if a *global* trace function is set for the thread. The global trace function is the thing you can't get at from Python.
participants (5)
-
Fazal Majid
-
Greg Ward
-
Michael Hudson
-
Phillip J. Eby
-
Tim Peters