Why do we need Traceback Objects?
Hi, while fixing my design flaws after Just's Stackless Mac port, I was dealing with some overflow conditions and tracebacks. When there is a recursion depth overflow condition, we create a lot of new structure for the tracebacks. This usually happens in a situation where memory is quite exhausted. Even worse if we crash because of a memory error: The system will not have enough memory to build the traceback structure, to report the error. Puh :-) When I look into tracebacks, it turns out to be just a chain like the frame chain, but upward down. It holds references to the frames in a 1-to-1 manner, and it keeps copies of f->f_lasti and f->f_lineno. I don't see why this is needed. I'm thinking to replace the tracebacks by a single pointer in the frames for this purpose. It appears further to be possible to do that without any extra memory, since all the frames have extra temporary fields for exception info, and that isn't used in this context. Traceback objects exist each for one and only one frame, and they could be embedded into their frame. Does this make sense? Do I miss something? I'm considering this for Stackless and would like to know if I should prepare it for orthodox Python as well? ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
When I look into tracebacks, it turns out to be just a chain like the frame chain, but upward down. It holds references to the frames in a 1-to-1 manner, and it keeps copies of f->f_lasti and f->f_lineno. I don't see why this is needed.
I'm thinking to replace the tracebacks by a single pointer in the frames for this purpose. It appears further to be possible to do that without any extra memory, since all the frames have extra temporary fields for exception info, and that isn't used in this context. Traceback objects exist each for one and only one frame, and they could be embedded into their frame.
Does this make sense? Do I miss something?
Yes. It is quite possible to have multiple stack traces lingering around that all point to the same stack frames. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote: [me, about embedding tracebacks into frames]
Does this make sense? Do I miss something?
Yes. It is quite possible to have multiple stack traces lingering around that all point to the same stack frames.
Oh, I see. This is a Standard Python specific thing, which I was about to forget. In my version, this can happen, too, unless you are in a continuation-protected context already. There (and that was what I looked at while debugging), this situation can never happen, since an exception creates continuation-copies of all the frames while it crawls up. Since the traceback causes refcount increase, all the frames protect themselves. Thank you. I see it is a stackless feature. I can implement it if I put protection into the core, not just the co-extension. Frames can carry the tracebacks under the condition that they are protected (copied) if the traceback fields are occupied. Great, since this is a rare condition. Thanks again for the enlightment - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
[Christian]
When I look into tracebacks, it turns out to be just a chain like the frame chain, but upward down. It holds references to the frames in a 1-to-1 manner, and it keeps copies of f->f_lasti and f->f_lineno. I don't see why this is needed. ... Does this make sense? Do I miss something?
[Guido]
Yes. It is quite possible to have multiple stack traces lingering around that all point to the same stack frames.
This reminds me that some time ago I made an experimental patch for removing SET_LINENO. There was the problem of generating callbacks for pdb (which I think I solved somehow but I don't remember the details). I do remember that I had to look at pdb again for some reason. Is there any interest in reviving this idea? -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Vladimir> This reminds me that some time ago I made an experimental Vladimir> patch for removing SET_LINENO. There was the problem of Vladimir> generating callbacks for pdb (which I think I solved somehow Vladimir> but I don't remember the details). I do remember that I had to Vladimir> look at pdb again for some reason. Is there any interest in Vladimir> reviving this idea? I believe you can get line number information from a code object's co_lnotab attribute, though I don't know the format. I think this should be sufficient to allow SET_LINENO to be eliminated altogether. It's just that there are places in various modules that predate the appearance of co_lnotab. Whoops, wait a minute. I just tried >>> def foo(): pass ... >>> foo.func_code.co_lnotab with both "python" and "python -O". co_lnotab is empty for python -O. I thought it was supposed to always be generated? -- Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/
Skip Montanaro wrote:
Whoops, wait a minute. I just tried
>>> def foo(): pass ... >>> foo.func_code.co_lnotab
with both "python" and "python -O". co_lnotab is empty for python -O. I thought it was supposed to always be generated?
It is always generated, but since co_lnotab contains only lineno increments starting from co_firstlineno (i.e. only deltas) and your function is a 1-liner (no lineno increments starting from the first line of the function), the table is empty. Move 'pass' to the next line and the table will contain 1-entry (of 2 bytes: delta_addr, delta_line). Generally speaking, the problem really boils down to the callbacks from C to Python when a tracefunc is set. My approach is not that bad in this regard. A decent processor nowadays has (an IRQ pin) a flag for generating interrupts on every processor instruction (trace flag). In Python, we have the same problem - we need to interrupt the (virtual) processor, implemented in eval_code2() on regular intervals. Actually, what we need (for pdb) is to interrupt the processor on every source line, but one could easily imagine a per instruction interrupt (with a callback installed with sys.settracei(). This is exactly what the patch does under the grounds. It interrupts the processor on every new source line (but interrupting it on every instruction would be a trivial extension -- all opcodes in the code stream would be set to CALL_TRACE!) And this is exactly what LINENO does (+ some processor state saving in the frame: f_lasti, f_lineno). Clearly, there are 2 differences with the existing code: a) The interrupting opcodes are installed dynamically, on demand, only when a trace function is set, for the current traced frame. Presently, these opcodes are SET_LINENO; I introduced a new one byte CALL_TRACE opcode which does the same thing (thus preserving backwards compatibility with old .pyc that contain SET_LINENO). b) f_lasti and f_lineno aren't updated when the frame is not traced :-( I wonder whether we really care about them, though. The other implementation details aren't so important. Yet, they look scary, but no more than the co_lnotab business. The problem with my patch is point b). I believe the approach is good, though -- if it weren't, I woudn't have taken the care to talk about it detail. :-) -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Vladimir Marangozov wrote:
[Christian]
When I look into tracebacks, it turns out to be just a chain like the frame chain, but upward down. It holds references to the frames in a 1-to-1 manner, and it keeps copies of f->f_lasti and f->f_lineno. I don't see why this is needed. ... Does this make sense? Do I miss something?
[Guido]
Yes. It is quite possible to have multiple stack traces lingering around that all point to the same stack frames.
This reminds me that some time ago I made an experimental patch for removing SET_LINENO. There was the problem of generating callbacks for pdb (which I think I solved somehow but I don't remember the details). I do remember that I had to look at pdb again for some reason. Is there any interest in reviving this idea?
This is a very cheap opcode (at least in my version). What does it buy? Can you drop the f_lineno field from frames, and calculate it for the frame's f_lineno attribute? -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com
Vladimir Marangozov wrote: This reminds me that some time ago I made an experimental patch for removing SET_LINENO. There was the problem of generating callbacks for pdb (which I think I solved somehow but I don't remember the details). I do remember that I had to look at pdb again for some reason. Is there any interest in reviving this idea?
I think the details are important. The only thing the SET_LINENO opcode does is to call a trace function if one is installed. It's necessary to have some way to invoke the trace function when the line number changes (or it will be relatively difficult to execute code line-by-line in the debugger <wink>). Off the top of my head, the only other way I see to invoke the trace function would be to add code at the head of the mainloop that computed the line number for each instruction (from lnotab) and called the trace function if the current line number is different than the previous time through the loop. That doesn't sound faster or simpler. Jeremy
Jeremy Hylton wrote:
Vladimir Marangozov wrote: This reminds me that some time ago I made an experimental patch for removing SET_LINENO. There was the problem of generating callbacks for pdb (which I think I solved somehow but I don't remember the details). I do remember that I had to look at pdb again for some reason. Is there any interest in reviving this idea?
I think the details are important. The only thing the SET_LINENO opcode does is to call a trace function if one is installed. It's necessary to have some way to invoke the trace function when the line number changes (or it will be relatively difficult to execute code line-by-line in the debugger <wink>).
Looking back at the discussion and the patch I ended up with at that time, I think the callback issue was solved rather elegantly. I'm not positive that it does not have side effects, though... For an overview of the approach and the corresponding patch, go back to: http://www.python.org/pipermail/python-dev/1999-August/002252.html http://sirac.inrialpes.fr/~marangoz/python/lineno/ What happens is that in tracing mode, a copy of the original code stream is created, a new CALL_TRACE opcode is stored in it at the addresses corresponding to each source line number, then the instruction pointer is redirected to execute the modified code string. Whenever a CALL_TRACE opcode is reached, the callback is triggered. On a successful return, the original opcode at the current address is fetched from the original code string, then directly goto the dispatch code. This code string duplication & conditional break-point setting occurs only when a trace function is set; in the "normal case", the interpreter executes a code string without SET_LINENO. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
What happens is that in tracing mode, a copy of the original code stream is created, a new CALL_TRACE opcode is stored in it at the addresses corresponding to each source line number, then the instruction pointer is redirected to execute the modified code string. Whenever a CALL_TRACE opcode is reached, the callback is triggered. On a successful return, the original opcode at the current address is fetched from the original code string, then directly goto the dispatch code.
This code string duplication & conditional break-point setting occurs only when a trace function is set; in the "normal case", the interpreter executes a code string without SET_LINENO.
Ai! This really sounds like a hack. It may be a standard trick in the repertoire of virtual machine implementers, but it is still a hack, and makes my heart cry. I really wonder if it makes enough of a difference to warrant all that code, and the risk that that code isn't quite correct. (Is it thread-safe?) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
What happens is that in tracing mode, a copy of the original code stream is created, a new CALL_TRACE opcode is stored in it at the addresses corresponding to each source line number, then the instruction pointer is redirected to execute the modified code string. Whenever a CALL_TRACE opcode is reached, the callback is triggered. On a successful return, the original opcode at the current address is fetched from the original code string, then directly goto the dispatch code.
This code string duplication & conditional break-point setting occurs only when a trace function is set; in the "normal case", the interpreter executes a code string without SET_LINENO.
Ai! This really sounds like a hack. It may be a standard trick in the repertoire of virtual machine implementers, but it is still a hack, and makes my heart cry.
The implementation sounds tricky, yes. But there's nothing hackish in the principle of setting breakpoints. The modified code string is in fact the stripped code stream (without LINENO), reverted back to a standard code stream with LINENO. However, to simplify things, the LINENO (aka CALL_TRACE) are not inserted between the instructions for every source line. They overwrite the original opcodes in the copy whenever a trace function is set (i.e. we set all conditional breakpoints (LINENO) at once). And since we overwrite for simplicity, at runtime, we read the ovewritten opcodes from the original stream, after the callback returns. All this magic occurs before the main loop, with finalization on exit of eval_code2. A tricky implementation of the principle of having a set of conditional breakpoints for every source line (these cond. bp are currently the SET_LINENO opcodes, in a more redundant version).
I really wonder if it makes enough of a difference to warrant all that code, and the risk that that code isn't quite correct.
Well, all this business is internal to ceval.c and doesn't seem to affect the rest of the world. I can see only two benefits (if this idea doesn't hide other mysteries -- so anyone interested may want check it out): 1) Some tiny speedup -- we'll reach -O in a standard setup 2) The .pyc files become smaller. (Lib/*.pyc is reduced by ~80K for 1.5.2) No other benefits (hmmm, maybe the pdb code will be simplified wrt linenos) I originally developped this idea because of the redundant, consecutive SET_LINENO in a code object.
(Is it thread-safe?)
I think so. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
participants (5)
-
Christian Tismer -
Guido van Rossum -
Jeremy Hylton -
Skip Montanaro -
Vladimir.Marangozov＠inrialpes.fr