Set-next-statement in Python debuggers
Hi, First a brief introduction, since this is my first post to python-dev: I'm a professional software engineer, mostly using C/C++ and Python. I use Python a lot for my own projects and some of you will know me from the spambayes project, as the author of pop3proxy.py. I was interested to see Armin's patch [617312] to "allow the C profile and trace functions to alter some fields of the current PyFrameObject". I'm writing a Python debugger, and as Armin says, his patch "could also be used by advanced debuggers to allow the execution point to be modified by the user." A "Set-next-statement" feature is on my list of nice-to-have-but-hard-to-do jobs, and this patch could make it easy. (I know the blockstack presents a problem, but if it's insurmountable I'm happy to ignore that and to only allow skipping to lines in the same block - 90% of my use of the equivalent feature in MSVC is to skip back one line or forward one line.) There's one fly in the ointment - I'm trying to keep my debugger in pure Python, and Armin's patch only applies to C trace functions. If frame.f_lasti were writable by Python trace functions, pure Python debuggers (including pdb) could implement Set-Next-Statement. Here's a small script that demonstrates what I mean, and the one-line patch to frameobject.c that enables it: ----------------------------------------------------------------------- """Demonstration of debugger 'Set-next-statement' feature. Requires Python 2.2.2 with writeable frame.f_lasti. Prints "1, 2, 2, 3".""" import sys, pprint def example(): print 1, print 2, print 3 # Line 9; we skip back to 8 the first time we hit this. class DebuggerHook: def __init__(self): self.done = False # Have we done the Set-next-statement yet? self.lasti = 0 # The frame.f_lasti of the previous line. sys.settrace(self.trace) def trace(self, frame, action, arg): if not self.done and frame.f_lineno == 9: # This is the first time we've hit line 9; skip back to 8. frame.f_lasti = self.lasti self.done = True else: # Store the instruction offset of this line. self.lasti = frame.f_lasti return self.trace debugger = DebuggerHook() example() # Prints "1, 2, 2, 3" ----------------------------------------------------------------------- *** frameobject.c Tue Oct 8 08:16:39 2002 --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 *************** *** 16,20 **** {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti)}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, --- 16,20 ---- {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti), RO}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, ----------------------------------------------------------------------- For completeness, f_lineno should probably be writable as well, but I'm keeping the changes minimal for the purposes of discussion. Is this a reasonable suggestion for 2.3, or is it giving people too much rope? A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen. I would say that the power of a Set-next-statement feature in pdb and other debuggers is worth the price, but others may disagree...? -- Richie Hindle richie@entrian.com
Hi, [Hell's bells, sorry, here's this again with the line-breaks fixed! How embarrassing...] First a brief introduction, since this is my first post to python-dev: I'm a professional software engineer, mostly using C/C++ and Python. I use Python a lot for my own projects and some of you will know me from the spambayes project, as the author of pop3proxy.py. I was interested to see Armin's patch [617312] to "allow the C profile and trace functions to alter some fields of the current PyFrameObject". I'm writing a Python debugger, and as Armin says, his patch "could also be used by advanced debuggers to allow the execution point to be modified by the user." A "Set-next-statement" feature is on my list of nice-to-have-but-hard-to-do jobs, and this patch could make it easy. (I know the blockstack presents a problem, but if it's insurmountable I'm happy to ignore that and to only allow skipping to lines in the same block - 90% of my use of the equivalent feature in MSVC is to skip back one line or forward one line.) There's one fly in the ointment - I'm trying to keep my debugger in pure Python, and Armin's patch only applies to C trace functions. If frame.f_lasti were writable by Python trace functions, pure Python debuggers (including pdb) could implement Set-Next-Statement. Here's a small script that demonstrates what I mean, and the one-line patch to frameobject.c that enables it: ----------------------------------------------------------------------- """Demonstration of debugger 'Set-next-statement' feature. Requires Python 2.2.2 with writeable frame.f_lasti. Prints "1, 2, 2, 3".""" import sys, pprint def example(): print 1, print 2, print 3 # Line 9; we skip back to 8 the first time we hit this. class DebuggerHook: def __init__(self): self.done = False # Have we done the Set-next-statement yet? self.lasti = 0 # The frame.f_lasti of the previous line. sys.settrace(self.trace) def trace(self, frame, action, arg): if not self.done and frame.f_lineno == 9: # This is the first time we've hit line 9; skip back to 8. frame.f_lasti = self.lasti self.done = True else: # Store the instruction offset of this line. self.lasti = frame.f_lasti return self.trace debugger = DebuggerHook() example() # Prints "1, 2, 2, 3" ----------------------------------------------------------------------- *** frameobject.c Tue Oct 8 08:16:39 2002 --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 *************** *** 16,20 **** {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti)}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, --- 16,20 ---- {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti), RO}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, ----------------------------------------------------------------------- For completeness, f_lineno should probably be writable as well, but I'm keeping the changes minimal for the purposes of discussion. Is this a reasonable suggestion for 2.3, or is it giving people too much rope? A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen. I would say that the power of a Set-next-statement feature in pdb and other debuggers is worth the price, but others may disagree...? -- Richie Hindle richie@entrian.com
Hello Richie, On Tue, 8 Oct 2002 richie@entrian.com wrote:
(...) A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen.
You can already crash the interpreter with pure Python code, for example via the new.code() constructor or by writing crappy .pyc files. On Linux you can also open("/proc/self/mem", "w"). I don't think that people get their hands on frame objects by pure chance, but the possibility exists. Moreover, conditionally allowing changes to f_lasti is limiting and complex because of the stack and block stack. For safety I'd consider writing the frame-object-modifying code in a C extension module, carefully documented as "don't use this". Armin
On Tue, 8 Oct 2002 richie@entrian.com wrote:
(...) A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen.
[Armin]
You can already crash the interpreter with pure Python code, for example via the new.code() constructor or by writing crappy .pyc files.
Yes, and I occasionally lose sleep over those. I definitely don't want to add more loopholes like that, and I'd like to fix those. In the past, the new module was optional for precisely this reason (and so is the dl module). I would like to have a check on .pyc files (or on unmarshalled code objects, really).
On Linux you can also open("/proc/self/mem", "w").
That doesn't mean that python should abandon the policy "a core dump is always considered Python's fault unless proven otherwise." Protecting you against using *external* tools may not be possible; but Python should not include features (like writable code object attributes) that can cause crashes when used inexpertly.
I don't think that people get their hands on frame objects by pure chance, but the possibility exists. Moreover, conditionally allowing changes to f_lasti is limiting and complex because of the stack and block stack. For safety I'd consider writing the frame-object-modifying code in a C extension module, carefully documented as "don't use this".
That's a reasonable solution: f_lasti should be read-only from Python code, but you can write an extension that can write it. --Guido van Rossum (home page: http://www.python.org/~guido/)
Hello Guido, On Wed, 9 Oct 2002, Guido van Rossum wrote:
[Armin]
You can already crash the interpreter with pure Python code, for example via the new.code() constructor or by writing crappy .pyc files.
Yes, and I occasionally lose sleep over those. I definitely don't want to add more loopholes like that, and I'd like to fix those.
Ok. I already more or less committed to have a serious look over all the existing overflow problems that could cause a core dump (http://www.python.org/sf/618623). I may add in my to-do list the checks that no boggy data can crash unmarshall (after a first survey of the code I'd say that such data might exist; e.g. r_string() has a memcpy() that could be called with a negative size). More importantly, if you think that there is use for a complete "code object checker" that could catch any code doing silly things, then I'd love to contribute. Note however that this is something that will have to be kept closely in sync with future compiler extensions. A bientot, Armin.
[Armin]
You can already crash the interpreter with pure Python code, for example via the new.code() constructor or by writing crappy .pyc files.
[Guido]
Yes, and I occasionally lose sleep over those. I definitely don't want to add more loopholes like that, and I'd like to fix those.
[Armin]
Ok. I already more or less committed to have a serious look over all the existing overflow problems that could cause a core dump (http://www.python.org/sf/618623). I may add in my to-do list the checks that no boggy data can crash unmarshall (after a first survey of the code I'd say that such data might exist; e.g. r_string() has a memcpy() that could be called with a negative size).
Fixes for this and similar problems will be most welcome!
More importantly, if you think that there is use for a complete "code object checker" that could catch any code doing silly things, then I'd love to contribute. Note however that this is something that will have to be kept closely in sync with future compiler extensions.
Yes, the fact that our VM isn't standardized makes this a bit tricky. But I think it ought to be done. --Guido van Rossum (home page: http://www.python.org/~guido/)
Hi, [Guido]
Python should not include features (like writable code object attributes) that can cause crashes when used inexpertly.
If I do implement this, I'll do it in such a way as to make crashes impossible (my first suggestion said that crashes would be possible, but I realise that was a mistake). It'll verify that the new value of f_lasti is a valid bytecode position, and it'll either fix up the stack/blockstack or refuse to move the position in ways that would affect them. (Armin, I guess that won't restrict you because you're doing this from C code?) [Armin]
For safety I'd consider writing the frame-object-modifying code in a C extension module, carefully documented as "don't use this".
[Guido]
That's a reasonable solution: f_lasti should be read-only from Python code, but you can write an extension that can write it.
I'd like to see pure Python debuggers (including pdb) have this feature. If it's safe to use, might it still be considered for inclusion? (I can't guarantee I'll have the time to do it, but I'll try.) I'm also coming to realise that it's not f_lasti I really want to change, but f_lineno. The line number is the 'unit' that debuggers use - a debugger would set f_lineno, and f_lasti would be calculated and changed to the appropriate value. I don't think that changes any of the arguments either way, except maybe to distance the idea a little from what Armin is doing. -- Richie Hindle richie@entrian.com
[Guido]
Python should not include features (like writable code object attributes) that can cause crashes when used inexpertly.
If I do implement this, I'll do it in such a way as to make crashes impossible (my first suggestion said that crashes would be possible, but I realise that was a mistake). It'll verify that the new value of f_lasti is a valid bytecode position, and it'll either fix up the stack/blockstack or refuse to move the position in ways that would affect them. (Armin, I guess that won't restrict you because you're doing this from C code?)
Great.
[Armin]
For safety I'd consider writing the frame-object-modifying code in a C extension module, carefully documented as "don't use this".
[Guido]
That's a reasonable solution: f_lasti should be read-only from Python code, but you can write an extension that can write it.
I'd like to see pure Python debuggers (including pdb) have this feature. If it's safe to use, might it still be considered for inclusion? (I can't guarantee I'll have the time to do it, but I'll try.)
Sure, *if* it's safe.
I'm also coming to realise that it's not f_lasti I really want to change, but f_lineno. The line number is the 'unit' that debuggers use - a debugger would set f_lineno, and f_lasti would be calculated and changed to the appropriate value. I don't think that changes any of the arguments either way, except maybe to distance the idea a little from what Armin is doing.
Sure. --Guido van Rossum (home page: http://www.python.org/~guido/)
*** frameobject.c Tue Oct 8 08:16:39 2002 --- frameobject-222b1.c Tue Oct 8 08:16:12 2002 *************** *** 16,20 **** {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti)}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO}, --- 16,20 ---- {"f_builtins", T_OBJECT, OFF(f_builtins),RO}, {"f_globals", T_OBJECT, OFF(f_globals), RO}, ! {"f_lasti", T_INT, OFF(f_lasti), RO}, {"f_lineno", T_INT, OFF(f_lineno), RO}, {"f_restricted",T_INT, OFF(f_restricted),RO},
A backwards diff! He posted a backwards diff! Aargh! :-)
Is this a reasonable suggestion for 2.3, or is it giving people too much rope? A nasty consequence is that you can write Python code that causes Python to seg-fault, but you have to be doing some fairly advanced stuff for that to happen. I would say that the power of a Set-next-statement feature in pdb and other debuggers is worth the price, but others may disagree...?
I can't approve a Python-level feature that explicitly opens up a loophole of this kind. But if you add a C "getset" wrapper that allows setting this only to safe values (i.e. within the same block and pointing at the beginning of a bytecode) it might be deemed safe enough. It should probably silently update f_lineno as well. Spare no cost. It's up to you to write the patch, submit it to SF, and prove that it's safe. --Guido van Rossum (home page: http://www.python.org/~guido/)
A backwards diff! He posted a backwards diff! Aargh! :-)
Oops, mental note well and truly made...
if you add a C "getset" wrapper that allows setting this only to safe values (i.e. within the same block and pointing at the beginning of a bytecode) it might be deemed safe enough. It should probably silently update f_lineno as well.
OK, I'll have a look at doing that, and at creating a suitably convincing test suite. -- Richie Hindle richie@entrian.com
participants (4)
-
Armin Rigo -
Guido van Rossum -
Richie Hindle -
richie＠entrian.com