Can I introspect/reflect to get arguments exec()?
[asked on comp.lang.python but no takers. So I'm bumping it up a notch.] I have ported my Python debugger pydbgr to Python3. See [1] or [2]. Inside the debugger, when there is an exec() somewhere in the call stack, I'd like to be able to retrieve the string parameter. With this, the debugger can show part of the string in a call stack. Or it can show the text when the frame is set to that exec() frame. Going further, the debugger could write the exec string out to a temporary file. And when reporting locations, it could report not just something like "<string> line 4", but also give that temporary file name which a front-end could use as well. So consider this code using inspect.getargvalues() and inspect.currentframe(): import inspect def my_exec(string): show_args(inspect.currentframe()) # simulate exec(string) def show_args(frame): print(inspect.getargvalues(frame)) my_exec("show_args(inspect.currentframe())") exec("show_args(inspect.currentframe())") When run this is the output: python3 exec-args.py ArgInfo(args=['string'], varargs=None, keywords=None, locals={'string': 'show_args(inspect.currentframe())'}) ArgInfo(args=[], varargs=None, keywords=None, locals={'my_exec': <function my_exec at 0xb6f828ec>,, ... In a different setting, CPython byte-code assembly that gets generated for running exec() is: 25 88 LOAD_GLOBAL 10 (exec) 91 LOAD_CONST 4 ('show_args(inspect.currentframe())') --> 94 CALL_FUNCTION 1 97 POP_TOP What's going on? Also, I have the same question for CPython 2.6 or Python 2.7. Thanks! [1] http://code.google.com/p/pydbgr/ [2] http://code.google.com/p/python3-trepan/
On Tue, Mar 26, 2013 at 10:18 PM, Benjamin Peterson <benjamin@python.org>wrote:
Okay. But is the string is still somewhere in the CPython VM stack? (The result of LOAD_CONST 4 above). Is there a way to pick it up from there? At the point that we are stopped the exec action hasn't taken place yet.
On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein <rocky@gnu.org> wrote:
Okay. But is the string is still somewhere in the CPython VM stack? (The result of LOAD_CONST 4 above). Is there a way to pick it up from there?
Maybe using C you could peek into the frame's value stack, but that's not exposed to any Python API I know of. But that still doesn't help you, because the value will be removed from the stack before exec() is actually called, which means if you go looking for it in code called from the exec (e.g. the call event itself), you aren't going to see the data.
At the point that we are stopped the exec action hasn't taken place yet.
That doesn't help if you're using line-level tracing events. At the beginning of the line, the data's not on the call stack yet, and by the time you enter the frame of the code being exec'd, it'll be off the stack again. Basically, there is no way to do what you're asking for, short of replacing the built-in exec function with your own version. And it still won't help you with stepping through the source of functions that are *compiled* using an exec() or compile(), or any other way of ending up with dynamically-generated code you want to debug. (Unless you use something like DecoratorTools to generate it, that is -- DecoratorTools has some facilities for caching dynamically-generated code so that it works properly with debuggers. But that has to be done by the code doing the generation, not the debugger. If the code generator uses DecoratorTools' caching support, then any debugger that uses the linecache module will Just Work. It might be nice for the stdlib should have something like this, but you could also potentially fake it by replacing the builtin eval, exec, compile, etc. functions w/versions that cache the source.)
Thank you for your very thoughtful and detailed explanation of what is going on and for your considerations as to how to get this done (as opposed to why it *can't *be done). It looks like DecoratorTools or more likely a customized version of it is the way to go! (The important info is above. The rest are just some geeky details) You see, there are some places where additional care may be needed in my setting. The debuggers I write use sometimes don't use just the getline module but also use my own pyficache module. I want to also cache things like file stat() info, provide a SHA1 for the text of the file, and provide colorized syntax-highlighted versions of the text when desired. But since I control pyficache, I can mirror the changes made to getline. Of course the debugger uses sys.settrace too, so the evil-ness of that is definitely not a concern. But possibly I need to make sure that since the DecoratorTools and the debugger both hook into trace hooks they play nice together and fire in the right order. And for that I created another module called tracer() which takes into account that one might want to specify priorities in the chain hook order, and that one might want a to filter out (i.e. ignore) certain calls to the hook function for specific hooks. It may be a while before I seriously get to this, but again, it is good to have in mind an approach to take. So thanks again. On Thu, Mar 28, 2013 at 1:45 AM, PJ Eby <pje@telecommunity.com> wrote:
On Thu, Mar 28, 2013 at 6:43 AM, Rocky Bernstein <rocky@gnu.org> wrote:
DecoratorTools' trace hooking is unrelated to its linecache functionality. All you need from it is the cache_source() function; you can pretty much ignore everything else for your purposes. You'll just need to give it a phony filename to work with, and the associated string.
On Tue, Mar 26, 2013 at 10:18 PM, Benjamin Peterson <benjamin@python.org>wrote:
Okay. But is the string is still somewhere in the CPython VM stack? (The result of LOAD_CONST 4 above). Is there a way to pick it up from there? At the point that we are stopped the exec action hasn't taken place yet.
On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein <rocky@gnu.org> wrote:
Okay. But is the string is still somewhere in the CPython VM stack? (The result of LOAD_CONST 4 above). Is there a way to pick it up from there?
Maybe using C you could peek into the frame's value stack, but that's not exposed to any Python API I know of. But that still doesn't help you, because the value will be removed from the stack before exec() is actually called, which means if you go looking for it in code called from the exec (e.g. the call event itself), you aren't going to see the data.
At the point that we are stopped the exec action hasn't taken place yet.
That doesn't help if you're using line-level tracing events. At the beginning of the line, the data's not on the call stack yet, and by the time you enter the frame of the code being exec'd, it'll be off the stack again. Basically, there is no way to do what you're asking for, short of replacing the built-in exec function with your own version. And it still won't help you with stepping through the source of functions that are *compiled* using an exec() or compile(), or any other way of ending up with dynamically-generated code you want to debug. (Unless you use something like DecoratorTools to generate it, that is -- DecoratorTools has some facilities for caching dynamically-generated code so that it works properly with debuggers. But that has to be done by the code doing the generation, not the debugger. If the code generator uses DecoratorTools' caching support, then any debugger that uses the linecache module will Just Work. It might be nice for the stdlib should have something like this, but you could also potentially fake it by replacing the builtin eval, exec, compile, etc. functions w/versions that cache the source.)
Thank you for your very thoughtful and detailed explanation of what is going on and for your considerations as to how to get this done (as opposed to why it *can't *be done). It looks like DecoratorTools or more likely a customized version of it is the way to go! (The important info is above. The rest are just some geeky details) You see, there are some places where additional care may be needed in my setting. The debuggers I write use sometimes don't use just the getline module but also use my own pyficache module. I want to also cache things like file stat() info, provide a SHA1 for the text of the file, and provide colorized syntax-highlighted versions of the text when desired. But since I control pyficache, I can mirror the changes made to getline. Of course the debugger uses sys.settrace too, so the evil-ness of that is definitely not a concern. But possibly I need to make sure that since the DecoratorTools and the debugger both hook into trace hooks they play nice together and fire in the right order. And for that I created another module called tracer() which takes into account that one might want to specify priorities in the chain hook order, and that one might want a to filter out (i.e. ignore) certain calls to the hook function for specific hooks. It may be a while before I seriously get to this, but again, it is good to have in mind an approach to take. So thanks again. On Thu, Mar 28, 2013 at 1:45 AM, PJ Eby <pje@telecommunity.com> wrote:
On Thu, Mar 28, 2013 at 6:43 AM, Rocky Bernstein <rocky@gnu.org> wrote:
DecoratorTools' trace hooking is unrelated to its linecache functionality. All you need from it is the cache_source() function; you can pretty much ignore everything else for your purposes. You'll just need to give it a phony filename to work with, and the associated string.
participants (3)
-
Benjamin Peterson -
PJ Eby -
Rocky Bernstein