On Wed, Mar 20, 2013 at 5:28 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

Hi,

I've created a module to dump function trace during execution of Python script. You can see session example in attachment. The format is of PHP Xdebug tool [2] just because I had some scripts from the past to analyze it.

The module uses sys.settrace() to analyse frames in 'call' events with callback(frame, event, arg).

Recently I've got an anonymous report that some frame misses filename information when run under IDE:

https://bitbucket.org/techtonik/xtrace/issue/2/use-of-xtrace-module-within-i...

There is unlikely to be any more feedback from the user to make me understand and reproduce the behavior, nor there is a sufficient documentation in sys.trace description [1]. So I am on my own. But I can not read C code the CPython is written in, so I ask here.

What are possible execution contexts for Python? How each execution context affects data structure that is passed to trace function (frame and arg)?

It's simpler than that: when running from an interactive session, f_back is empty...

...

...

...

def trace(frame, event, arg): ... funcname = frame.f_code.co_name ... if funcname == '<module>': ... print frame.f_back.f_code.co_filename ... import sys sys.settrace(trace) 1 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in trace AttributeError: 'NoneType' object has no attribute 'f_code'

Thanks a lot. =) User said he was using module from the IDE and at first I thought it wasn't possible to use module without previous stack frame (f_back), but I was wrong:

py -2 -s -S Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32

...

...

import xtrace xtrace.start() TRACE START [2013-03-20 18:49:27] z = 0 -> decode() C:\Python27\lib\encodings\cp437.py:14 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "xtrace.py", line 66, in function_trace_xdebug filename = self._strip_cwd(frame.f_back.f_code.co_filename) AttributeError: 'NoneType' object has no attribute 'f_code'

On the way I've discovered that my Python is hacked. Without -s -S key it executes modules from local temp directory. ... -> ensure_unicode() c:\users\user\appdata\local\temp\easy_install-k8gvbp\pyreadline-1.7.1-py2.7-win32.egg.tmp\pyre dline\unicode_helper.py:20 ... So there are absolutely no differences in running code in these execution contexts? - code runs from a file - code runs from a interactive console I realize that I don't know how to set trace function for external file, so that when the file is executed, the first frame won't have the previous stack frame. I do it like so: start() execfile(script) stop() -- anatoly t.

On Wed, Mar 20, 2013 at 7:36 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

On Wed, Mar 20, 2013 at 5:28 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com

...

wrote:

...

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

Hi,

I've created a module to dump function trace during execution of Python script. You can see session example in attachment. The format is of PHP Xdebug tool [2] just because I had some scripts from the past to analyze it.

The module uses sys.settrace() to analyse frames in 'call' events with callback(frame, event, arg).

Recently I've got an anonymous report that some frame misses filename information when run under IDE:

https://bitbucket.org/techtonik/xtrace/issue/2/use-of-xtrace-module-within-i...

There is unlikely to be any more feedback from the user to make me understand and reproduce the behavior, nor there is a sufficient documentation in sys.trace description [1]. So I am on my own. But I can not read C code the CPython is written in, so I ask here.

What are possible execution contexts for Python? How each execution context affects data structure that is passed to trace function (frame and arg)?

It's simpler than that: when running from an interactive session, f_back is empty...

...

...

...

def trace(frame, event, arg): ... funcname = frame.f_code.co_name ... if funcname == '<module>': ... print frame.f_back.f_code.co_filename ... import sys sys.settrace(trace) 1 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in trace AttributeError: 'NoneType' object has no attribute 'f_code'

Thanks a lot. =)

User said he was using module from the IDE and at first I thought it wasn't possible to use module without previous stack frame (f_back), but I was wrong:

...

py -2 -s -S Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32

...

...

import xtrace xtrace.start() TRACE START [2013-03-20 18:49:27] z = 0 -> decode() C:\Python27\lib\encodings\cp437.py:14 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "xtrace.py", line 66, in function_trace_xdebug filename = self._strip_cwd(frame.f_back.f_code.co_filename) AttributeError: 'NoneType' object has no attribute 'f_code'

On the way I've discovered that my Python is hacked. Without -s -S key it executes modules from local temp directory. ... -> ensure_unicode() c:\users\user\appdata\local\temp\easy_install-k8gvbp\pyreadline-1.7.1-py2.7-win32.egg.tmp\pyre dline\unicode_helper.py:20 ...

So there are absolutely no differences in running code in these execution contexts? - code runs from a file - code runs from a interactive console

Of course there is a difference; the code below works run from a file, but not when pasted into the interactive console:

def trace(frame, event, arg): funcname = frame.f_code.co_name print frame.f_code, repr(frame.f_code.co_code), event, arg if funcname == '<module>': print " module", repr(frame.f_back.f_code.co_code)

import sys sys.settrace(trace) exec '1 + 1'

For console it pretty clear why no previous frame exists - stack is resolved at >>> prompt. But what is this top frame that appears inside file context? The following code gives: <module> 7 <module> 1 def trace(frame, event, arg): print frame.f_back.f_code.co_name, frame.f_back.f_lineno print frame.f_code.co_name, frame.f_lineno import sys sys.settrace(trace) exec '1 + 1' How to identify the frames here? How come that 7th line of module is executed before 1 st? I expected to have module name available somewhere for <module> object. All right, this kind of tricky - '1 +1' a separate module, right? It is anonymous and doesn't have a name. Still is it possible to identify it somehow? For printing in function traces.

...

I realize that I don't know how to set trace function for external file, so that when the file is executed, the first frame won't have the previous stack frame. I do it like so: start() execfile(script) stop()

I'ts indeed difficult to detach the "script" from the frame running the execfile statement. You could handle this case specially in your trace function, though.

Now that I understand, I can handle it. I guess stripping the 0 level calls (where it will be only one) will help. Thanks again.

On Wed, Mar 20, 2013 at 8:21 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

On Wed, Mar 20, 2013 at 7:36 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com

...

wrote:

...

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

On Wed, Mar 20, 2013 at 5:28 PM, Amaury Forgeot d'Arc < amauryfa@gmail.com> wrote:

...

2013/3/20 anatoly techtonik <techtonik@gmail.com>

...

Hi,

I've created a module to dump function trace during execution of Python script. You can see session example in attachment. The format is of PHP Xdebug tool [2] just because I had some scripts from the past to analyze it.

The module uses sys.settrace() to analyse frames in 'call' events with callback(frame, event, arg).

Recently I've got an anonymous report that some frame misses filename information when run under IDE:

https://bitbucket.org/techtonik/xtrace/issue/2/use-of-xtrace-module-within-i...

There is unlikely to be any more feedback from the user to make me understand and reproduce the behavior, nor there is a sufficient documentation in sys.trace description [1]. So I am on my own. But I can not read C code the CPython is written in, so I ask here.

What are possible execution contexts for Python? How each execution context affects data structure that is passed to trace function (frame and arg)?

It's simpler than that: when running from an interactive session, f_back is empty...

...

>> def trace(frame, event, arg): ... funcname = frame.f_code.co_name ... if funcname == '<module>': ... print frame.f_back.f_code.co_filename ... >> import sys >> sys.settrace(trace) >> 1 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in trace AttributeError: 'NoneType' object has no attribute 'f_code'

Thanks a lot. =)

User said he was using module from the IDE and at first I thought it wasn't possible to use module without previous stack frame (f_back), but I was wrong:

...

py -2 -s -S Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32

...

> import xtrace > xtrace.start() TRACE START [2013-03-20 18:49:27] > z = 0 -> decode() C:\Python27\lib\encodings\cp437.py:14 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "xtrace.py", line 66, in function_trace_xdebug filename = self._strip_cwd(frame.f_back.f_code.co_filename) AttributeError: 'NoneType' object has no attribute 'f_code'

On the way I've discovered that my Python is hacked. Without -s -S key it executes modules from local temp directory. ... -> ensure_unicode() c:\users\user\appdata\local\temp\easy_install-k8gvbp\pyreadline-1.7.1-py2.7-win32.egg.tmp\pyre dline\unicode_helper.py:20 ...

So there are absolutely no differences in running code in these execution contexts? - code runs from a file - code runs from a interactive console

Of course there is a difference; the code below works run from a file, but not when pasted into the interactive console:

def trace(frame, event, arg): funcname = frame.f_code.co_name print frame.f_code, repr(frame.f_code.co_code), event, arg if funcname == '<module>': print " module", repr(frame.f_back.f_code.co_code)

import sys sys.settrace(trace) exec '1 + 1'

For console it pretty clear why no previous frame exists - stack is resolved at >>> prompt. But what is this top frame that appears inside file context? The following code gives: <module> 7 <module> 1

def trace(frame, event, arg): print frame.f_back.f_code.co_name, frame.f_back.f_lineno print frame.f_code.co_name, frame.f_lineno

import sys sys.settrace(trace) exec '1 + 1'

How to identify the frames here? How come that 7th line of module is executed before 1 st? I expected to have module name available somewhere for <module> object. All right, this kind of tricky - '1 +1' a separate module, right? It is anonymous and doesn't have a name. Still is it possible to identify it somehow? For printing in function traces.

An exception traceback gives you the answer; with the script foo.py below:

def f(): exec 'x + 1' f()

I get:

Traceback (most recent call last): File "/tmp/foo.py", line 4, in <module> f() File "/tmp/foo.py", line 2, in f exec 'x + 1' File "<string>", line 1, in <module> NameError: name 'x' is not defined

Do you see all the information you need here? If yes, it's possible to extract it from the current frames.

Not really. It got too far from original user story about execution contexts, so I need to choose a different entry point to continue discussion. What I need is the graph of state machine of Python execution. I want to inspect possible starting points, from which select which are visible to the user in which cases. Still, thanks to the pointer. I fixed the .f_back exception in version 0.3 of xtrace, and added a separate function - beep() that allows extending xtrace with your own stuff for inspecting execution process. This module opened a Pandora box of Python internals. Version 0.4 still fails to trace files specified on command line, and I am lost in internal details of execfile + locals()/globals()/namespacing/scoping. Python tracker doesn't help here. -- anatoly t.