On Mon, Mar 25, 2013 at 8:13 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com>wrote:
2013/3/25 anatoly techtonik <techtonik@gmail.com>
This module opened a Pandora box of Python internals. Version 0.4 still fails to trace files specified on command line, and I am lost in internal details of execfile + locals()/globals()/namespacing/scoping. Python tracker doesn't help here.
Seriously: always always pass your own locals and globals to functions like exec and execfile. Good luck.
I'd like to understand what's going on there, because it directly affects if I will be able to extend xtrace further and troubleshoot reported problems when more errors appear. I know that locals() is not an ordinary dictionary ( http://bugs.python.org/issue17546). And the first question - what should I pass to execfile as 'my own locals'? 1. an exact livedict that is returned by locals() function 2. my own dict, which will become live inside the execfile call 3. locals function that returns a reference to livedict The second question - why should I do this? To make the explanation easier, here is a real world problem that made me seek the answers. https://bitbucket.org/techtonik/xtrace/issue/3/unable-to-xtrace-indefpy The file that execfile() *inside* xtrace fails to execute is below. I named it indef.py, because subprocess call is inside defined function. import subprocess def ret(): print subprocess.check_output(['hg', 'id', '-nib']) ret() The problem repeats if the file is executed from console, but only in special case: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] >>> execfile('indef.py') 0cef574f7f62+ 23+ default >>> def x(): ... execfile('indef.py') ... >>> x() 0cef574f7f62+ 23+ default Now I restart the console and do the same stuff, but without the first call: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] >>> def x(): ... execfile('indef.py') ... >>> x() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in x File "indef.py", line 6, in <module> ret() File "indef.py", line 4, in ret print subprocess.check_output(['hg', 'id', '-nib']) NameError: global name 'subprocess' is not defined I guess the same stuff happens with my xtrace(), where execfile() is executed inside the function. Now the questions: 1. I understand that when Python starts executing code, it creates a namespace. This is the 'first_namespace' where Python starts to store all variables encountered. I intentionally do not call it global, as this term is not introduced at this point. Python continues to put and change variables there until it encounters [something], this [something] causes it to create a 'new_namespace' to store variables. I know that a function call makes Python create 'new_namespace' to store variables encountered inside this function, so "function call" is in [something]. What else is in [something], i.e. what else causes Python to create 'new_namespaces'? It will make me comfortable to see the full list described somewhere. 2. 'new_namespace's are created when function is entered and destroyed when it exits. When variable is requested, the lookup mechanism looks first inside 'new_namespace', then checks its parent (caller namespace), then parent of the parent and so on until the 'first_namespace'. Does it work like this? 3. Now the problem. When I run execfile() without parameters, I expect the code inside to be 'virtualized' - isolated from parent process, like in LXC or VirtualBox, but on Python level. I also expect that if I want to communicate with this code inside execfile(), I craft my own 'first_namespace' and pass it down. When execfile() returns, I inspect the state of this modified namespace, and maybe import some results back into my 'current_namespace'. The example with indef.py above shows the execfile() doesn't work this way. So the question is - why execfile() doesn't it work the way I described? 4. At this point I hope to reach the state there it is clear how execfile() works in reality, and the next question in this state - what was the reason to make it so complicated? Optimization? Thanks for the feedback. -- anatoly t.