[Python-Dev] variable name resolution in exec is incorrect

[Nick Coghlan]

On 27/05/10 10:38, Guido van Rossum wrote:
> On Wed, May 26, 2010 at 5:12 PM, Nick Coghlan<ncoghlan at gmail.com>  wrote:
>> Lexical scoping only works for code that is compiled as part of a single
>> operation - the separation between the compilation of the individual string
>> and the code defining that string means that the symbol table analysis
>> needed for lexical scoping can't cross the boundary.
>
> Hi Nick,
>
> I don't think Colin was asking for such things.

Yes, I realised some time after sending that message that I'd gone off 
on a tangent unrelated to the original question (as a result of earlier 
parts of the discussion I'd been pondering the scoping differences 
between exec with two namespaces and a class definition and ended up 
writing about that instead of the topic Colin originally brought up).

I suspect Thomas is right that the current two namespace exec behaviour 
is mostly a legacy of the standard scoping before nested scopes were added.

To state the problem as succinctly as I can, the basic issue is that a 
code object which includes a function definition that refers to top 
level variables will execute correctly when the same namespace is used 
for both locals and globals (i.e. like module level code) but will fail 
when these namespaces are different (i.e. like code in class definition).

So long as the code being executed doesn't define any functions that 
refer to top level variables in the executed code the two argument form 
is currently perfectly usable, so deprecating it would be an overreaction.

However, attaining the (sensible) behaviour Colin is requesting when 
such top level variable references exist would actually be somewhat 
tricky. Considering Guido's suggestion to treat two argument exec like a 
function rather than a class and generate a closure with full lexical 
scoping a little further, I don't believe this could be done in exec 
itself without breaking code that expects the current behaviour. 
However, something along these lines could probably be managed as a new 
compilation mode for compile() (e.g. compile(code_str, name, 
"closure")), which would then allow these code objects to be passed to 
exec to get the desired behaviour.

Compare and contrast:

 >>> def f():
...   x = 1
...   def g():
...     print x
...   g()
...
 >>> exec f.func_code in globals(), {}
1

 >>> source = """\
... x = 1
... def g():
...   print x
... g()
... """
 >>> exec source in globals(), {}
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<string>", line 4, in <module>
   File "<string>", line 3, in g
NameError: global name 'x' is not defined

Breaking out dis.dis on these examples is fairly enlightening, as they 
generate *very* different bytecode for the definition of g().

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------