[ python-Bugs-1153622 ] eval does not bind variables in lambda bodies correctly

[SourceForge.net]

Bugs item #1153622, was opened at 2005-02-28 11:48
Message generated for change (Comment added) made by tjreedy
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1153622&group_id=5470

Category: Parser/Compiler
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Mattias Engdegård (yorick)
Assigned to: Nobody/Anonymous (nobody)
Summary: eval does not bind variables in lambda bodies correctly

Initial Comment:
eval() does not bind variables in lambda expressions
correctly:

>>>def f(g): return eval('lambda x: g(x)')
>>>f(lambda y: y * 2)(17)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 1, in <lambda>
NameError: global name 'g' is not defined

The docs say this about eval():

# If both dictionaries are omitted, the expression is
# executed in the environment where eval is called.

and using plain local variables work as expected:

>>>def h(d): return eval('d(10)')
>>>h(lambda y: y * 2)
20

Also, if locals() is presented as the global dict to
eval(), it works:

>>>def f(g): return eval('lambda x: g(x)', locals(),
locals())
>>>f(lambda y: y * 2)(17)
34

but this does not allow the expression to reference
global variables of course.


----------------------------------------------------------------------

Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-02 17:59

Message:
Logged In: YES 
user_id=593130

No, this issue is not specific to either eval or lambda:

>>> def f(g):
...   exec 'def h(x): return g(x)'
...   return h
...
>>> f(lambda y: y * 2)(17)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 1, in h
NameError: global name 'g' is not defined

It is specific to creating a function at top-level in a separate 
execution environment, as done, by design, by both eval and 
exec, and with either a def statement or lambda abbreviation 
thereof.

In Python, lexical scoping works because nested functions are 
compiled along with the outer function, so that scoped variables 
can be identified and both functions adjusted so that the coupling 
works.  In particular, the scoped variable has to not be deleted 
when the outer function returns.  Eval/exec compile their string 
only later, when the function is called.

"it works that way because it is the way it works".
Those are your words, not mine.  If you want Python to work 
differently, write a PEP or a patch, or raise the question in the 
newsgroup/mailing list. I'm done discussing it here.

----------------------------------------------------------------------

Comment By: Mattias Engdegård (yorick)
Date: 2005-03-02 11:42

Message:
Logged In: YES 
user_id=432579

No, this issue is specific to eval of lambda expressions.
Please read my problem description. Please refer to the
Python documentation if you are confused with how standard
function declaration or lexical scoping works.


----------------------------------------------------------------------

Comment By: Branko (bbange)
Date: 2005-03-01 19:20

Message:
Logged In: YES 
user_id=1230541

Obviously I forgot a statement in my previous comment's 
code example. So here's the complete version:

n=2
def f(x): return n*x
del n
f(2) 
# the Python implementation will result in a name error here. 
But what should happen if Python had bound variable n at the 
time of f's definitionf?
# let's define n again
n=3
f(2)
# the implementation will return 6, but how about your 
expected implementation?


----------------------------------------------------------------------

Comment By: Branko (bbange)
Date: 2005-03-01 19:15

Message:
Logged In: YES 
user_id=1230541

I think this issue is not special for eval and can be also 
reproduced with a def statement. The point is that at function 
definition time Python does not do any variable binding 
concerning variables not local to the function. Instead Python 
looks for that variable in the namespace of the module in 
which the function was created at the time the function is 
executed. Python determines that module by evaluating the 
variable __module__ at function definition time and 
remembers it by setting the function attribute with the same 
name. That's why only the variable __module__ is relevant at 
function definition time. Simply put, Python does only do a 
module level variable binding at function definition time. This 
is simple and sensible. If you don't agree consider this:

n=2
def f(x): return n*x
del n
f(2) 
# the Python implementation will result in a name error here. 
But what should happen if Python had bound variable n at the 
time of f's definitionf?
# let's define n again
f(2)
# the implementation will return 6, but how about your 
expected implementation?

As you see, changing the implementation would either make 
Pythons semantics more complicated or would remove much 
of Pythons dynanism.




----------------------------------------------------------------------

Comment By: Mattias Engdegård (yorick)
Date: 2005-03-01 13:26

Message:
Logged In: YES 
user_id=432579

What you are saying is "it works that way because it is the
way it works". I see no reason at all for this odd behaviour
other than bug-compatibility. I find nothing at all in the
documentation supporting this behaviour either; please
inform me if  I have missed something.

All other languages supporting eval and lexical scoping
(Lisp, Scheme, Perl, Ruby, etc) work in the expected way. I
have no problems if Python wants to be different for
whatever reason, but it should be documented.

I did a quick Google in comp.lang.python but could not find
anything that supported this "exception" or gave a rational
explanation. Kindly direct me to any resource you know of
that could help enlighten me on this issue.

>#  From your comments, I suspect you expect 0.

Of course not. I know very well how lexical scoping works,
so please don't put words in my mouth.

None of your examples have anything to do with scoping. As
we both know, it is not the _values_ of the variables that
is important for variable binding, it is their identity;
which variable is chosen, not what they happen to contain at
the time the lambda expression is evaluated.


----------------------------------------------------------------------

Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-01 12:29

Message:
Logged In: YES 
user_id=593130

Whoops.  eval('x') ==  x as code snippets has an exception, 
which is the one tripping you up.  When the eval is within a 
function definition (and lambda expressions are abbreviated 
simple function definitions) and 'x' contains a function definition, 
then the body of the contained function definition does not have 
access, when it is run, to the locals of the containing function 
(the lexical scope), whereas it will when x is compiled directly *as 
part of the containing function body*.  eval('x') removes x from 
that part of its context.  eval only has the simple two-level 
globals/locals environment, which can be anything the caller 
passes in, so it compiles x as if it were top-level code.  Hence 
free variables in contained functions are looked up in the global 
passed to  eval when the evaled function is called.

This issue has been discussed on the Python newsgroup/mailing 
list more than once.  If my explanation is not clear, you might be 
able to find others in Google c.l.p archives.  Do consider that 
core functions which have been debugged for over a decade are 
unlike to have many bugs left, although the docs are still being 
improved.

While Python's scoping is lexical, its free variable binding is late.
Consider
>>> def f():
...   x = 0
...   def g(): print x
...   x = 1
...   return g
...
>>> f()()
# What gets printed? 0 or 1?
#  From your comments, I suspect you expect 0.
#  Irregardless, it is
1

Similarly
>>> f()()
1
>>> d={'x': 0}
>>> h=eval('lambda: x', d, d)
>>> h()
0
>>> d['x'] = 1
>>> h()
# now what gets printed?
1


----------------------------------------------------------------------

Comment By: Mattias Engdegård (yorick)
Date: 2005-03-01 04:11

Message:
Logged In: YES 
user_id=432579

>Variables in Python functions are resolved 
>when the function is *called*, not when it is defined.

I'm not sure what you mean by that, since Python obeys
lexical scoping, not dynamic.Consider:

def f(x): lambda y: x + y

When the inner lambda expression above is evaluated, x
inside the lambda body is bound to the parameter of the call
of f, even if x+y is not evaluated until that function is
called.
So since

def f(x): return eval('x')

fetches its definition of x from the lexical variable x, why
shouldn't

def f(g): return eval('lambda x: g(x)')

fetch its definition of g from the lexical variable g? A
lambda expression is just a way of delaying evaluation,
*not* delaying how variables are bound --- this is done
immediately.


----------------------------------------------------------------------

Comment By: Terry J. Reedy (tjreedy)
Date: 2005-03-01 00:30

Message:
Logged In: YES 
user_id=593130

I am 99.5% sure that this is 'invalid' (a Resolution category) and 
should be closed.  With the default environment, eval('x') is the 
same as unquoted x.  Variables in Python functions are resolved 
when the function is *called*, not when it is defined.  There is no 
resolution for g in the default globals.  Eval does not change this.  
The NameError is exactly correct.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1153622&group_id=5470