List comprehension - NameError: name '_[1]' is not defined ?

[Peter Otten]

Peter Otten wrote:

> mario ruggier wrote:
> 
>> Hello,
>> 
>> I would like to evaluate list comprehension expressions, from within
>> which I'd like to call a function. For a first level it works fine but
>> for second level it seems to lose the "_[1]" variable it uses
>> internally to accumulate the results. Some sample code is:
>> 
>> class GetItemEvaluator(object):
>>     def __init__(self):
>>         self.globals = globals() # some dict (never changes)
>>         self.globals["ts"] = self.ts
>>         self.globals["join"] = "".join
>>         self.locals = {} # changes on each evaluation
>>     def __getitem__(self, expr):
>>         return eval(expr, self.globals, self.locals)
>>     def ts(self, ts, name, value):
>>         self.locals[name] = value
>>         #print ts, name, value, "::::", self.locals, "::::", ts % self
>>         return ts % self
>> 
>> gie = GetItemEvaluator()
>> gie.locals["inner"] = ("a","b","c","d")
>> print """
>> pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
>> (inner)]))s post
>> """ % gie
>> # OK, outputs: pre 0.a 1.b 2.c 3.d  post
>> 
>> gie = GetItemEvaluator()
>> gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
>> print """
>> pre %(join([ts(
>>   '''inner pre
>>   %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
>> (inner)]))s
>>    inner post''',
>>   "inner", inner) # END CALL outer ts()
>>   for i,inner in enumerate(outer)])
>> )s post
>> """ % gie
>> 
>> The second 2-level comprehension gives:
>> 
>>   File "scratch/eval_test.py", line 8, in __getitem__
>>     return eval(expr, self.globals, self.locals)
>>   File "<string>", line 4, in <module>
>> NameError: name '_[1]' is not defined
>> 
>> If the print was to be enable, the last line printed out is:
>> 
>> 0.3.%(k)s  k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
>> 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
>> '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p
>> 
>> i.e. it has correctly processed the first inner sequence, until the
>> (last) "p" element. But on exit of the last inner ts() call, it seems
>> to lose the '_[1]' on self.locals.
>> 
>> Any ideas why?
>> 
>> Note, i'd like that the first parameter to ts() is as independent as
>> possible from teh context in expression context, a sort of independent
>> mini-template. Thus, the i,j enumerate counters would normally not be
>> subbed *within* the comprehension itself, but in a similar way to how
>> k is evaluated, within the call to ts() -- I added them this way here
>> to help follow easier what the execution trail is. Anyhow, within that
>> mini-template, i'd like to embed other expressions for the % operator,
>> and that may of course also be list comprehensions.
> 
> I have no idea what you are trying to do. Please reread the Zen of Python
> ;)
> 
> What happens is:
> 
> List comprehensions delete the helper variable after completion:
> 
>>>> def f(): [i for i in [1]]
> ...
>>>> dis.dis(f)
>   1           0 BUILD_LIST               0
>               3 DUP_TOP
>               4 STORE_FAST               0 (_[1])
>               7 LOAD_CONST               1 (1)
>              10 BUILD_LIST               1
>              13 GET_ITER
>         >>   14 FOR_ITER                13 (to 30)
>              17 STORE_FAST               1 (i)
>              20 LOAD_FAST                0 (_[1])
>              23 LOAD_FAST                1 (i)
>              26 LIST_APPEND
>              27 JUMP_ABSOLUTE           14
>         >>   30 DELETE_FAST              0 (_[1])
>              33 POP_TOP
>              34 LOAD_CONST               0 (None)
>              37 RETURN_VALUE
> 
> If you manage to run two nested listcomps in the same namespace you get a
> name clash and the inner helper variable overwrites/deletes the outer:
> 
>>>> def xeval(x): return eval(x, ns)
> ...
>>>> ns = dict(xeval=xeval)
>>>> xeval("[xeval('[k for k in ()]') for i in (1,)]")
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 1, in xeval
>   File "<string>", line 1, in <module>
> NameError: name '_[1]' is not defined
> 
> Peter
> 

I'd like to add: this can only happen because the code snippets are compiled
independently. Otherwise Python uses different names for each listcomp:

>>> def f():
...     [i for i in ()]
...     [i for i in ()]
...
>>> f.func_code.co_varnames
('_[1]', 'i', '_[2]')

Peter