Python3 compiled listcomp can't see local var - bug or feature?
Is this a bug or a feature? Consider the following program:
# TestProgram.py def Test(): # global x x = 1 exec(compile('print([x+1,x+2])', 'MyTest', 'exec')) exec(compile('print([x+i for i in range(1,3)])', 'MyTest', 'exec')) Test()
In Python 2.7.15 the output is
[2, 3] [2, 3]
In Python 3.6.5 the output is [2, 3] Traceback (most recent call last): File "TestProgram.py", line 7, in <module> Test() File "TestProgram.py", line 6, in Test exec(compile('print([x+i for i in range(1,3)])', 'MyTest', 'exec')) File "MyTest", line 1, in <module> File "MyTest", line 1, in <listcomp> NameError: name 'x' is not defined
If the "global x" declaration is uncommented, this "fixes" the Python 3.6.5 behaviour, i.e. no error occurs and the output is the same as for Python 2.7.15.
*In other words, it looks as if in Python 3.6.5, the compiled list comprehension** **can "see" a pre-existing global variable but not a local one.*
I have used dis to examine the code objects returned by compile() (they are the same with or without the "global x"):
Python 2.7.15 first code object from 'print([x+1,x+2])': 1 0 LOAD_NAME 0 (x) 3 LOAD_CONST 0 (1) 6 BINARY_ADD 7 LOAD_NAME 0 (x) 10 LOAD_CONST 1 (2) 13 BINARY_ADD 14 BUILD_LIST 2 17 PRINT_ITEM 18 PRINT_NEWLINE 19 LOAD_CONST 2 (None) 22 RETURN_VALUE Python 2.7.15 second code object from 'print([x+i for i in range(1,3)])': 1 0 BUILD_LIST 0 3 LOAD_NAME 0 (range) 6 LOAD_CONST 0 (1) 9 LOAD_CONST 1 (3) 12 CALL_FUNCTION 2 15 GET_ITER >> 16 FOR_ITER 16 (to 35) 19 STORE_NAME 1 (i) 22 LOAD_NAME 2 (x) 25 LOAD_NAME 1 (i) 28 BINARY_ADD 29 LIST_APPEND 2 32 JUMP_ABSOLUTE 16 >> 35 PRINT_ITEM 36 PRINT_NEWLINE 37 LOAD_CONST 2 (None) 40 RETURN_VALUE Python 3.6.5 first code object from 'print([x+1,x+2])': 1 0 LOAD_NAME 0 (print) 2 LOAD_NAME 1 (x) 4 LOAD_CONST 0 (1) 6 BINARY_ADD 8 LOAD_NAME 1 (x) 10 LOAD_CONST 1 (2) 12 BINARY_ADD 14 BUILD_LIST 2 16 CALL_FUNCTION 1 18 POP_TOP 20 LOAD_CONST 2 (None) 22 RETURN_VALUE Python 3.6.5 second code object from 'print([x+i for i in range(1,3)])': 1 0 LOAD_NAME 0 (print) 2 LOAD_CONST 0 (<code object <listcomp> at 0x00000000029F79C0, file "MyTest", line 1>) 4 LOAD_CONST 1 ('<listcomp>') 6 MAKE_FUNCTION 0 8 LOAD_NAME 1 (range) 10 LOAD_CONST 2 (1) 12 LOAD_CONST 3 (3) 14 CALL_FUNCTION 2 16 GET_ITER 18 CALL_FUNCTION 1 20 CALL_FUNCTION 1 22 POP_TOP 24 LOAD_CONST 4 (None) 26 RETURN_VALUE
You will see that in Python 3.6.5 the dis output for the second code object does not show the internals of the listcomp, and in particular whether, and how, it refers to the variable 'x'. I don't know how to investigate further.
Best wishes Rob Cliffe
On 6 June 2018 at 15:31, Rob Cliffe via Python-Dev python-dev@python.org wrote:
Is this a bug or a feature? Consider the following program:
# TestProgram.py def Test(): # global x x = 1 exec(compile('print([x+1,x+2])', 'MyTest', 'exec')) exec(compile('print([x+i for i in range(1,3)])', 'MyTest', 'exec')) Test()
In Python 2.7.15 the output is
[2, 3] [2, 3]
In Python 3.6.5 the output is [2, 3] Traceback (most recent call last): File "TestProgram.py", line 7, in <module> Test() File "TestProgram.py", line 6, in Test exec(compile('print([x+i for i in range(1,3)])', 'MyTest', 'exec')) File "MyTest", line 1, in <module> File "MyTest", line 1, in <listcomp> NameError: name 'x' is not defined
If the "global x" declaration is uncommented, this "fixes" the Python 3.6.5 behaviour, i.e. no error occurs and the output is the same as for Python 2.7.15.
*In other words, it looks as if in Python 3.6.5, the compiled list comprehension* *can "see" a pre-existing global variable but not a local one.*
Yes, this is expected behaviour - the two-namespace form of exec (which is what you get implicitly when you use it inside a function body) is similar to a class body, and hence nested functions (including the implicit ones created for comprehensions) can't see the top level local variables.
You can override that and force the use of the single-namespace form by passing the locals() namespace into exec() explicitly:
def explicit_local_namespace(): x = 1 exec(compile('print([x+i for i in range(1,3)])', 'MyTest', 'exec'), locals()) explicit_local_namespace()
(Note: you'll then need to use collections.ChainMap instead of separate locals and globals namespaces if you want the exec'ed code to still be able to see the module globals in addition to the function locals)
Cheers, Nick.
On 06/06/2018 03:51 PM, Nick Coghlan wrote:
On 6 June 2018 at 15:31, Rob Cliffe via Python-Dev <python-dev@python.org mailto:python-dev@python.org> wrote: ... *In other words, it looks as if in Python 3.6.5, the compiled list comprehension** **can "see" a pre-existing global variable but not a local one.*
Yes, this is expected behaviour - the two-namespace form of exec (which is what you get implicitly when you use it inside a function body) is similar to a class body, and hence nested functions (including the implicit ones created for comprehensions) can't see the top level local variables.
In issue 13557 [1] Amaury gives the following explanation by quoting the documentation [2] "Free variables are not resolved in the nearest enclosing namespace, but in the global namespace" and hints at the same solution that is proposed by Nick.
FWIW in issue 21161 [3] folks have been bitten by this when trying to run a list comprehension in pdb.
[1] https://bugs.python.org/issue13557 [2] http://docs.python.org/py3k/reference/executionmodel.html#interaction-with-d... [3] https://bugs.python.org/issue21161
Xavier
Is this a bug or a feature?
The bug was me being so excited about the new construct (I pushed in someone else's work, can't recall who now, maybe Fredrik Lundh?) that I didn't consider that leaking the loop variable out of the list comprehension was a bad idea. Think of the Py3 behavior as one of those "corrections" to things which were "got wrong" in Python 1 or 2. :-)
Skip
Skip, I think you have misunderstood the point I was making. It was not whether the loop variable should leak out of a list comprehension. Rather, it was whether a local variable should, so to speak, "leak into" a list comprehension. And the answer is: it depends on whether the code is executed normally, or via exec/eval. Example:
def Test(): x = 1 print([x+i for i in range(1,3)]) # Prints [2,3] exec('print([x+i for i in range(1,3)])') # Raises NameError (x) Test()
I (at least at first) found the difference in behaviour surprising.
Regards
Rob Cliffe
On 08/06/2018 19:27, Skip Montanaro wrote:
Is this a bug or a feature?
The bug was me being so excited about the new construct (I pushed in someone else's work, can't recall who now, maybe Fredrik Lundh?) that I didn't consider that leaking the loop variable out of the list comprehension was a bad idea. Think of the Py3 behavior as one of those "corrections" to things which were "got wrong" in Python 1 or 2. :-)
Skip
This email has been checked for viruses by AVG. http://www.avg.com
Skip, I think you have misunderstood the point I was making. It was not whether the loop variable should leak out of a list comprehension. Rather, it was whether a local variable should, so to speak, "leak into" a list comprehension. And the answer is: it depends on whether the code is executed normally, or via exec/eval.
Got it. Yes, you'll have to pass in locals to exec. (Can't verify, as I'm on the train, on my phone.) Builtins like range are global to everything, so no problem there.
Your clarification also make it more of a Python programming question , I think.
Skip
Skip Montanaro wrote:
Yes, you'll have to pass in locals to exec.
Exec changed between python 2 and 3. It used to be treated specially by the compiler so that it could see and modify the locals where it was used. But now it's just an ordinary function, so you can't expect it to magically know about anything that's not passed into it.
On Mon, Jun 11, 2018 at 3:10 PM Rob Cliffe via Python-Dev < python-dev@python.org> wrote:
Skip, I think you have misunderstood the point I was making. It was not whether the loop variable should leak out of a list comprehension. Rather, it was whether a local variable should, so to speak, "leak into" a list comprehension. And the answer is: it depends on whether the code is executed normally, or via exec/eval. Example:
def Test(): x = 1 print([x+i for i in range(1,3)]) # Prints [2,3] exec('print([x+i for i in range(1,3)])') # Raises NameError (x) Test()
I (at least at first) found the difference in behaviour surprising.
Change 'def' to 'class' and run it again. You'll be even more surprised.
Ah yes, I see what you mean:
class Test(): x = 1 print (x) # Prints 1 print([x+i for i in range(1,3)]) # NameError (x)
Anyway, I apologise for posting to Python-Dev on was a known issue, and turned out to be more me asking for help with development with Python, rather than development of Python. (My original use case was a scripting language that could contain embedded Python code). Thanks to Nick for his original answer.
Rob Cliffe
On 11/06/2018 23:31, Eric Fahlgren wrote:
On Mon, Jun 11, 2018 at 3:10 PM Rob Cliffe via Python-Dev <python-dev@python.org mailto:python-dev@python.org> wrote:
Skip, I think you have misunderstood the point I was making. It was not whether the loop variable should leak out of a list comprehension. Rather, it was whether a local variable should, so to speak, "leak into" a list comprehension. And the answer is: it depends on whether the code is executed normally, or via exec/eval. Example: def Test(): x = 1 print([x+i for i in range(1,3)]) # Prints [2,3] exec('print([x+i for i in range(1,3)])') # Raises NameError (x) Test() I (at least at first) found the difference in behaviour surprising.Change 'def' to 'class' and run it again. You'll be even more surprised.
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
participants (6)
-
Eric Fahlgren -
Greg Ewing -
Nick Coghlan -
Rob Cliffe -
Skip Montanaro -
Xavier de Gaye