problem with genexp

There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
generates "NameError: name 'i' is not defined" when run because: 2 0 LOAD_GLOBAL 0 (foo) 3 LOAD_CONST 1 ('a') 6 LOAD_GLOBAL 1 (i) 9 CALL_FUNCTION 256 12 POP_TOP 13 LOAD_CONST 0 (None) 16 RETURN_VALUE If you add parens around the code: foo(a = i for i in range(10)) You get something quite different: 2 0 LOAD_GLOBAL 0 (foo) 3 LOAD_CONST 1 ('a') 6 LOAD_CONST 2 (<code object <generator expression> at 0x2a960baae8, file "<stdin>", line 2>) 9 MAKE_FUNCTION 0 12 LOAD_GLOBAL 1 (range) 15 LOAD_CONST 3 (10) 18 CALL_FUNCTION 1 21 GET_ITER 22 CALL_FUNCTION 1 25 CALL_FUNCTION 256 28 POP_TOP 29 LOAD_CONST 0 (None) 32 RETURN_VALUE I agree with the bug report that the code should either raise a SyntaxError or do the right thing. n

Neal Norwitz wrote:
I agree it should be a SyntaxError - I believe the AST compiler actually raises one in this situation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

On 10/11/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
Could someone add a test for this on the AST branch? BTW, it looks like doctest is the way to go for SyntaxError tests. There are older tests, like test_scope.py, that use separate files with bad syntax (and lots of extra kludges in the infrastructure to ignore the fact that those .py files can't be compiled). Jeremy

Nick Coghlan wrote:
I was half right. Both the normal compiler and the AST compiler give a SyntaxError if you write: foo((a=i for i in range(10))) The problem is definitely on the parser end though: Py> compiler.parse("foo(x=i for i in range(10))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', Name('i'))], None, None))])) It's getting to what looks like a valid keyword argument in "x=i" and throwing the rest of it away, when it should be flagging a syntax error (the parser's limited lookahead should still be enough to spot the erroneous 'for' keyword and bail out). The error will be even more obscure if there is an "i" visible from the location of the function call. Whereas when it's parenthesised correctly, the parse tree looks more like this: Py> compiler.parse("foo(x=(i for i in range(10)))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', GenExpr(GenExprInner(Name('i'), [GenExprFor(AssName('i', 'OP_ASSIGN'), CallFunc(Name('range'), [Const(10)], None, None), [])])))], None, None))])) Cheers, Nick. P.S. I added a comment showing the parser output to the SF bug report. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

On 10/10/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem? -argument: [test '='] test [gen_for] # Really [keyword '='] test +argument: test [gen_for] | test '=' test ['(' gen_for ')'] # Really [keyword '='] test n

On 10/16/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
The other option would be changes in the Python/compile.c (somewhat) like following diff -r2.352 compile.c 6356c6356,6362 - if (TYPE(n) == argument && NCH(n) == 3) { --- + if (TYPE(n) == argument && NCH(n) == 4) { + PyErr_SetString(PyExc_SyntaxError, + "invalid syntax"); + symtable_error(st, n->n_lineno); + return; + } + else if (TYPE(n) == argument && NCH(n) == 3) { but IMO, changing the Grammar looks more obvious.

On 10/16/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
Since no one responded other than Jiwon, I checked in this change. I did *not* backport it since what was syntactically correct in 2.4.2 would raise an error in 2.4.3. I'm not sure which is worse. I'll leave it up to Anthony whether this should be backported. BTW, the change was the same regardless of old code vs. new AST code. n

Regarding this Grammar change; (last October) from argument: [test '=' ] test [gen_for] to argument: test [gen_for] | test '=' test ['(' gen_for ')'] - to raise error for "bar(a = i for i in range(10)) )" I think we should change it to argument: test [gen_for] | test '=' test instead of argument: test [gen_for] | test '=' test ['(' gen_for ')'] that is, without ['(' gen_for ')'] . We don't need that extra term, because "test" itself includes generator expressions - with all those parensises. Actually with that extra ['(' gen_for ')'] , foo(a= 10 (for y in 'a')) is grammartically correct ; although that error seems to be checked elsewhere. I tested without ['(' gen_for ')'] , and worked fine passing Lib/test/test_genexps.py -Jiwon On 10/20/05, Neal Norwitz <nnorwitz@gmail.com> wrote:

Neal Norwitz wrote:
I agree it should be a SyntaxError - I believe the AST compiler actually raises one in this situation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

On 10/11/05, Nick Coghlan <ncoghlan@gmail.com> wrote:
Could someone add a test for this on the AST branch? BTW, it looks like doctest is the way to go for SyntaxError tests. There are older tests, like test_scope.py, that use separate files with bad syntax (and lots of extra kludges in the infrastructure to ignore the fact that those .py files can't be compiled). Jeremy

Nick Coghlan wrote:
I was half right. Both the normal compiler and the AST compiler give a SyntaxError if you write: foo((a=i for i in range(10))) The problem is definitely on the parser end though: Py> compiler.parse("foo(x=i for i in range(10))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', Name('i'))], None, None))])) It's getting to what looks like a valid keyword argument in "x=i" and throwing the rest of it away, when it should be flagging a syntax error (the parser's limited lookahead should still be enough to spot the erroneous 'for' keyword and bail out). The error will be even more obscure if there is an "i" visible from the location of the function call. Whereas when it's parenthesised correctly, the parse tree looks more like this: Py> compiler.parse("foo(x=(i for i in range(10)))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', GenExpr(GenExprInner(Name('i'), [GenExprFor(AssName('i', 'OP_ASSIGN'), CallFunc(Name('range'), [Const(10)], None, None), [])])))], None, None))])) Cheers, Nick. P.S. I added a comment showing the parser output to the SF bug report. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

On 10/10/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem? -argument: [test '='] test [gen_for] # Really [keyword '='] test +argument: test [gen_for] | test '=' test ['(' gen_for ')'] # Really [keyword '='] test n

On 10/16/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
The other option would be changes in the Python/compile.c (somewhat) like following diff -r2.352 compile.c 6356c6356,6362 - if (TYPE(n) == argument && NCH(n) == 3) { --- + if (TYPE(n) == argument && NCH(n) == 4) { + PyErr_SetString(PyExc_SyntaxError, + "invalid syntax"); + symtable_error(st, n->n_lineno); + return; + } + else if (TYPE(n) == argument && NCH(n) == 3) { but IMO, changing the Grammar looks more obvious.

On 10/16/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
Since no one responded other than Jiwon, I checked in this change. I did *not* backport it since what was syntactically correct in 2.4.2 would raise an error in 2.4.3. I'm not sure which is worse. I'll leave it up to Anthony whether this should be backported. BTW, the change was the same regardless of old code vs. new AST code. n

Regarding this Grammar change; (last October) from argument: [test '=' ] test [gen_for] to argument: test [gen_for] | test '=' test ['(' gen_for ')'] - to raise error for "bar(a = i for i in range(10)) )" I think we should change it to argument: test [gen_for] | test '=' test instead of argument: test [gen_for] | test '=' test ['(' gen_for ')'] that is, without ['(' gen_for ')'] . We don't need that extra term, because "test" itself includes generator expressions - with all those parensises. Actually with that extra ['(' gen_for ')'] , foo(a= 10 (for y in 'a')) is grammartically correct ; although that error seems to be checked elsewhere. I tested without ['(' gen_for ')'] , and worked fine passing Lib/test/test_genexps.py -Jiwon On 10/20/05, Neal Norwitz <nnorwitz@gmail.com> wrote:
participants (5)
-
Brett Cannon
-
Jeremy Hylton
-
Jiwon Seo
-
Neal Norwitz
-
Nick Coghlan