problem with genexp
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
generates "NameError: name 'i' is not defined" when run because:
2 0 LOAD_GLOBAL 0 (foo) 3 LOAD_CONST 1 ('a') 6 LOAD_GLOBAL 1 (i) 9 CALL_FUNCTION 256 12 POP_TOP 13 LOAD_CONST 0 (None) 16 RETURN_VALUE
If you add parens around the code: foo(a = i for i in range(10)) You get something quite different:
2 0 LOAD_GLOBAL 0 (foo) 3 LOAD_CONST 1 ('a') 6 LOAD_CONST 2 (<code object <generator expression> at 0x2a960baae8, file "<stdin>", line 2>) 9 MAKE_FUNCTION 0 12 LOAD_GLOBAL 1 (range) 15 LOAD_CONST 3 (10) 18 CALL_FUNCTION 1 21 GET_ITER 22 CALL_FUNCTION 1 25 CALL_FUNCTION 256 28 POP_TOP 29 LOAD_CONST 0 (None) 32 RETURN_VALUE
I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
n
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
generates "NameError: name 'i' is not defined" when run because:
[SNIP]
If you add parens around the code: foo(a = i for i in range(10)) You get something quite different:
Do you mean having ``(foo(a = i for i in range(10))``? Otherwise I see no difference when compared to the first value.
-Brett
On 10/10/05, Brett Cannon bcannon@gmail.com wrote:
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
generates "NameError: name 'i' is not defined" when run because:
[SNIP]
If you add parens around the code: foo(a = i for i in range(10)) You get something quite different:
Do you mean having ``(foo(a = i for i in range(10))``? Otherwise I see no difference when compared to the first value.
Sorry, I think I put it in the bug report, but forgot to add it here:
foo(a = (i for i in range(10)))
n
Neal Norwitz wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code: I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
I agree it should be a SyntaxError - I believe the AST compiler actually raises one in this situation.
Cheers, Nick.
On 10/11/05, Nick Coghlan ncoghlan@gmail.com wrote:
Neal Norwitz wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code: I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
I agree it should be a SyntaxError - I believe the AST compiler actually raises one in this situation.
Could someone add a test for this on the AST branch?
BTW, it looks like doctest is the way to go for SyntaxError tests. There are older tests, like test_scope.py, that use separate files with bad syntax (and lots of extra kludges in the infrastructure to ignore the fact that those .py files can't be compiled).
Jeremy
Nick Coghlan wrote:
Neal Norwitz wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code: I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
I agree it should be a SyntaxError - I believe the AST compiler actually raises one in this situation.
I was half right. Both the normal compiler and the AST compiler give a SyntaxError if you write:
foo((a=i for i in range(10)))
The problem is definitely on the parser end though:
Py> compiler.parse("foo(x=i for i in range(10))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', Name('i'))], None, None))]))
It's getting to what looks like a valid keyword argument in "x=i" and throwing the rest of it away, when it should be flagging a syntax error (the parser's limited lookahead should still be enough to spot the erroneous 'for' keyword and bail out). The error will be even more obscure if there is an "i" visible from the location of the function call.
Whereas when it's parenthesised correctly, the parse tree looks more like this: Py> compiler.parse("foo(x=(i for i in range(10)))") Module(None, Stmt([Discard(CallFunc(Name('foo'), [Keyword('x', GenExpr(GenExprInner(Name('i'), [GenExprFor(AssName('i', 'OP_ASSIGN'), CallFunc(Name('range'), [Const(10)], None, None), [])])))], None, None))]))
Cheers, Nick.
P.S. I added a comment showing the parser output to the SF bug report.
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem?
-argument: [test '='] test [gen_for] # Really [keyword '='] test +argument: test [gen_for] | test '=' test ['(' gen_for ')'] # Really [keyword '='] test
n
On 10/16/05, Neal Norwitz nnorwitz@gmail.com wrote:
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem?
-argument: [test '='] test [gen_for] # Really [keyword '='] test +argument: test [gen_for] | test '=' test ['(' gen_for ')'] # Really [keyword '='] test
The other option would be changes in the Python/compile.c (somewhat) like following
diff -r2.352 compile.c 6356c6356,6362 - if (TYPE(n) == argument && NCH(n) == 3) { --- + if (TYPE(n) == argument && NCH(n) == 4) { + PyErr_SetString(PyExc_SyntaxError, + "invalid syntax"); + symtable_error(st, n->n_lineno); + return; + } + else if (TYPE(n) == argument && NCH(n) == 3) {
but IMO, changing the Grammar looks more obvious.
n _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/seojiwon%40gmail.com
On 10/16/05, Neal Norwitz nnorwitz@gmail.com wrote:
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem?
Since no one responded other than Jiwon, I checked in this change. I did *not* backport it since what was syntactically correct in 2.4.2 would raise an error in 2.4.3. I'm not sure which is worse. I'll leave it up to Anthony whether this should be backported.
BTW, the change was the same regardless of old code vs. new AST code.
n
Regarding this Grammar change; (last October) from argument: [test '=' ] test [gen_for] to argument: test [gen_for] | test '=' test ['(' gen_for ')']
- to raise error for "bar(a = i for i in range(10)) )"
I think we should change it to argument: test [gen_for] | test '=' test
instead of argument: test [gen_for] | test '=' test ['(' gen_for ')']
that is, without ['(' gen_for ')'] . We don't need that extra term, because "test" itself includes generator expressions - with all those parensises. Actually with that extra ['(' gen_for ')'] , foo(a= 10 (for y in 'a')) is grammartically correct ; although that error seems to be checked elsewhere.
I tested without ['(' gen_for ')'] , and worked fine passing Lib/test/test_genexps.py
-Jiwon
On 10/20/05, Neal Norwitz nnorwitz@gmail.com wrote:
On 10/16/05, Neal Norwitz nnorwitz@gmail.com wrote:
On 10/10/05, Neal Norwitz nnorwitz@gmail.com wrote:
There's a problem with genexp's that I think really needs to get fixed. See http://python.org/sf/1167751 the details are below. This code:
foo(a = i for i in range(10))
I agree with the bug report that the code should either raise a SyntaxError or do the right thing.
The change to Grammar/Grammar below seems to fix the problem and all the tests pass. Can anyone comment on whether this fix is correct/appropriate? Is there a better way to fix the problem?
Since no one responded other than Jiwon, I checked in this change. I did *not* backport it since what was syntactically correct in 2.4.2 would raise an error in 2.4.3. I'm not sure which is worse. I'll leave it up to Anthony whether this should be backported.
BTW, the change was the same regardless of old code vs. new AST code.
n _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/seojiwon%40gmail.com
On 2/20/06, Jiwon Seo seojiwon@gmail.com wrote:
Regarding this Grammar change; (last October) from argument: [test '=' ] test [gen_for] to argument: test [gen_for] | test '=' test ['(' gen_for ')']
- to raise error for "bar(a = i for i in range(10)) )"
I think we should change it to argument: test [gen_for] | test '=' test
instead of argument: test [gen_for] | test '=' test ['(' gen_for ')']
that is, without ['(' gen_for ')'] . We don't need that extra term, because "test" itself includes generator expressions - with all those parensises.
Works for me, committed.
n
participants (5)
-
Brett Cannon -
Jeremy Hylton -
Jiwon Seo -
Neal Norwitz -
Nick Coghlan