Modified parser does not seem to work. What am I doing wrong?
Hello, I am trying to implement PEP-637, and I started modifying the parser and the grammar, but I don't know what I am missing. The PR is here https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-imp... It includes other stuff but the core is that I extended the Subscript in the asdl to accept the keyword args | Subscript(expr value, expr slice, keyword* keywords, expr_context ctx) which seems to work:
ast.parse('a[3]').body[0].value.keywords []
ast.parse('a[k=3]').body[0].value.keywords Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
I also added a few "productions" (I believe they are called like this, my compiler theory is very approximated), one of which is now primary[expr_ty]: .... | a=primary '[' b=slices c=[',' k=kwargs {k}]']' { _Py_Subscript(a, b, c, Load, EXTRA) } I also tried with additional formats like | a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c, Load, EXTRA) } or even just | a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) } but I always get a SyntaxError: line 50, in parse return compile(source, filename, mode, flags, File "<unknown>", line 1 a[k=3] ^ SyntaxError: invalid syntax Note that I always recreated ast parser and pegen.c with 'make regen-all' and recompiled with make. I tried to enable debug and print the pegen.c debug, but it's a bit heavy and I could not immediately spot the issue. I suspect I am missing something somewhere. Thanks -- Kind regards, Stefano Borini
Hi Stefano, One of the problems you have is that the rule for slices has a negative lookahead for the comma: slices[expr_ty]: | a=slice !',' { a } IIRC the reason that is there is to allow "x[3,]" to be parsed. Also, to allow "a[k=3]" you need to to create a rule that allows skipping the "slices" part of the subscript and allow the 2nd argument of _Py_Subscript to be NULL. For example: | a=t_primary '[' k=kwargs ']' !t_lookahead { _Py_Subscript(a, NULL, k, Store, EXTRA) } Regards, Pablo On Fri, 16 Oct 2020 at 22:07, Stefano Borini <stefano.borini@gmail.com> wrote:
Hello,
I am trying to implement PEP-637, and I started modifying the parser and the grammar, but I don't know what I am missing.
The PR is here
https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-imp...
It includes other stuff but the core is that I extended the Subscript in the asdl to accept the keyword args
| Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
which seems to work:
ast.parse('a[3]').body[0].value.keywords []
I also added a few "productions" (I believe they are called like this, my compiler theory is very approximated), one of which is now
primary[expr_ty]: .... | a=primary '[' b=slices c=[',' k=kwargs {k}]']' { _Py_Subscript(a, b, c, Load, EXTRA) }
I also tried with additional formats like
| a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c, Load, EXTRA) }
or even just
| a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
but I always get a SyntaxError:
ast.parse('a[k=3]').body[0].value.keywords Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py", line 50, in parse return compile(source, filename, mode, flags, File "<unknown>", line 1 a[k=3] ^ SyntaxError: invalid syntax
Note that I always recreated ast parser and pegen.c with 'make regen-all' and recompiled with make. I tried to enable debug and print the pegen.c debug, but it's a bit heavy and I could not immediately spot the issue. I suspect I am missing something somewhere.
Thanks
-- Kind regards,
Stefano Borini _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZP... Code of Conduct: http://python.org/psf/codeofconduct/
That's what I suspected looking at the parser debug in fact. Good to see that I was on the right track. Thanks! I'll play with it a bit more tonight. On Fri, 16 Oct 2020 at 23:43, Pablo Galindo Salgado <pablogsal@gmail.com> wrote:
Hi Stefano,
One of the problems you have is that the rule for slices has a negative lookahead for the comma:
slices[expr_ty]: | a=slice !',' { a }
IIRC the reason that is there is to allow "x[3,]" to be parsed.
Also, to allow "a[k=3]" you need to to create a rule that allows skipping the "slices" part of the subscript and allow the 2nd argument of _Py_Subscript to be NULL. For example:
| a=t_primary '[' k=kwargs ']' !t_lookahead { _Py_Subscript(a, NULL, k, Store, EXTRA) }
Regards, Pablo
On Fri, 16 Oct 2020 at 22:07, Stefano Borini <stefano.borini@gmail.com> wrote:
Hello,
I am trying to implement PEP-637, and I started modifying the parser and the grammar, but I don't know what I am missing.
The PR is here
https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-imp...
It includes other stuff but the core is that I extended the Subscript in the asdl to accept the keyword args
| Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
which seems to work:
ast.parse('a[3]').body[0].value.keywords []
I also added a few "productions" (I believe they are called like this, my compiler theory is very approximated), one of which is now
primary[expr_ty]: .... | a=primary '[' b=slices c=[',' k=kwargs {k}]']' { _Py_Subscript(a, b, c, Load, EXTRA) }
I also tried with additional formats like
| a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c, Load, EXTRA) }
or even just
| a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
but I always get a SyntaxError:
ast.parse('a[k=3]').body[0].value.keywords Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py", line 50, in parse return compile(source, filename, mode, flags, File "<unknown>", line 1 a[k=3] ^ SyntaxError: invalid syntax
Note that I always recreated ast parser and pegen.c with 'make regen-all' and recompiled with make. I tried to enable debug and print the pegen.c debug, but it's a bit heavy and I could not immediately spot the issue. I suspect I am missing something somewhere.
Thanks
-- Kind regards,
Stefano Borini _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZP... Code of Conduct: http://python.org/psf/codeofconduct/
-- Kind regards, Stefano Borini
On Fri, 16 Oct 2020, 23:04 Stefano Borini, <stefano.borini@gmail.com> wrote:
Hello,
I am trying to implement PEP-637, and I started modifying the parser and the grammar, but I don't know what I am missing.
The PR is here
https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-imp...
It includes other stuff but the core is that I extended the Subscript in the asdl to accept the keyword args
| Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
which seems to work:
ast.parse('a[3]').body[0].value.keywords []
I also added a few "productions" (I believe they are called like this, my compiler theory is very approximated), one of which is now
primary[expr_ty]: .... | a=primary '[' b=slices c=[',' k=kwargs {k}]']' { _Py_Subscript(a, b, c, Load, EXTRA) }
I also tried with additional formats like
| a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c, Load, EXTRA) }
or even just
| a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
but I always get a SyntaxError:
ast.parse('a[k=3]').body[0].value.keywords Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py", line 50, in parse return compile(source, filename, mode, flags, File "<unknown>", line 1 a[k=3] ^ SyntaxError: invalid syntax
Note that I always recreated ast parser and pegen.c with 'make regen-all' and recompiled with make. I tried to enable debug and print the pegen.c debug, but it's a bit heavy and I could not immediately spot the issue. I suspect I am missing something somewhere.
Have you manually checked that the generated source in pegen.c contains your changes? You should be able to see by just looking for calls to _Py_Subscript, or other code snippets from the modified grammar. Or, you can introduce an error in the grammar's code snippets and check that you get the expected error. FWIW, I found I had to explicitly regenerate pegen.c with 'make regen-pegen'; 'make regen-all' wasn't enough.
Thanks
-- Kind regards,
Stefano Borini _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZP... Code of Conduct: http://python.org/psf/codeofconduct/
participants (3)
-
Pablo Galindo Salgado -
Stefano Borini -
Thomas Wouters