[Python-ideas] Hooking between lexer and parser

[Nick Coghlan]

On 6 June 2015 at 15:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 6 June 2015 at 12:21, Neil Girdhar <mistersheik at gmail.com> wrote:
>> I'm curious what other people will contribute to this discussion as I think
>> having no great parsing library is a huge hole in Python.  Having one would
>> definitely allow me to write better utilities using Python.
>
> The design of *Python's* grammar is deliberately restricted to being
> parsable with an LL(1) parser. There are a great many static analysis
> and syntax highlighting tools that are able to take advantage of that
> simplicity because they only care about the syntax, not the full
> semantics.
>
> Anyone actually doing their *own* parsing of something else *in*
> Python, would be better advised to reach for PLY
> (https://pypi.python.org/pypi/ply ). PLY is the parser underlying
> https://pypi.python.org/pypi/pycparser, and hence the highly regarded
> CFFI library, https://pypi.python.org/pypi/cffi

For the later stages of the pipeline (i.e. AST -> code generation),
CPython now uses Eli Bendersky's asdl_parser:
https://github.com/eliben/asdl_parser

More background on that:
http://eli.thegreenplace.net/2014/06/04/using-asdl-to-describe-asts-in-compilers

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia