Claudiu, Peter

Does anyone of you know why lib2to3 is using both a bottom-up parser and pgen2? It "looks" like there are two parsers. I'm quite confused by the parser design of lib2to3, because in older versions (python 2.6) there's only pgen2.

~ Dave

2014-09-29 19:15 GMT+02:00 Claudiu Popa <>:
On Mon, Sep 29, 2014 at 1:02 PM, Dave Halter <> wrote:
> Hi all!
> I will be writing a new parser for Jedi. This is necessary for a quite a few
> reasons described in:
> I'm posting this here, because some of you may be interested in actually
> using the new parser. It's going to be pure Python and contrary to the
> Python's internal parser will have error recovery and indent positions.
> If you're interested, I'd be really glad to hear your feedback!
> ~ Dave

Hello, Dave.

That's awesome, If your parser can generate a new AST, then we can
definitely use it, although we have
in mind to use lib2to3 to replace the standard ast module for our use
cases, but that can change if your parser
is finished until taking action from our side. Having always correct
information, such as line and col offsets definitely will
help us (we actually have a couple of problems regarding this, where
the lineno for some nodes is different than reality).
Regarding the parser approach, I'm not sure what's best in this situation.
I'm working on a static analysis tool, in the vein of pylint, for the
Powershell language
( where I'm using modgrammar, a
recursive descent top down parser to parse the code
and retrieve an AST from it. modgrammar is pretty good, easy to use,
but slow sometimes and backtracks too much on ambiguities.
But it can memorize whitespace tokens and such, as long as you
explicitly define the whitespace token in the grammar, which
can be a pretty daunting task.

Anyway, if there is something I can help you with your parser, don't
hesitate to send me an email.