[pypy-dev] parser compiler and _ast

Maciej Fijalkowski fijall at gmail.com
Sat Jan 24 20:42:55 CET 2009

I did a bit of research on a matter of different python interfaces to
parsing. This is a bit of a mess, but it is as follows:

a) there is a parser module which gives you very low-level interface
    (so-called concrete syntax tree), which is essentially
    a list of tuples with numbers inside. you can compile those tuples
    to bytecode afterwards. It's slowly being deprecated.

b) there is a compiler module, which is a wrapper around parser. it
    does not let you compile ast, but it gives you ast. It's deprecated since

c) there is an ast module which lets you have ast *and* compile it. very neat.

Now the state of the art is that we have parser, compiler is pure
app-level and we
don't provide ast. We already discussed a bit whether to have ast module for 1.1
and the answer is "very likely no", raise hands if you want to have it.

Now I want to know how much do we care about all of this and how much do we care
about any of those? My suggestions are as follows:

1. We keep parser as it is and don't try to fix bugs. Bugs (as
reported by CPython's
test failures) are that we don't do extra walk around the tree to
detect syntax errors.
those errors are only detected when we generated the code

- or -

1a. We just always generate the code and throw it away. We loose
performance, but we don't care
(and we pass tests)

2. We slowly deprecate parser & compiler, with some removal in future

3. If we develop a new parser, we drop compatibility with exact
concrete syntax trees and
we only keep ast around (as ast module) if people want to interact
with it (this is one less
worry for writing parser).

What do you think?


More information about the Pypy-dev mailing list