[Python-Dev] Memory management in the AST parser & compiler

[Brett Cannon]

On 11/29/05, Guido van Rossum <guido at python.org> wrote:
> On 11/28/05, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> > [...] My mental model
> > of parsing & compiling in the presence of a parse tree
> > is like this:
> >
> >    [source] -> scanner -> [tokens]
> >      -> parser -> [AST] -> code_generator -> [code]
> >
> > The fact that there still seems to be another kind of
> > parse tree in between the scanner and the AST generator
> > is an oddity which I hope will eventually disappear.
>
> Have a look at http://python.org/sf/1337696 -- a reimplementation of
> pgen in Python that I did for Elemental and am contributing to the
> PSF. It customizes the tree generation callback so as to let you
> produce an style of AST you like.
>
> > > I know
> > > Guido has said he doesn't like punishing the performance of small
> > > scripts in the name of large-scale apps
> >
> > To me, that's an argument in favour of always generating
> > a .pyc, even for scripts.
>
> I'm not sure I follow the connection.

Greg was proposing having parser, AST, and bytecode compilation all be
written in Python and frozen into the executable instead of it being
all C code.  I said that would be slower and would punish single file
scripts that don't get a .pyc generated for them because they would
need to have the file compiled every execution.  Greg said that is
just a good argument for  having *any* file, imported or passed in on
the command line, to have a .pyc generated when possible.

> But I wouldn't mind if someone
> contributed code that did this. :)
>

=)  Shouldn't be that complicated (but I don't have time for it right
now so it isn't dead simple either  =).

-Brett