[Python-Dev] Memory management in the AST parser & compiler

[Guido van Rossum]

[Guido]
> > Then I don't understand why there was discussion of alloca() earlier
> > on -- surely the lifetime of a node should not be limited by the stack
> > frame that allocated it?

[Jeremy]
> Actually this is a pretty good limit, because all these data
> structures are temporaries used by the compiler.  Once compilation has
> finished, there's no need for the AST or the compiler state.

Are you really saying that there is one function which is called only
once (per compilation) which allocates *all* the AST nodes? That's the
only situation where I'd see alloca() working -- unless your alloca()
doesn't allocate memory on the stack. I was somehow assuming that the
tree would be built piecemeal by parser callbacks or some such
mechanism. There's still a stack frame whose lifetime limits the AST
lifetime, but it is not usually the current stackframe when a new node
is allocated, so alloca() can't be used.

I guess I don't understand the AST compiler code enough to participate
in this discussion. Or perhaps we are agreeing violently?

> > I'm not in principle against having an arena for this purpose, but I
> > worry that this will make it really hard to provide a Python API for
> > the AST, which has already been requested and whose feasibility
> > (unless I'm mistaken) also was touted as an argument for switching to
> > the AST compiler in the first place. I hope we'll never have to deal
> > with an API like the parser module provides...
>
> My preference would be to have the ast shared by value.  We generate
> code to serialize it to and from a byte stream and share that between
> Python and C.  It is less efficient, but it is also very simple.

So there would still be a Python-objects version of the AST but the
compiler itself doesn't use it.

At least by-value makes sense to me -- if you're making tree
transformations you don't want accidental sharing to cause unexpected
side effects.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)