[Python-Dev] Memory management in the AST parser & compiler

[Guido van Rossum]

On 11/28/05, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
> On 11/28/05, Guido van Rossum <guido at python.org> wrote:
> > On 11/18/05, Neil Schemenauer <nas at arctrix.com> wrote:
> > > Perhaps we should use the memory management technique that the rest
> > > of Python uses: reference counting.  I don't see why the AST
> > > structures couldn't be PyObjects.
> >
> > Me neither. Adding yet another memory allocation scheme to Python's
> > already staggering number of memory allocation strategies sounds like
> > a bad idea.
>
> The reason this thread started was the complaint that reference
> counting in the compiler is really difficult.  Almost every line of
> code can lead to an error exit.

Sorry, I forgot that (I've been off-line for a week of quality time
with Orlijn, and am now digging my self out from under several hundred
emails :-).

> The code becomes quite cluttered when
> it uses reference counting.  Right now, the AST is created with
> malloc/free, but that makes it hard to free the ast at the right time.

Would fixing the code to add free() calls in all the error exits make
it more or less cluttered than using reference counting?

>  It would be fairly complex to convert the ast nodes to pyobjects.
> They're just simple discriminated unions right now.

Are they all the same size?

> If they were
> allocated from an arena, the entire arena could be freed when the
> compilation pass ends.

Then I don't understand why there was discussion of alloca() earlier
on -- surely the lifetime of a node should not be limited by the stack
frame that allocated it?

I'm not in principle against having an arena for this purpose, but I
worry that this will make it really hard to provide a Python API for
the AST, which has already been requested and whose feasibility
(unless I'm mistaken) also was touted as an argument for switching to
the AST compiler in the first place. I hope we'll never have to deal
with an API like the parser module provides...

--
--Guido van Rossum (home page: http://www.python.org/~guido/)