[Python-Dev] Memory management in the AST parser & compiler

[Niko Matsakis]

> As Neal pointed out, it's tricky to write code for the AST parser  
> and compiler
> without accidentally letting memory leak when the parser or  
> compiler runs into
> a problem and has to bail out on whatever it was doing. Thomas's  
> patch got to
> v5 (based on Neal's review comments) with memory leaks still in it,  
> my review
> got rid of some of them, and we think Neal's last review of v6 of  
> the patch
> got rid of the last of them.

Another lurker's 2 cents:

My experience with compilers in particular is that an arena is the  
way to go for memory management.  I haven't looked at the AST code,  
but this can take a variety of forms: anything from linked lists of  
pointers to free from something which allocates memory in large  
blocks and parcels them out.  The goal is just to be able to free the  
memory en-masse whatever happens and not have to track individual  
pointers.

Generally, compilers have memory allocations which operate in phases  
and so are very amenable to arenas.  You might have one memory pool  
for long lived representation, one that  is freed and recreated  
between passes, etc.

If you need to keep the AST around long term, then a mark-sweep  
garbage collector combined with a linked list might even be a good idea.

Obviously, the whole thing is a tradeoff of peak memory size (which  
goes up) against correctness (which is basically ensured, and at  
least easily auditable).


Niko