[Python-Dev] Memory management in the AST parser & compiler
Niko Matsakis
niko at alum.mit.edu
Tue Nov 15 18:50:53 CET 2005
> As Neal pointed out, it's tricky to write code for the AST parser
> and compiler
> without accidentally letting memory leak when the parser or
> compiler runs into
> a problem and has to bail out on whatever it was doing. Thomas's
> patch got to
> v5 (based on Neal's review comments) with memory leaks still in it,
> my review
> got rid of some of them, and we think Neal's last review of v6 of
> the patch
> got rid of the last of them.
Another lurker's 2 cents:
My experience with compilers in particular is that an arena is the
way to go for memory management. I haven't looked at the AST code,
but this can take a variety of forms: anything from linked lists of
pointers to free from something which allocates memory in large
blocks and parcels them out. The goal is just to be able to free the
memory en-masse whatever happens and not have to track individual
pointers.
Generally, compilers have memory allocations which operate in phases
and so are very amenable to arenas. You might have one memory pool
for long lived representation, one that is freed and recreated
between passes, etc.
If you need to keep the AST around long term, then a mark-sweep
garbage collector combined with a linked list might even be a good idea.
Obviously, the whole thing is a tradeoff of peak memory size (which
goes up) against correctness (which is basically ensured, and at
least easily auditable).
Niko
More information about the Python-Dev
mailing list