[Python-Dev] Memory management in the AST parser & compiler
jeremy at alum.mit.edu
Tue Nov 15 20:42:13 CET 2005
On 11/15/05, Niko Matsakis <niko at alum.mit.edu> wrote:
> > As Neal pointed out, it's tricky to write code for the AST parser
> > and compiler
> > without accidentally letting memory leak when the parser or
> > compiler runs into
> > a problem and has to bail out on whatever it was doing. Thomas's
> > patch got to
> > v5 (based on Neal's review comments) with memory leaks still in it,
> > my review
> > got rid of some of them, and we think Neal's last review of v6 of
> > the patch
> > got rid of the last of them.
> Another lurker's 2 cents:
> My experience with compilers in particular is that an arena is the
> way to go for memory management. I haven't looked at the AST code,
> but this can take a variety of forms: anything from linked lists of
> pointers to free from something which allocates memory in large
> blocks and parcels them out. The goal is just to be able to free the
> memory en-masse whatever happens and not have to track individual
Thanks for the message. I was going to suggest the same thing. I
think it's primarily a question of how to add an arena layer. The AST
phase has a mixture of malloc/free and Python object allocation. It
should be straightforward to change the malloc/free code to use an
arena API. We'd probably need a separate mechanism to associate a set
of PyObject* with the arena and have those DECREFed.
> Generally, compilers have memory allocations which operate in phases
> and so are very amenable to arenas. You might have one memory pool
> for long lived representation, one that is freed and recreated
> between passes, etc.
> If you need to keep the AST around long term, then a mark-sweep
> garbage collector combined with a linked list might even be a good idea.
> Obviously, the whole thing is a tradeoff of peak memory size (which
> goes up) against correctness (which is basically ensured, and at
> least easily auditable).
> Python-Dev mailing list
> Python-Dev at python.org
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
More information about the Python-Dev