[Python-Dev] Memory management in the AST parser & compiler

[Brett Cannon]

On 11/15/05, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
> On 11/15/05, Niko Matsakis <niko at alum.mit.edu> wrote:
> > > As Neal pointed out, it's tricky to write code for the AST parser
> > > and compiler
> > > without accidentally letting memory leak when the parser or
> > > compiler runs into
> > > a problem and has to bail out on whatever it was doing. Thomas's
> > > patch got to
> > > v5 (based on Neal's review comments) with memory leaks still in it,
> > > my review
> > > got rid of some of them, and we think Neal's last review of v6 of
> > > the patch
> > > got rid of the last of them.
> >
> > Another lurker's 2 cents:
> >
> > My experience with compilers in particular is that an arena is the
> > way to go for memory management.  I haven't looked at the AST code,
> > but this can take a variety of forms: anything from linked lists of
> > pointers to free from something which allocates memory in large
> > blocks and parcels them out.  The goal is just to be able to free the
> > memory en-masse whatever happens and not have to track individual
> > pointers.
>
> Thanks for the message.  I was going to suggest the same thing.  I
> think it's primarily a question of how to add an arena layer.  The AST
> phase has a mixture of malloc/free and Python object allocation.  It
> should be straightforward to change the malloc/free code to use an
> arena API.  We'd probably need a separate mechanism to associate a set
> of PyObject* with the arena and have those DECREFed.
>

Might just need two lists; malloc'ed pointers and PyObject pointers. 
Could redefine Py_INCREF and Py_DECREF locally for ast.c and compile.c
to use the arena API and thus hide the detail.  Otherwise just a big,
flashing "USE THIS API" sign will be needed.

I have gone ahead and added this as a possible topic to sprint on at PyCon.

-Brett

> Jeremy
>
> >
> > Generally, compilers have memory allocations which operate in phases
> > and so are very amenable to arenas.  You might have one memory pool
> > for long lived representation, one that  is freed and recreated
> > between passes, etc.
> >
> > If you need to keep the AST around long term, then a mark-sweep
> > garbage collector combined with a linked list might even be a good idea.
> >
> > Obviously, the whole thing is a tradeoff of peak memory size (which
> > goes up) against correctness (which is basically ensured, and at
> > least easily auditable).
> >
> >
> > Niko
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu
> >
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>