[Python-Dev] Memory management in the AST parser & compiler

[Brett Cannon]

On 11/15/05, Neal Norwitz <nnorwitz at gmail.com> wrote:
> On 11/15/05, Jeremy Hylton <jeremy at alum.mit.edu> wrote:
> >
> > Thanks for the message.  I was going to suggest the same thing.  I
> > think it's primarily a question of how to add an arena layer.  The AST
> > phase has a mixture of malloc/free and Python object allocation.  It
> > should be straightforward to change the malloc/free code to use an
> > arena API.  We'd probably need a separate mechanism to associate a set
> > of PyObject* with the arena and have those DECREFed.
>
> Well good.  It seems we all agree there is a problem and on the
> general solution.  I haven't thought about Brett's idea to see if it
> could work or not.  It would be great if we had someone start working
> to improve the situation.  It could well be that we live with the
> current code for 2.5, but it would be great to use arenas for 2.6 at
> least.
>

 I have been thinking about this some more  to put off doing homework
and I have some random ideas I just wanted to toss out there to make
sure I am not thinking about arena memory management incorrectly
(never actually encountered it directly before).

I think an arena API is going to be the best solution.  Pulling
trickery with redefining Py_INCREF and such like I suggested seems
like a pain and possibly error-prone.  With the compiler being a
specific corner of the core having a special API for handling the
memory for PyObject* stuff seems reasonable.

We might need PyArena_Malloc() and PyArena_New() to handle malloc()
and PyObject* creation.  We could then have a struct that just stored
pointers to the allocated memory (linked list for each pointer which
gives high memory overhead or linked list of arrays that should lower
memory but make having possible holes in the array for stuff already
freed a pain to handle).  We would then have PyArena_FreeAll() that
would be strategically placed in the code for when bad things happen
that would just traverse the lists and free everything.  I assume
having a way to free individual items might be useful.  Could have the
PyArena_New() and _Malloc() return structs with the needed info for a
PyArena_Free(location_struct) to be able to fee the specific item
without triggering a complete freeing of all memory.  But this usage
should be discouraged and only used when proper memory management is
guaranteed.

Boy am I wanting RAII from C++ for automatic freeing when scope is
left.  Maybe we need to come up with a similar thing, like all memory
that should be freed once a scope is left must use some special struct
that stores references to all created memory locally and then a free
call must be made at all exit points in the function using the special
struct.  Otherwise the pointer is stored in the arena and handled
en-mass later.

Hopefully this is all made some sense.  =)  Is this the basic strategy
that an arena setup would need?  if not can someone enlighten me?


-Brett