[Python-Dev] Memory management in the AST parser & compiler

[Fredrik Lundh]

Jeremy Hylton wrote:

> > Me neither. Adding yet another memory allocation scheme to Python's
> > already staggering number of memory allocation strategies sounds like
> > a bad idea.
>
> The reason this thread started was the complaint that reference
> counting in the compiler is really difficult.  Almost every line of
> code can lead to an error exit.  The code becomes quite cluttered when
> it uses reference counting.  Right now, the AST is created with
> malloc/free, but that makes it hard to free the ast at the right time.
>  It would be fairly complex to convert the ast nodes to pyobjects.
> They're just simple discriminated unions right now.  If they were
> allocated from an arena, the entire arena could be freed when the
> compilation pass ends.

if you're using PyObject's for everything, you can use a list object as the
arena.  just append every "transient" value to the arena list, and a single
DECREF will get rid of it all.  if you want to copy something out from the
arena, just INCREF the object and it's yours.

(for performance reasons, it might be a good idea to add a _PyList_APPEND
helper that works like app1 but steals the value reference; e.g.

PyObject*
_PyList_APPEND(PyListObject *self, PyObject *v)
{
    int n;
    if (!v)
        return v;
    n = PyList_GET_SIZE(self);
    if (n == INT_MAX) {
        PyErr_SetString(PyExc_OverflowError,
        "cannot add more objects to list");
        return NULL;
    }
    if (list_resize(self, n+1) == -1)
        return NULL;
    PyList_SET_ITEM(self, n, v);
    return v;
}

which can be called as

    obj = _PyList_APPEND(c->arena, AST_Foobar_New(...));
    if (!obj)
        return NULL;

</F>