[Python-Dev] Memory management in the AST parser & compiler

[Greg Ewing]

Neal Norwitz wrote:

> This is an entire function from Python/ast.c.

> Sequences do not know what type they hold, so there needs to be
> different dealloc functions to free them properly (asdl_*_seq_free()).

Well, that's one complication that would go away if
the nodes were PyObjects.

> The memory leak occurs when FunctionDef fails.  name, args, body, and
> decorator_seq are all local and would not be freed.  The simple
> variables can be freed in each "constructor" like FunctionDef(), but
> the sequences cannot unless they keep the info about which type they
> hold.

If FunctionDef's reference semantics are defined so
that it steals references to its arguments, then here
is how the same function would look with PyObject
AST nodes, as far as I can see:

  static PyObject *
  ast_for_funcdef(struct compiling *c, const node *n)
  {
      /* funcdef: 'def' [decorators] NAME parameters ':' suite */
      PyObject *name = NULL;
      PyObject *args = NULL;
      PyObject *body = NULL;
      PyObject *decorator_seq = NULL;
      int name_i;

      REQ(n, funcdef);

      if (NCH(n) == 6) { /* decorators are present */
  	decorator_seq = ast_for_decorators(c, CHILD(n, 0));
  	if (!decorator_seq)
  	    goto error;
  	name_i = 2;
      }
      else {
  	name_i = 1;
      }

      name = NEW_IDENTIFIER(CHILD(n, name_i));
      if (!name)
  	goto error;
      else if (!strcmp(STR(CHILD(n, name_i)), "None")) {
  	ast_error(CHILD(n, name_i), "assignment to None");
  	goto error;
      }
      args = ast_for_arguments(c, CHILD(n, name_i + 1));
      if (!args)
  	goto error;
      body = ast_for_suite(c, CHILD(n, name_i + 3));
      if (!body)
  	goto error;

      return FunctionDef(name, args, body, decorator_seq, LINENO(n));

  error:
      Py_XDECREF(body);
      Py_XDECREF(decorator_seq);
      Py_XDECREF(args);
      Py_XDECREF(name);
      return NULL;
  }

The only things I've changed are turning some type
declarations into PyObject * and replacing the
deallocation functions at the end with Py_XDECREF!

Maybe there are other functions where it would not
be so straightforward, but if this really is a
typical AST function, switching to PyObjects looks
like it wouldn't be difficult at all, and would
actually make some things simpler.

-- 
Greg Ewing, Computer Science Dept, +--------------------------------------+
University of Canterbury,	   | A citizen of NewZealandCorp, a	  |
Christchurch, New Zealand	   | wholly-owned subsidiary of USA Inc.  |
greg.ewing at canterbury.ac.nz	   +--------------------------------------+