[Python-Dev] Reference counting when entering and exiting scopes

[Brett C.]

Matthew F. Barnes wrote:
> Someone on python-help suggested that I forward this question to
> python-dev.
> 
> I've been studying Python's core compiler and bytecode interpreter as a
> model for my own interpreted language,

Might want to take a peek at the AST branch in CVS; that is what the compiler
is going to change to as soon as it is complete.

> and I've come across what appears
> to be a reference counting problem in the `symtable_exit_scope' function
> in <Python/compile.c>.
> 
> At this point I assume that I'm just misunderstanding what's going on.  So
> I was hoping to contact one of the core developers before I go filing what
> could very well be a spurious bug report against Python's core.
> 

Spurious bug reports are fine.  If they turn out to be that they get closed as
such.  Either way time is spent checking it whether it goes there or here.  But
at least with a bug report it can be tracked more easily.

So for future reference, just go ahead and file the bug report.

> Here's the function copied from CVS HEAD:
> 
> static int
> symtable_exit_scope(struct symtable *st)
> {
> 	int end;
> 
> 	if (st->st_pass == 1)
> 		symtable_update_free_vars(st);
> 	Py_DECREF(st->st_cur);
> 	end = PyList_GET_SIZE(st->st_stack) - 1;
> 	st->st_cur = (PySymtableEntryObject *)PyList_GET_ITEM(st->st_stack,
> 							      end);
> 	if (PySequence_DelItem(st->st_stack, end) < 0)
> 		return -1;
> 	return 0;
> }
> 
> My issue is with the use of PyList_GET_ITEM to fetch a new value for the
> current scope.  As I understand it, PyList_GET_ITEM does not increment the
> reference count for the returned value.  So in effect we're borrowing the
> reference to the symtable entry object from the tail of the scope stack. 
> But then we turn around and delete the object from the tail of the scope
> stack, which DOES decrement the reference count.
> 
> So `symtable_exit_scope' has a net effect of decrementing the reference
> count of the new current symtable entry object, when it seems to me like
> it should stay the same.  Shouldn't the reference count be incremented
> when we assign to "st->st_cur" (either explicitly or by fetching the
> object using the PySequence API instead of PyList)?
> 
> Can someone explain the rationale here?
> 

If you look at how symtable_enter_scope() and symtable_exit_scope() work
together you will notice there is actually no leak.  symtable_enter_scope()
appends the existing PySymtableEntryObject on to the symtable stack and then
places a new PySymtableEntryObject into st->st_cur.  Both at this point have a
refcount of one; enough to stay alive.

Now look at symtable_exit_scope().  When the current PySymtableEntryObject is
no longer needed, it is DECREF'ed, putting it at 0 and thus leading to eventual
collection.  What is on top of the symtable stack, which has a refcount of 1
still, is then put in to st->st_cur.

So no leak.  Yes, there should be more explicit refcounting to be proper, but
the compiler cheats in a couple of places for various reasons.  But basically
everything is fine since st->st_cur and st->st_stack are only played with
refcount-wise by either symtable_enter_scope() and symtable_exit_scope() and
they are always called in pairs in the end.

-Brett