[Patches] Fix for memory leak in _PyResize_Tuple

[Charles G Waldman]

In the course of testing for memory leaks I found this gem in
test_extcall (there are actually several different memory leaks
triggered by test_extcall, this is only the first of several fixes,
which I am submitting as separate patches).



Test Case:  run the following code:

class Nothing:
    def __len__(self):
        return 5
    def __getitem__(self, i):
        if i < 3:
            return i
        else:
            raise IndexError, i

def g(a,*b,**c):
    return

for x in xrange(1000000):
    g(*Nothing())


and watch Python's memory use go up and up.


Diagnosis:

The analysis begins with the call to PySequence_Tuple at line 1641 in
ceval.c - the argument to g is seen to be a sequence but not a tuple,
so it needs to be converted from an abstract sequence to a concrete
tuple.  PySequence_Tuple starts off by creating a new tuple of length
5 (line 1122 in abstract.c).  Then at line 1149, since only 3 elements
were assigned, _PyTuple_Resize is called to make the 5-tuple into a
3-tuple.  When we're all done the 3-tuple is decrefed, but rather than
being freed it is placed on the free_tuples cache.

The basic problem is that the 3-tuples are being added to the cache
but never picked up again, since _PyTuple_Resize doesn't make use of
the free_tuples cache.  If you are resizing a 5-tuple to a 3-tuple and
there is already a 3-tuple in free_tuples[3], instead of using this
tuple, _PyTuple_Resize will realloc the 5-tuple to a 3-tuple.  It
would more efficient to use the existing 3-tuple and cache the
5-tuple.

By making _PyTuple_Resize aware of the free_tuples (just as
PyTuple_New), we not only save a few calls to realloc, but also
prevent this misbehavior whereby tuples are being added to the
free_tuples list but never properly "recycled".

Patch:
Index: Objects/tupleobject.c
===================================================================
RCS file: /projects/cvsroot/python/dist/src/Objects/tupleobject.c,v
retrieving revision 2.30
diff -c -r2.30 tupleobject.c
*** tupleobject.c	2000/03/13 16:01:29	2.30
--- tupleobject.c	2000/04/15 22:08:27
***************
*** 469,482 ****
  		Py_XDECREF(v->ob_item[i]);
  		v->ob_item[i] = NULL;
  	}
! 	sv = (PyTupleObject *)
! 		realloc((char *)v,
! 			sizeof(PyTupleObject) + newsize * sizeof(PyObject *));
! 	*pv = (PyObject *) sv;
! 	if (sv == NULL) {
! 		PyMem_DEL(v);
! 		PyErr_NoMemory();
! 		return -1;
  	}
  	_Py_NewReference((PyObject *)sv);
  	for (i = sv->ob_size; i < newsize; i++)
--- 469,515 ----
  		Py_XDECREF(v->ob_item[i]);
  		v->ob_item[i] = NULL;
  	}
! #if MAXSAVESIZE > 0
! 	if (newsize == 0 && free_tuples[0]) {
! 		sv = free_tuples[0];
! 		sv->ob_size = 0;
! 		Py_INCREF(sv);
! #ifdef COUNT_ALLOCS
! 		tuple_zero_allocs++;
! #endif
! 		_Py_Dealloc((PyObject*)v);
! 		*pv = (PyObject*) sv;
! 		return 0;
! 	}
! 	if (0 < newsize && newsize < MAXSAVESIZE &&
! 	    (sv = free_tuples[newsize]) != NULL)
! 	{
! 		free_tuples[newsize] = (PyTupleObject *) sv->ob_item[0];
! #ifdef COUNT_ALLOCS
! 		fast_tuple_allocs++;
! #endif
! #ifdef Py_TRACE_REFS 
! 		sv->ob_type = &PyTuple_Type; 
! #endif 
! 		for (i = 0; i < newsize; ++i){
! 			sv->ob_item[i] = v->ob_item[i];
! 			v->ob_item[i] = NULL;
! 		}
! 		sv->ob_size = v->ob_size;
! 		_Py_Dealloc((PyObject*)v);
! 		*pv = (PyObject *) sv;
! 	} else 
! #endif		
! 	{
! 		sv = (PyTupleObject *)
! 			realloc((char *)v,
! 				sizeof(PyTupleObject) + newsize * sizeof(PyObject *));
! 		*pv = (PyObject *) sv;
! 		if (sv == NULL) {
! 			PyMem_DEL(v);
! 			PyErr_NoMemory();
! 			return -1;
! 		}
  	}
  	_Py_NewReference((PyObject *)sv);
  	for (i = sv->ob_size; i < newsize; i++)





Disclaimer:

     I confirm that, to the best of my knowledge and belief, this
     contribution is free of any claims of third parties
     under copyright, patent or other rights or interests ("claims").  To
     the extent that I have any such claims, I hereby grant to CNRI a
     nonexclusive, irrevocable, royalty-free, worldwide license to
     reproduce, distribute, perform and/or display publicly, prepare
     derivative versions, and otherwise use this contribution as part
     of the Python software and its related documentation, or any
     derivative versions thereof, at no cost to CNRI or its licensed
     users, and to authorize others to do so.

     I acknowledge that CNRI may, at its sole discretion, decide
     whether or not to incorporate this contribution in the Python
     software and its related documentation.  I further grant CNRI
     permission to use my name and other identifying information
     provided to CNRI by me for use in connection with the Python
     software and its related documentation.