Hunting a memory leak

[Francesc Alted]

[Ooops. Something went wrong with my newsreader config ;-)]

Thanks for the responses!. I started by fetching the TrackRefs() class
from http://cvs.zope.org/Zope3/test.py and pasted it in my local copy
of unittest.py. Then, I've modified the TestCase.__call__ try: block
from the original:


            try:
                testMethod()
                ok = 1

to read:

           try:
                rc1 = rc2 = None
                #Pre-heating
                for i in xrange(10):
                    testMethod()
                gc.collect()
                rc1 = sys.gettotalrefcount()
                track = TrackRefs()
		# Second (first "valid") loop
                for i in xrange(10):
                    testMethod()
                gc.collect()
                rc2 = sys.gettotalrefcount()
                print "First output of TrackRefs:"
                track.update()
                print >>sys.stderr, "%5d %s.%s.%s()" % (rc2-rc1,
                testMethod.__module__, testMethod.im_class.__name__,
                testMethod.im_func.__name__)
		# Third loop
                for i in xrange(10):
                    testMethod()
                gc.collect()
                rc3 = sys.gettotalrefcount()
                print "Second output of TrackRefs:"
                track.update()
                print >>sys.stderr, "%5d %s.%s.%s()" % (rc3-rc2,
		testMethod.__module__, testMethod.im_class.__name__,
		testMethod.im_func.__name__)
                ok = 1

However, I'm not sure if I have made a good implementation.  My
understanding is that the first loop is for pre-heating (to avoid
false count-refs due to cache issues and so). The second loop should
already give good count references and, thereby, I've made a call to
track.update(). Finally, I wanted to re-check the results of the
second loop with a third one. Therefore, I expected more or less the
same results in second and third loops.

But... the results are different!. Following are the results of this run:

$ python2.3 widetree3.py
First output of TrackRefs:
<type 'str'>                                               13032    85335
<type 'tuple'>                                              8969    38402
<type 'Cfunc'>                                              1761    11931
<type 'code'>                                               1215     4871
<type 'function'>                                           1180     5189
<type 'dict'>                                                841     4897
<type 'builtin_function_or_method'>                          516     2781
<type 'int'>                                                 331     3597
<type 'wrapper_descriptor'>                                  295     1180
<type 'method_descriptor'>                                   236      944
<type 'classobj'>                                            145     1092
<type 'module'>                                              107      734
<type 'list'>                                                 94      440
<type 'type'>                                                 86     1967
<type 'getset_descriptor'>                                    84      336
<type 'weakref'>                                              75      306
<type 'float'>                                                73      312
<type 'member_descriptor'>                                    70      280
<type 'ufunc'>                                                52      364
<type 'instance'>                                             42      435
<type 'instancemethod'>                                       41      164
<class 'numarray.ufunc._BinaryUFunc'>                         25      187
<class 'numarray.ufunc._UnaryUFunc'>                          24      173
<type 'frame'>                                                 9       44
<type 'long'>                                                  7       28
<type 'property'>                                              6       25
<type 'PyCObject'>                                             4       20
<class 'unittest.TestSuite'>                                   3       31
<type 'file'>                                                  3       23
<type 'listiterator'>                                          3       12
<type 'bool'>                                                  2       41
<class 'random.Random'>                                        2       30
<type '_sre.SRE_Pattern'>                                      2        9
<type 'complex'>                                               2        8
<type 'thread.lock'>                                           2        8
<type 'NoneType'>                                              1     2371
<class 'unittest._TextTestResult'>                             1       16
<type 'ellipsis'>                                              1       12
<class '__main__.WideTreeTestCase'>                            1       11
<class 'tables.IsDescription.metaIsDescription'>               1       10
<class 'unittest.TestProgram'>                                 1        9
<class 'numarray.ufunc._ChooseUFunc'>                          1        8
<class 'unittest.TestLoader'>                                  1        7
<class 'unittest.TrackRefs'>                                   1        6
<class 'unittest.TextTestRunner'>                              1        6
<type 'NotImplementedType'>                                    1        6
<class 'numarray.ufunc._PutUFunc'>                             1        5
<class 'numarray.ufunc._TakeUFunc'>                            1        5
<class 'unittest._WritelnDecorator'>                           1        5
<type 'staticmethod'>                                          1        4
<type 'classmethod'>                                           1        4
<type 'classmethod_descriptor'>                                1        4
<type 'unicode'>                                               1        4
    7 __main__.WideTreeTestCase.test00_Leafs()
Second output of TrackRefs:
<type 'int'>                                                  37      218
<type 'type'>                                                  0       74
  212 __main__.WideTreeTestCase.test00_Leafs()
.
----------------------------------------------------------------------
Ran 1 test in 0.689s

OK
[21397 refs]
$

As you can see, for the second loop (first output of TrackRefs), a lot
of objects appear, but after the third loop (second output of
TrackRefs), much less appear (only objects of type "int" and
"type"). Besides, the increment of the total references for the second
loop is only 7 while for the third loop is 212. Finally, to add even
more confusion, these numbers are *totally* independent of the number
of iterations I put in the loops. You see 10 in the code, but you can
try with 100 (in one or all the loops) and you get exactly the same
figures.

I definitely think that I have made a bad implementation of the try:
code block, but I can't figure out what's going wrong.

I would appreciate some ideas.

Francesc Alted