[Python-Dev] refleak hunting fun!

[Michael Hudson]

Walter Dörwald <walter at livinglogic.de> writes:

> Michael Hudson wrote:
>
>> Prompted by finding my own refcounting leaks in test_descr, I'm now
>> looking to spread the misery :-)
>> I've hacked regrtest (hacks attached) to do some very crude
>> monitoring
>> of sys.gettotalrefcount().
>> Here's the output of a recent run:
>>
>  > [...]
>> test_codeccallbacks leaked 1107 references
>  > [...]
>> Some however, seem to be real.  test_sax, test_socket and
>> test_codeccallbacks seem to be the most alarming.  The TrackRef class
>> Tim posted a link to is likely to be indispensable in hunting these.
>
> The number for test_codeccallbacks really looks scary. Some of it is
> the result of the callback registry,

A ha.  I found a lot of things like that that I didn't know existed in
the last few days... some of the leaks I posted are due to use of
warnings.filterwarnings, too.

> but there seem to be real leaks here.

In a perverse kind of way, phew :-) Wouldn't want to have gone to all
this effort to uncover *nothing* but a bunch of false alarms...

> I've used the attached patch to unittest to get refcount diffs for
> individual test methods. Here are the results for test_codeccallbacks:
>
>    212 test_backslashescape
>      1 test_badandgoodbackslashreplaceexceptions
>      0 test_badandgoodignoreexceptions
>      0 test_badandgoodreplaceexceptions
>      0 test_badandgoodstrictexceptions
>      0 test_badandgoodxmlcharrefreplaceexceptions
>    271 test_badhandlerresults
>      0 test_badregistercall
>    153 test_callbacks
>      3 test_charmapencode
>    121 test_decodehelper
>     26 test_encodehelper
>    285 test_longstrings
>      0 test_lookup
>      9 test_relaxedutf8
>      1 test_translatehelper
>     11 test_unencodablereplacement
>    265 test_unicodedecodeerror
>    792 test_unicodeencodeerror
>      0 test_unicodetranslateerror
>     12 test_uninamereplace
>      0 test_unknownhandler
>     14 test_xmlcharnamereplace
>      0 test_xmlcharrefreplace
>      6 test_xmlcharrefvalues
>
> Bye,
>     Walter Dörwald
>
> Index: Lib/unittest.py
> ===================================================================
> RCS file: /cvsroot/python/python/dist/src/Lib/unittest.py,v
> retrieving revision 1.24
> diff -u -r1.24 unittest.py
> --- Lib/unittest.py	13 Jul 2003 15:18:12 -0000	1.24
> +++ Lib/unittest.py	14 Aug 2003 18:37:19 -0000
> @@ -376,7 +376,13 @@
>          for test in self._tests:
>              if result.shouldStop:
>                  break
> +            import gc, sys
> +            gc.collect()
> +            rc1 = sys.gettotalrefcount()
>              test(result)
> +            gc.collect()
> +            rc2 = sys.gettotalrefcount()
> +            print >>sys.stderr, "%5d %s" % (rc2-rc1, test)
>          return result
>  
>      def debug(self):

In general (not sure about these tests) you want to run each test a
few time to let things settle down before measuring the effect on
gettotalrefcount().

Cheers,
mwh

-- 
  same software, different verbosity settings (this one goes to
  eleven)                             -- the effbot on the martellibot