Ref counting fun (was: Re: [Python-Dev] CALL_ATTR patch)

[Michael Hudson]

"Tim Peters" <tim.one at comcast.net> writes:

> [Michael Hudson]
>> ...
>> And then I find out running test_descr in a loop leaks 166(!)
>> references *anyway* (run the attached blah.py in a debug build, and
>> some print test or other that I don't understand fails when run like
>> this, so I fiddled it)
>>
>> Should I have been expecting this, or is it time for some ref-leak
>> hunting?  Actually, the process seems to grow -- slowly -- when you
>> loop, so it's probably a real problem.  I gather it's best to ask
>> about this sort of thing when Tim is off sick from work :-)
>
> Only if you want help <wink>.

Actually, I found the worst culprit this morning.  Some dunderhead had
got the decrefs wrong on error return from type_set_bases...

I'll check my fix in momentarily.

> Since Zope has worse leak problems than
> Python, Zope testing has more tools to dig into it:
>
>     http://cvs.zope.org/Zope3/test.py
>
> That's the Zope3 unittest test driver.  The TrackRefs class is a handy
> little beast, which uses a debug build's sys.getobjects() to keep track of
> how many objects of each *type* exist across test runs, and how many
> refcounts total they account for.  After the first few iterations, TrackRefs
> doesn't itself distort the results it prints.  It will tell you the types of
> the objects that are leaking, and also whether objects are actually leaking,
> or that merely refcounts to existing objects are leaking (e.g., a common
> kind of "leak" is to forget to decref Py_None, but memory doesn't actually
> grow then).

Cool!  I'd blundered my way to something like an ad hoc version of
this, next time I'll use something that someone's actually thought
about :-)

> Do use 2.3 if you try this.

Well, yeah.  Turned out the worst leaks weren't in 2.2 code anyway.

> The "list of all objects" before 2.3 turned out to be missing many
> objects that tend to leak (None, booleans, any number of type
> objects, essentially all statically allocated objects), and far
> fewer are missing in 2.3.
>
> It's typically easy to find out which types of objects are leaking this way.
> Then it's typical to add more code to TraceRefs.update() to get more
> details.  In Zope use, it's surprising how often that still left us
> scratching our heads!  But Zope has lots of extension types coded in C,
> massive interconnections among them, and none of them originally
> participated in cyclic gc.  Python leaks are usually easier to plug (for
> example, in Python, they're almost always in new code).

Bingo!

I did all these tests incorporating my safety fixes to
PyObject_GenericGetAttr so I think they're probably solid, too.

Cheers,
mwh

-- 
  The PROPER way to handle HTML postings is to cancel the article,
  then hire a hitman to kill the poster, his wife and kids, and fuck
  his dog and smash his computer into little bits. Anything more is
  just extremism.                                 -- Paul Tomblin, asr