Ref counting fun (was: Re: [Python-Dev] CALL_ATTR patch)

Michael Hudson mwh at
Thu Aug 7 15:08:52 EDT 2003

"Tim Peters" < at> writes:

> [Michael Hudson]
>> ...
>> And then I find out running test_descr in a loop leaks 166(!)
>> references *anyway* (run the attached in a debug build, and
>> some print test or other that I don't understand fails when run like
>> this, so I fiddled it)
>> Should I have been expecting this, or is it time for some ref-leak
>> hunting?  Actually, the process seems to grow -- slowly -- when you
>> loop, so it's probably a real problem.  I gather it's best to ask
>> about this sort of thing when Tim is off sick from work :-)
> Only if you want help <wink>.

Actually, I found the worst culprit this morning.  Some dunderhead had
got the decrefs wrong on error return from type_set_bases...

I'll check my fix in momentarily.

> Since Zope has worse leak problems than
> Python, Zope testing has more tools to dig into it:
> That's the Zope3 unittest test driver.  The TrackRefs class is a handy
> little beast, which uses a debug build's sys.getobjects() to keep track of
> how many objects of each *type* exist across test runs, and how many
> refcounts total they account for.  After the first few iterations, TrackRefs
> doesn't itself distort the results it prints.  It will tell you the types of
> the objects that are leaking, and also whether objects are actually leaking,
> or that merely refcounts to existing objects are leaking (e.g., a common
> kind of "leak" is to forget to decref Py_None, but memory doesn't actually
> grow then).

Cool!  I'd blundered my way to something like an ad hoc version of
this, next time I'll use something that someone's actually thought
about :-)

> Do use 2.3 if you try this.

Well, yeah.  Turned out the worst leaks weren't in 2.2 code anyway.

> The "list of all objects" before 2.3 turned out to be missing many
> objects that tend to leak (None, booleans, any number of type
> objects, essentially all statically allocated objects), and far
> fewer are missing in 2.3.
> It's typically easy to find out which types of objects are leaking this way.
> Then it's typical to add more code to TraceRefs.update() to get more
> details.  In Zope use, it's surprising how often that still left us
> scratching our heads!  But Zope has lots of extension types coded in C,
> massive interconnections among them, and none of them originally
> participated in cyclic gc.  Python leaks are usually easier to plug (for
> example, in Python, they're almost always in new code).


I did all these tests incorporating my safety fixes to
PyObject_GenericGetAttr so I think they're probably solid, too.


  The PROPER way to handle HTML postings is to cancel the article,
  then hire a hitman to kill the poster, his wife and kids, and fuck
  his dog and smash his computer into little bits. Anything more is
  just extremism.                                 -- Paul Tomblin, asr

More information about the Python-Dev mailing list