Ref counting fun (was: Re: [Python-Dev] CALL_ATTR patch)
mwh at python.net
Thu Aug 7 15:08:52 EDT 2003
"Tim Peters" <tim.one at comcast.net> writes:
> [Michael Hudson]
>> And then I find out running test_descr in a loop leaks 166(!)
>> references *anyway* (run the attached blah.py in a debug build, and
>> some print test or other that I don't understand fails when run like
>> this, so I fiddled it)
>> Should I have been expecting this, or is it time for some ref-leak
>> hunting? Actually, the process seems to grow -- slowly -- when you
>> loop, so it's probably a real problem. I gather it's best to ask
>> about this sort of thing when Tim is off sick from work :-)
> Only if you want help <wink>.
Actually, I found the worst culprit this morning. Some dunderhead had
got the decrefs wrong on error return from type_set_bases...
I'll check my fix in momentarily.
> Since Zope has worse leak problems than
> Python, Zope testing has more tools to dig into it:
> That's the Zope3 unittest test driver. The TrackRefs class is a handy
> little beast, which uses a debug build's sys.getobjects() to keep track of
> how many objects of each *type* exist across test runs, and how many
> refcounts total they account for. After the first few iterations, TrackRefs
> doesn't itself distort the results it prints. It will tell you the types of
> the objects that are leaking, and also whether objects are actually leaking,
> or that merely refcounts to existing objects are leaking (e.g., a common
> kind of "leak" is to forget to decref Py_None, but memory doesn't actually
> grow then).
Cool! I'd blundered my way to something like an ad hoc version of
this, next time I'll use something that someone's actually thought
> Do use 2.3 if you try this.
Well, yeah. Turned out the worst leaks weren't in 2.2 code anyway.
> The "list of all objects" before 2.3 turned out to be missing many
> objects that tend to leak (None, booleans, any number of type
> objects, essentially all statically allocated objects), and far
> fewer are missing in 2.3.
> It's typically easy to find out which types of objects are leaking this way.
> Then it's typical to add more code to TraceRefs.update() to get more
> details. In Zope use, it's surprising how often that still left us
> scratching our heads! But Zope has lots of extension types coded in C,
> massive interconnections among them, and none of them originally
> participated in cyclic gc. Python leaks are usually easier to plug (for
> example, in Python, they're almost always in new code).
I did all these tests incorporating my safety fixes to
PyObject_GenericGetAttr so I think they're probably solid, too.
The PROPER way to handle HTML postings is to cancel the article,
then hire a hitman to kill the poster, his wife and kids, and fuck
his dog and smash his computer into little bits. Anything more is
just extremism. -- Paul Tomblin, asr
More information about the Python-Dev