[Python-Dev] A "new" kind of leak

[Tim Peters]

[Guido]
> You can get the same effect with sys._getframe() instead of
> inspect.currentframe().

Sure.

> ...
> If Neil won't, someone else can do it.  Adding GC to an object isn't
> hard, and the tp_traverse/tp_clear implementations probably port right
> over.

As Neil said, Everything Was Different in 2.1, and frames are especially
delicate.  I'll add that I expect adding traceback objects to GC in 2.1 too
would plug a lot more real-life leaks than adding frameobjects alone.  And
then there are bound method objects, and ... Neil did a lot of work here for
2.2.

>> Question #3: Can we use pymalloc to manage frame objects instead?

> How would it help?

pymalloc is generally faster than platform malloc/free.

>> Question #4: Can we just get rid of the free list?  I don't think so
>> -- a frame object is needed on every function invocation, and saving
>> malloc/free overhead is a real win.

> Yup.  Try it though.

Here's a one-liner patch that's close enough for timing purposes.  But
please note that the relative speed of such changes varies widely across
platforms, and the more we fiddle, the greater the odds of creating a real
problem for someone (also of creating a real benefit for someone -- but
nobody will remember that).  I'm rarely hesitant to try stuff like this in
current CVS, because developers are beating on the head daily, across a
variety of platforms.  Changes checked in to a maintenance branch are likely
to get tested for the second time <wink> in their life by Python's end
users.

Index: frameobject.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Objects/frameobject.c,v
retrieving revision 2.60
diff -c -r2.60 frameobject.c
*** frameobject.c       28 Mar 2002 20:34:59 -0000      2.60
--- frameobject.c       13 Apr 2002 02:09:01 -0000
***************
*** 91,98 ****
        Py_XDECREF(f->f_exc_type);
        Py_XDECREF(f->f_exc_value);
        Py_XDECREF(f->f_exc_traceback);
!       f->f_back = free_list;
!       free_list = f;
        Py_TRASHCAN_SAFE_END(f)
  }

--- 91,97 ----
        Py_XDECREF(f->f_exc_type);
        Py_XDECREF(f->f_exc_value);
        Py_XDECREF(f->f_exc_traceback);
!       PyObject_GC_Del(f);
        Py_TRASHCAN_SAFE_END(f)
  }


>> An effective solution would be to bound the size of the frameobject
>> free list ...

> Good idea.

It's something tuples already do in their free lists.  Do we have any other
gc types using free lists?  I expect the same kind of leak could be provoked
wrt any gc container using an unbounded free list.

>> Question #5: What's a good value for N? [1000]
>> ...

> I bet even lowering it to 50 would work just as well in practice.

For most programs, yes, although I expect Zope routinely gets 50 levels
deep, so would prefer the geometric mean <wink -- the GM is always the right
way to compromise>:

>>> math.sqrt(1000 * 50)
223.60679774997897
>>>

I'll stay up all night deciding whether 223 or 224 is better.

> I'm somewhat wary of solutions that use a really high bound for cases
> like this; the code that deals with exceeding the bound would almost
> never get executed, so bugs there would linger forever.

Good point.

>> Question #6: If we agree it's a good idea to put on a bound in 2.3,
>> is it a good idea to backport the bound?  I think yes to 2.2, but no
>> to 2.1.  In 2.1 the frame-cycle cases leak anyway.

> But it was reported for 2.1, right?

Yes, but it was reported as a request to enhance the docs.  I'm not sure
how, and asked the OP to elaborate (but haven't yet heard back).

> I could imagine making this a showcase of the new religion: more
> support for older versions.  2.1 is the limit though.

I'm afraid it may also serve as a showcase for how a one-hour bugfix can
spread to consume days.  Cut this one off at 2.2 instead and it's not a lot
of extra work.