[Python-Dev] A "new" kind of leak

[Tim Peters]

    http://www.python.org/sf/543148

reports a prodiguous leak under 2.1 via

import inspect

def leak():
    frame = inspect.currentframe()

while 1:
    leak()

This isn't surprising, since leak() assigns the current frame to a local in
the same frame, creating a reference cycle, and frame objects weren't added
to cyclic gc until 2.2.  Question #1:  Is adding frame objects to cyclic gc
a candidate for 2.1 backporting?  Question #1a:  Who's gonna pay Neil to
spend his free time doing that <0.5 wink>?

What is surprising is that it also leaks *memory* in 2.2 and current CVS,
although it doesn't leak *objects* anymore, at least not in the sense you're
thinking of:  gc does find and reclaim all the frames.  Yet the process size
grows about 50MB/minute on my box anyway.

What happens:  frames have their own free list.  gc is triggered by a net
excess of memory (not free list) allocations over deallocations.  When
enough frames in cycles have been created, that excess triggers, and gc
cleans up all of them.  However, the memory isn't freed, instead all the
frames end up on the frame object free list.  As the program goes on, the
entire free list is chewed up to create more frames in cycles, and then
we're back where we started:  it again takes the same excess of new frame
allocations to trigger gc again.  Now there are twice as many frames in
cycles as before (call it 2*F), and gc again cleans them all up, and they
all wind up in the frame object free list -- which is now twice as big as
before.  The program goes on, and creates 2*F frames in cycles out of the
free list before allocating any new memory, and it takes an additional F
frames' worth of new frame allocations to trigger gc again.

Etc.  As time goes on, the size of the frameobject free list grows without
bound.

Question #2:  Is this OK?  I think not, as this is a leak in reality.

Question #3:  Can we use pymalloc to manage frame objects instead?  Alas,
no:  frame objects are too big (on my box, over 350 bytes for a minimal
frame object, and pymalloc's limit is 256).

Question #4:  Can we just get rid of the free list?  I don't think so -- a
frame object is needed on every function invocation, and saving malloc/free
overhead is a real win.

An effective solution would be to bound the size of the frameobject free
list:  pick some maximum N.  When frame_dealloc sees that there are already
N frames on the free list, it frees the frame memory instead of adding it to
the free list.

Question #5:  What's a good value for N?  In the absence of pathologies due
to cycles, and in the absence of generators, the number of live frame
objects is equal to the current call-stack depth.  Since we're just trying
to bound worst-case quadratic pathologies in cycle cases, the bound can be
high -- say, 1000.  A non-pathological program creating thousands of
simultaneously active short-lived generators would suffer, but I'm not
prepared to go to my grave claiming that's not also pathological <wink>.

Question #6:  If we agree it's a good idea to put on a bound in 2.3, is it a
good idea to backport the bound?  I think yes to 2.2, but no to 2.1.  In 2.1
the frame-cycle cases leak anyway.