[Python-Dev] suggestion for smarter garbage collection in function of size (gc.set_collect_mem_growth(2))

[Andrea Arcangeli]

Hello,

I run into a problem recently with a reconnectingclientfactory with
twisted while write some spare time software, that turned out to be a gc
inefficiency.

In short the protocol memory wasn't released after the reconnect and the
protocol had about 50M attached to it. So with frequent reconnects the
rss of the task grown to >1G and it pushed the system into heavy swap.
In reality only 50M were allocated, so 950M were wasted by python.

A gc.collect() explicit invocation at every retry of the factory fixed
it. However this means there is significant room for improvement in the
default behaviour of the python gc.

In short the real bug is that the python gc isn't working in function of
size in any way. It doesn't matter the number of objects, it matters
their size!

The whole story can be found in this thread:

	http://twistedmatrix.com/pipermail/twisted-python/2005-December/012233.html

My suggestion to fix this problem in autopilot mode (without requiring
explicit gc.collect()) is to invoke a gc.collect() (or anyway to go deep
down freeing everything possible) at least once every time the amount of
anonymous memory allocated by the interpreter doubles. The tunable should
be a float >= 1. When the tunable is 1 the feature is disabled (so it
works like current python today). Default should be 2 (which means to
invoke gc.collect() after a 100% increase of the anonymous memory
allocated by the interpreter). We could also have yet another threshold
that sets a minimum of ram after which this heuristic in function of
size kicks in, but it's not very important and it may not be worth it
(whem little memory is involved gc.collect() should be fast anyway).

To implement this we need two hooks, in the malloc and free that
allocate python objects. Then we have to store the minimum of this
value (i.e. the last minimum of memory allocated by the interpreter).

The algorithm I'd suggest is like this (supposedly readable pseudocode):

	gc.set_collect_mem_growth(v):
		assert float(v) >= 1
		gc.collect_mem_growth = v

	python_malloc(size):
		ram_size += size
		if ram_size > min_ram_size * gc.collect_mem_growth:
			gc.collect() # python_free runs inside it
			min_ram_size = ram_size # ram size after gc.collect()

	python_free(size):
		ram_size -= size
		min_ram_size = min(min_ram_size, size)

The overhead of this should be zero, and it'll fix my testcase just
fine. I believe the default should be 2 (equivalent to 100% growth of
rss to trigger a full collect) even though it alters the behaviour of
the gc, I think it's a bug that so much memory can be leaked when it
could be reclaimed istantly.

I wouldn't change other parameters, this heuristic in function of size
would be completely orthogonal and disconnected by the current
heuristics in function of the number of elements.
 
It has taken me a day and precious help from the twisted folks to
realize it wasn't a memleak in my twisted spare time application (but
well, it was good since I learnt about the fact I created an heisenbug
by using __del__ to debug the apparent memleak ;).

Thanks.