[Python-Dev] suggestion for smarter garbage collection in function of size (gc.set_collect_mem_growth(2))

[Tim Peters]

[Martin v. Löwis]
> ...
> One challenge is that PyObject_GC_Del doesn't know how large the
> memory block is that is being released. So it is difficult to find out how
> much memory is being released in the collection.

"Impossible in some cases" is accurate.  When pymalloc isn't enabled,
all these things call the platform malloc/free directly, and there's
no portable/standard way to find out anything from those.  When
pymalloc is enabled, PyObject_GC_Del could be taught whether pymalloc
controls the block being freed, and, when so, how to suck up the
block's size index from the block's pool header; but when pymalloc
doesn't control the memory being freed, it's the same as if pymalloc
were not enabled.

[Neil Schemenauer]
> Another idea would be to add accounting to the PyMem_* interfaces.
> It could be that most memory is used by objects that are not tracked
> by the GC (e.g. strings).

I still expect this old code in pymem.h to go away for Python 2.5:

/* In order to avoid breaking old code mixing PyObject_{New, NEW} with
   PyMem_{Del, DEL} and PyMem_{Free, FREE}, the PyMem "release memory"
   functions have to be redirected to the object deallocator. */
#define PyMem_FREE           	PyObject_FREE

When goes away, PyMem_FREE will resolve directly to the platform
free(), and will no longer have even accidental relationships to any
memory involved in cyclic gc.

> I guess you still have the same problem in that PyMem_Free may not know how
> large the memory block is.

It will be more the case that we can guarantee it won't know -- but
since direct uses of malloc/free have no useful relationship to cyclic
gc behavior, the OP shouldn't care about that.

In any case, the OP's original "the overhead of this should be zero"
claim isn't credible (I checked, and there _still_ won't be free
lunches in 2006 -- unless you work at Google ;-)).