[Python-Dev] Add a new tracemalloc module to trace memory allocations

[Charles-François Natali]

2013/8/29 Victor Stinner <victor.stinner at gmail.com>:
> Charles-François Natali and Serhiy Storchaka asked me to add this
> module somewhere in Python 3.4: "how about adding pyfailmalloc to the
> main repo (maybe under Tools), with a script making it easy to run the
> tests suite with it enabled?"

There are two reasons I think it would be a great addition:
- since OOM conditions are - almost - never tested, the OOM handling
code is - almost - always incorrect: indeed, Victor has found and
fixed several dozens crashes thanks to this module
- this module is actually really simple (~150 LOC)

I have two comments on the API:
1)
failmalloc.enable(range: int=1000): schedule a memory allocation
failure in random.randint(1, range) allocations.

That's one shot, i.e. only one failure will be triggered. So if this
failure occurs in a place where the code is prepared to handle
MemoryError (e.g. bigmem tests), no failure will occur in the
remaining test. It would be better IMO to repeat this (i.e. reset the
next failure counter), to increase the coverage.

2)
It's a consequence of 1): since only one malloc() failure is
triggered, it doesn't really reflect how a OOM condition would appear
in real life: usually, it's either because you've exhausted your
address space or the machine is under memory pressure, which means
that once you've hit OOM, you're likely to encounter it again on
subsequent allocations, for example if your OOM handling code
allocates new memory (that's why it's so complicated to properly
handle OOM, and one might want to use "memory parachutes").
It might be interesting to be able to pass an absolute maximum memory
usage, or an option where once you've triggered an malloc() failure,
you record the current memory usage, and use it as ceiling for
subsequent allocations.