2013/8/29 Victor Stinner <victor.stinner@gmail.com>:
Charles-François Natali and Serhiy Storchaka asked me to add this module somewhere in Python 3.4: "how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?"
There are two reasons I think it would be a great addition: - since OOM conditions are - almost - never tested, the OOM handling code is - almost - always incorrect: indeed, Victor has found and fixed several dozens crashes thanks to this module - this module is actually really simple (~150 LOC) I have two comments on the API: 1) failmalloc.enable(range: int=1000): schedule a memory allocation failure in random.randint(1, range) allocations. That's one shot, i.e. only one failure will be triggered. So if this failure occurs in a place where the code is prepared to handle MemoryError (e.g. bigmem tests), no failure will occur in the remaining test. It would be better IMO to repeat this (i.e. reset the next failure counter), to increase the coverage. 2) It's a consequence of 1): since only one malloc() failure is triggered, it doesn't really reflect how a OOM condition would appear in real life: usually, it's either because you've exhausted your address space or the machine is under memory pressure, which means that once you've hit OOM, you're likely to encounter it again on subsequent allocations, for example if your OOM handling code allocates new memory (that's why it's so complicated to properly handle OOM, and one might want to use "memory parachutes"). It might be interesting to be able to pass an absolute maximum memory usage, or an option where once you've triggered an malloc() failure, you record the current memory usage, and use it as ceiling for subsequent allocations.