The OOM-Killer vs. Python

[Andrae Muys]

jimd at vega.starshine.org (Jim Dennis) wrote in message news:<a7msbn$110t$1 at news.idiom.com>...
>  Personally I think the various libraries, compilers and applications
>  which are blindly allocating far more memory then they actually use are
>  really the heart of the whole overcommit problem.  That might not be 
>  Python (or it might be indirectly due to Python's use of some libraries 
>  on your system, possibly it could be glibc and libm bloating).
>  Unfortunately memory is a shared resource, so it only takes one bad
>  appl. (so to speak) to ruin the whole basket for all of us.

Unfortunately the alternative is a substantial performance hit.

Note that with overcommit enabled, applications that over allocate but
don't use the memory won't trigger the OOM, which is triggered by VM
memory pressure.

The issue with overcommit isn't so much that applications get killed
when the system actually does run out of memory, but that applications
get killed asynchronously accessing successfully malloced memory (ie,
believed a promise the OS made, which it couldn't deliver on) when
they may have been perfectly willing to do a graceful recovery if they
had been given -ENOMEM on malloc.

The problem with not overcommitting is that system calls are
signifigantly slower then function calls, so malloc libraries normally
block allocate memory to save reduce the number of syscalls made. 
Another popular optimisation trick is to use anonymous mmapping of
/dev/zero to provide memory pools for malloc.  These are routinely
over-committed.

OTOH, the clincher is that regardless of any overcommit flag, if you
permit them (and they are very very useful :) private, rw, mmap's can
always leave you with undefined current memory allocation, and
therefore vunerable to overcommitting.

Andrae Muys

P.S. I may be mistaken, but IIRC this was the final resolution of the
overcommit flag thread.  ie. Any suggestion that you can actually
avoid overcommitting provides a false sense of security or tends to
break the very applications that make it an issue, so you're better
off not pretending to offer it or at least making it an obscure config
option for people who understand the consequences of enabling it.