The REALLY bad thing about Python lists ..

[Florian Weimer]

Hrvoje Niksic <hniksic at iskon.hr> writes:

> "Tim Peters" <tim_one at email.msn.com> writes:
> 
> > Actually, malloc under Linux "overcommits": it's possible for malloc
> > to return a non-NULL value, yet when you reference the memory later,
> > it blows up.
> 
> This is not really a (mis)feature of malloc, but of the underlying OS
> mechanisms (sbrk/mmap).  And I don't think it's enabled by default
> anymore.  Under 2.2 kernels, you can say something like:
> 
> $ cat /proc/sys/vm/overcommit_memory
> 0
> 
> ...meaning that overcommitting is not allowed.

Contrary to popular belief, you cannot switch off memory
overcommitment on Linux.  A memory region allocated by an anonymous
mmap() (this is what malloc() uses behind the scenes) isn't backed by
physical storage (RAM or swap space) until the process writes to it
for the first time.  The only exception to this rule occurs if the
process calls mlock() or mlockall().

If set to zero, the overcommit_memory flag activates a crude heuristic
(even for a heuristic it's crude) which is designed to prevent amok
processes from taking down the whole system: an anonymous mmap()
request is denied if it exceeds the memory (RAM and swap) which is
available _at the moment of the request_.  A successful anonymous
mmap() reduces the amount of available memory only by small amount,
though.  The real reduction doesn't happen until the allocated virtual
memory region is backed by physical storage.  As a result, you can
allocate gigabytes of memory even if overcommitment (in the Linux
terminology) is turned off.

Warning: The kernel documentation of the overcommit_memory sysctl flag
is badly wrong.  Please take this into account before you start
flaming me that my description is horribly inaccurate. ;)