[issue21233] Add *Calloc functions to CPython memory allocation API

Nathaniel Smith report at bugs.python.org
Sun Apr 27 20:27:31 CEST 2014


Nathaniel Smith added the comment:

Right, python3 -c 'b"x" * (2 ** 48)' does give an instant MemoryError for me. So I was wrong about it being the VM limit indeed.

The documentation on this is terrible! But, if I'm reading this right:
   http://lxr.free-electrons.com/source/mm/util.c#L434
the actual rules are:

overcommit mode 1: allocating a VM range always succeeds.
overcommit mode 2: (Slightly simplified) You can allocate total VM ranges up to (swap + RAM * overcommit_ratio), and overcommit_ratio is 50% by default. So that's a bit odd, but whatever. This is still entirely a limit on VM size.
overcommit mode 0 ("guess", the default): when allocating a VM range, the kernel imagines what would happen if you immediately used all those pages. If that would put you OOM, then we fall back to mode 2 rules. If that would *not* put you OOM, then the allocation unconditionally succeeds.

So yeah, touching pages can affect whether a later malloc returns ENOMEM.

I'm not sure any of this actually matters in the Python case though :-). There's still no reason to go touching pages pre-emptively just in case we might write to them later -- all that does is increase the interpreter's memory footprint, which can't help anything. If people are worried about overcommit, then they should turn off overcommit, not try and disable it on a piece-by-piece basis by trying to get individual programs to memory before they need it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21233>
_______________________________________


More information about the Python-bugs-list mailing list