[Python-Dev] The untuned tunable parameter ARENA_SIZE
siddhesh at gotplt.org
Thu Jun 1 04:55:22 EDT 2017
On Thursday 01 June 2017 01:27 PM, Victor Stinner wrote:
> The GNU libc malloc uses a variable threshold to choose between sbrk()
> (heap memory) or mmap(). It starts at 128 kB or 256 kB, and then is
> adapted depending on the workload (I don't know how exactly).
The threshold starts at 128K and increases whenever an mmap'd block is
freed. For example, if the program allocates 2M (which is returned
using mmap) and then frees that block, glibc malloc assumes that 2M
blocks will be needed again and optimizes that allocation by increasing
the threshold to 2M.
This works well in practice for common programs but it has been known to
cause issues in some cases, which is why there's MALLOC_MMAP_THRESHOLD_
to fix the threshold.
> I already read that CPU support "large pages" between 2 MB and 1 GB,
> instead of just 4 kB. Using large pages can have a significant impact
> on performance. I don't know if we can do something to help the Linux
> kernel to use large pages for our memory? I don't know neither how we
> could do that :-) Maybe using mmap() closer to large pages will help
> Linux to join them to build a big page? (Linux has something magic to
> make applications use big pages transparently.)
There's MAP_HUGETLB and friends for mmap flags, but it's generally
better to just let the kernel do this for you transparently (using
Transparent Huge Pages) by making sure that your arena allocations are
either contiguous or big enough.
More information about the Python-Dev