[Python-Dev] The untuned tunable parameter ARENA_SIZE

Antoine Pitrou solipsis at pitrou.net
Fri Jun 2 05:38:16 EDT 2017

On Thu, 1 Jun 2017 20:21:12 -0700
Larry Hastings <larry at hastings.org> wrote:
> On 06/01/2017 02:50 AM, Antoine Pitrou wrote:
> > Another possible strategy is: allocate several arenas at once (using a
> > larger mmap() call), and use MADV_DONTNEED to relinquish individual
> > arenas.  
> Thus adding a *fourth* layer of abstraction over memory we get from the OS?
>     block -> pool -> arena -> "multi-arena" -> OS

Not a layer of abstraction, just over-allocation.  You would
over-allocate arenas like you over-allocate a list object's elements
storage (though probably using a different over-allocation
strategy ;-)).  That would reduce the number of mmap() calls (though
not necessarily the number of munmap() or madvise() calls), and also
provide more opportunities for the kernel to allocate a large page.

But you would still handle individual arenas in the allocation code.

> Y'know, this might actually make things faster.  These multi-arenas 
> could be the dynamically growing thing Victor wants to try.  We allocate 
> 16mb, then carve it up into arenas (however big those are), then next 
> time allocate 32mb or what have you.

I hope those are not the actual numbers you're intending to use ;-)
I still think that allocating more than 1 or 2MB at once would be
foolish.  Remember this is data that's going to be carved up into
(tens of) thousands of small objects.  Large objects eschew the small
object allocator (not to mention that third-party libraries like Numpy
may be using different allocation routines when they allocate very
large data).



More information about the Python-Dev mailing list