[Python-Dev] The untuned tunable parameter ARENA_SIZE

Tim Peters tim.peters at gmail.com
Fri Jun 2 14:23:05 EDT 2017

[INADA Naoki <songofacandy at gmail.com>]
> ...
> Since current pool size is 4KB and there is pool_header in pool,
> we can't allocate 4KB block from pool.
> And if support 1KB block, only 3KB of 4KB can be actually used.
> I think 512 bytes / 4KB (1/8) is good ratio.
> Do you mean increase pool size?
> How about adding configure option like server-mode?
> POOL_SIZE  (16*1024)    // 4x
> ARENA_SIZE  (2*1024*1024)   // 8x, and same to huge page size.

While I would like to increase the pool size, it's fraught with
danger.  The problem:  Py_ADDRESS_IN_RANGE has to figure out whether
an "arbitrary" address is - or is not - controlled by Python's
small-object allocator.  The excruciating logic is spelled out at
length in obmalloc.c.

As is, the code reads up memory near "the start" of a pool to get the
pool's belief about which arena it's in.  If the memory is not in fact
controlled by pymalloc, this can be anything, even uninitialized
trash.  That's fine (as explained in the comments) - but that memory
_must_ be readable.

And that's why POOL_SIZE is set to 4K:  it's the smallest page size
across all the OSes Python is known to run on.  If pymalloc is handed
an "arbitrary" (but valid) address, the entire OS page containing that
address is readable.

If, e.g., an OS allocates 4K pages, but Python's POOL_SIZE is 8K
(anything bigger than the OS's page allocation unit), then perhaps the
OS page at 16K is unallocated, but the page at 20K is.  pymalloc sees
an address at 21K.  As is, pymalloc reads the putative arena index at
20K, which is fine.  But if POOL_SIZE were 8K, it would try to read
the putative arena index at 16K, and - boom! - segfault.

This failure mode would be rare but - of course - catastrophic when it occurs.

It would be nice to find a different way for pymalloc to figure out
which addresses belong to it.  The excruciating Py_ADDRESS_IN_RANGE
manages to do it in small constant (independent of the number of
arenas and pools in use) time, which is its only virtue ;-)

More information about the Python-Dev mailing list