[Python-Dev] The untuned tunable parameter ARENA_SIZE

Tim Peters tim.peters at gmail.com
Sun Jun 4 10:46:10 EDT 2017

>> A virtual address space span of a terabyte could hold 1M pools, so
>> would "only" need a 1M/8 = 128KB bit vector.  That's minor compared to
>> a terabyte (one bit per megabyte).

> The virtual address space currently supported by x86-64 is 48 bits
> wide (spanning an address space of 2**48 bytes, that is 256 TB), so you
> would need a 2**(48-20) bits vector, i.e. 256 Mbits or 32 MB.
> Note Intel has plans to extend the virtual address space to 2**57 bytes:
> https://www.phoronix.com/scan.php?page=news_item&px=Intel-5-Level-Paging

Fill in the blanks ;-)  There's only a need for the bit vector to
cover the range of addresses _actually returned_ by the system for
arenas allocated so far (we're only trying to identify the memory
obmalloc _does_ control) .  I didn't spell that out, but it was
implicit in glosses like "(or if the bit address is out of the
vector's domain)".  That is, it's up to the bit vector implementation
to intelligently represent what's almost always going to be a
relatively tiny slice of a theoretically massive address space.

So my observation about a terabyte is to be read as applying to a case
in which obmalloc has _actually allocated_ arenas spanning a terabyte
of address space.(2**14 arenas of 64MB each, and if the system is kind
enough to keep them contiguous).  Since that's about 100 times more
address space than any Python program I've run actually needed, it's a
bare minimum for a thought experiment.

More information about the Python-Dev mailing list