[Python-Dev] The untuned tunable parameter ARENA_SIZE
larry at hastings.org
Fri Jun 2 15:31:19 EDT 2017
On 06/02/2017 02:46 AM, Victor Stinner wrote:
> I would be curious of another test: use pymalloc for objects larger
> than 512 bytes. For example, allocate up to 4 KB?
> In the past, we already changed the maximum size from 256 to 512 to
> support most common Python objects on 64-bit platforms. Since Python
> objects contain many pointers: switching from 32 bit to 64 bit can
> double the size of the object in the worst case.
You've already seen Tim Peters' post about why we must leave pool size
set to 4k. Obviously This in turn means using obmalloc for larger
objects will mean more and more wasted memory.
For example, let's say we use obmalloc for allocations of 2048 bytes.
Pool size is 4096 bytes, and there's a 48-byte "pool_header" structure
on the front (on 64-bit platforms, if I counted right). So there are
only 4048 bytes usable per pool. After the first 2048 allocation, we're
left with 2000 bytes at the end. You can't use that memory for another
allocation class, that's impossible given obmalloc's design. So that
2000 bytes is just wasted.
Currently obmalloc's maximum allocation size is 512 bytes; after 7
allocations, this leaves 464 wasted bytes at the end. Which isn't
*great* exactly but it's only 11% of the overall allocated memory.
Anyway, I'm not super excited by the prospect of using obmalloc for
larger objects. There's an inverse relation between the size of
allocation and the frequency of allocation. In Python there are lots of
tiny allocations, but fewer and fewer as the size increases. (A
similarly-shaped graph to what retailers call the "long tail".) By no
small coincidence, obmalloc is great at small objects, which is where we
needed the help most. Let's leave it at that.
A more fruitful endeavor might be to try one of these fancy new
third-party allocators in CPython, e.g. tcmalloc, jemalloc. Try each
with both obmalloc turned on and turned off, and see what happens to
performance and memory usage. (I'd try it myself, but I'm already so
far behind on watching funny cat videos.)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev