[Python-Dev] The untuned tunable parameter ARENA_SIZE
victor.stinner at gmail.com
Fri Jun 2 05:46:43 EDT 2017
I would be curious of another test: use pymalloc for objects larger
than 512 bytes. For example, allocate up to 4 KB?
In the past, we already changed the maximum size from 256 to 512 to
support most common Python objects on 64-bit platforms. Since Python
objects contain many pointers: switching from 32 bit to 64 bit can
double the size of the object in the worst case.
2017-06-01 9:38 GMT+02:00 Larry Hastings <larry at hastings.org>:
> When CPython's small block allocator was first merged in late February 2001,
> it allocated memory in gigantic chunks it called "arenas". These arenas
> were a massive 256 KILOBYTES apiece.
> This tunable parameter has not been touched in the intervening 16 years.
> Yet CPython's memory consumption continues to grow. By the time a current
> "trunk" build of CPython reaches the REPL prompt it's already allocated 16
> I propose we make the arena size larger. By how much? I asked Victor to
> run some benchmarks with arenas of 1mb, 2mb, and 4mb. The results with 1mb
> and 2mb were mixed, but his benchmarks with a 4mb arena size showed
> measurable (>5%) speedups on ten benchmarks and no slowdowns.
> What would be the result of making the arena size 4mb?
> CPython could no longer run on a computer where at startup it could not
> allocate at least one 4mb continguous block of memory.
> CPython programs would die slightly sooner in out-of-memory conditions.
> CPython programs would use more memory. How much? Hard to say. It depends
> on their allocation strategy. I suspect most of the time it would be < 3mb
> additional memory. But for pathological allocation strategies the
> difference could be significant. (e.g: lots of allocs, followed by lots of
> frees, but the occasional object lives forever, which means that the arena
> it's in can never be freed. If 1 out of ever 16 256k arenas is kept alive
> this way, and the objects are spaced out precisely such that now it's 1 for
> every 4mb arena, max memory use would be the same but later stable memory
> use would hypothetically be 16x current)
> Many programs would be slightly faster now and then, simply because we call
> malloc() 1/16 as often.
> What say you? Vote for your favorite color of bikeshed now!
> Python-Dev mailing list
> Python-Dev at python.org
More information about the Python-Dev