[Python-Dev] The untuned tunable parameter ARENA_SIZE

Victor Stinner victor.stinner at gmail.com
Fri Jun 2 05:46:43 EDT 2017


I would be curious of another test: use pymalloc for objects larger
than 512 bytes. For example, allocate up to 4 KB?

In the past, we already changed the maximum size from 256 to 512 to
support most common Python objects on 64-bit platforms. Since Python
objects contain many pointers: switching from 32 bit to 64 bit can
double the size of the object in the worst case.

Victor

2017-06-01 9:38 GMT+02:00 Larry Hastings <larry at hastings.org>:
>
>
> When CPython's small block allocator was first merged in late February 2001,
> it allocated memory in gigantic chunks it called "arenas".  These arenas
> were a massive 256 KILOBYTES apiece.
>
> This tunable parameter has not been touched in the intervening 16 years.
> Yet CPython's memory consumption continues to grow.  By the time a current
> "trunk" build of CPython reaches the REPL prompt it's already allocated 16
> arenas.
>
> I propose we make the arena size larger.  By how much?  I asked Victor to
> run some benchmarks with arenas of 1mb, 2mb, and 4mb.  The results with 1mb
> and 2mb were mixed, but his benchmarks with a 4mb arena size showed
> measurable (>5%) speedups on ten benchmarks and no slowdowns.
>
> What would be the result of making the arena size 4mb?
>
> CPython could no longer run on a computer where at startup it could not
> allocate at least one 4mb continguous block of memory.
> CPython programs would die slightly sooner in out-of-memory conditions.
> CPython programs would use more memory.  How much?  Hard to say.  It depends
> on their allocation strategy.  I suspect most of the time it would be < 3mb
> additional memory.  But for pathological allocation strategies the
> difference could be significant.  (e.g: lots of allocs, followed by lots of
> frees, but the occasional object lives forever, which means that the arena
> it's in can never be freed.  If 1 out of ever 16 256k arenas is kept alive
> this way, and the objects are spaced out precisely such that now it's 1 for
> every 4mb arena, max memory use would be the same but later stable memory
> use would hypothetically be 16x current)
> Many programs would be slightly faster now and then, simply because we call
> malloc() 1/16 as often.
>
>
> What say you?  Vote for your favorite color of bikeshed now!
>
>
> /arry
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>


More information about the Python-Dev mailing list