[Python-Dev] The untuned tunable parameter ARENA_SIZE

[Victor Stinner]

2017-06-01 10:41 GMT+02:00 Larry Hastings <larry at hastings.org>:
> On 06/01/2017 01:19 AM, Antoine Pitrou wrote:
> If you'd like to go that way anyway, I would suggest 1MB as a starting
> point in 3.7.
>
> I understand the desire for caution.  But I was hoping maybe we could
> experiment with 4mb in trunk for a while?  We could change it to 1mb--or
> even 256k--before beta 1 if we get anxious.

While I fail to explain why in depth, I would prefer to *not* touch
the default arena size at this point.

We need more data, for example measure the memory usage on different
workloads using different arena sizes.

It's really hard to tune a memory allocator for *any* use cases.

A simple enhancement would be to add an environment variable to change
the arena size at Python startup. Example: PYTHONARENASIZE=1M. If you
*know* that your application will allocate at least 2 GB, you may even
want to try PYTHONARENASIZE=1G which is more likely to use a single
large page... Such parameter cannot be used by default: it would make
the default Python memory usage insane ;-)

Victor