[Python-Dev] The untuned tunable parameter ARENA_SIZE

Tim Peters tim.peters at gmail.com
Fri Jun 2 15:09:42 EDT 2017


[Tim]
>> While I would like to increase the pool size, it's fraught with
>> danger.

[Antoine Pitrou <solipsis at pitrou.net>]
> What would be the point of increasing the pool size?  Apart from being
> able to allocate 4KB objects out of it, I mean.
>
> Since 4KB+ objects are relatively uncommon (I mean we don't allocate
> hundreds of thousands of them per second), I don't think it's really
> worthwhile trying to have the small object allocator handle them.

I don't care about "large" objects here.  It's educated intuition
about speed:  so long as pymalloc is working within a pool, it's
blazing fast.  When it has to muck with a new pool, much slower
(compared to staying within a pool) code is required.  When it has to
muck with a new arena, slower still.

So the intuition is simple:  the larger a pool, the more object
operations it can handle staying within its best-case (fastest) code
paths.



>> It would be nice to find a different way for pymalloc to figure out
>> which addresses belong to it.  The excruciating Py_ADDRESS_IN_RANGE
>> manages to do it in small constant (independent of the number of
>> arenas and pools in use) time, which is its only virtue ;-)

> So, to sum it up, it's excruciating but fast and works reliably.  Why
> change it?

To enable using pools larger than the greatest common divisor of all
OS's native pool sizes.

There's also that Py_ADDRESS_IN_RANGE is responsible for countless
hours of head-scratching by poor developers trying to use magical
memory debuggers - it's at best unusual for code to read up memory
without caring a bit whether the memory has ever been stored to.

I should note that Py_ADDRESS_IN_RANGE is my code - this isn't a
backhanded swipe at someone else.  It's always been near the top of
the "code stink" scale.  So I thought I'd mention it in case someone
has been sitting on a cleaner idea but has held back because they
didn't want to offend me ;-)


More information about the Python-Dev mailing list