[Python-Dev] pymalloc on 2.1.3
tim.peters at gmail.com
Tue Feb 15 16:19:01 CET 2005
> does anyone remember if there were any big changes in pymalloc between
> the 2.1 series (where it was introduced) and 2.3 (where it was enabled by
Yes, huge -- few original lines survived exactly, although many
survived in intent.
> or in other words, is the 2.1.3 pymalloc stable enough for production use?
Different question entirely <wink>. It _was_ used in production by
some people, and happily so. Major differences:
+ 2.1 used a probabilistic scheme for guessing whether addresses passed
to it were obtained from pymalloc or from the system malloc. It was
easy for a malicous pure-Python program to corrupt pymalloc and/or malloc
internals as a result, leading to things like segfaults, and even sneaky ways
to mutate the Python bytecode stream. It's extremely unlikely that a non-
malicious program could bump into these.
+ Horrid hackery went into 2.3's version to cater to broken extension modules
that called PyMem functions without holding the GIL. 2.1's may not be
as thread-safe in these cases.
+ 2.1's only fields requests up to 64 bytes, 2.3's up to 256 bytes. Changes in
the dict implementation, and new-style classes, for 2.3 made it a pragmatic
necessity to boost the limit for 2.3.
> (we're having serious memory fragmentation problems on a 2.1.3 system,
> and while I can patch/rebuild the interpreter if necessary, we cannot update
> the system right now...)
I'd give it a shot -- pymalloc has always been very effective at
handling large numbers of small objects gracefully. The meaning of
"small" got 4x bigger since 2.1, which appeared to be a pure win, but
64 bytes was enough under 2.1 that most small instance dicts fit.
More information about the Python-Dev