[Python-Dev] pymalloc on 2.1.3

[Tim Peters]

[Fredrik Lundh]
> does anyone remember if there were any big changes in pymalloc between
> the 2.1 series (where it was introduced) and 2.3 (where it was enabled by
> default).

Yes, huge -- few original lines survived exactly, although many
survived in intent.

> or in other words, is the 2.1.3 pymalloc stable enough for production use?

Different question entirely <wink>.  It _was_ used in production by
some people, and happily so.  Major differences:

+ 2.1 used a probabilistic scheme for guessing whether addresses passed
  to it were obtained from pymalloc or from the system malloc.  It was
  easy for a malicous pure-Python program to corrupt pymalloc and/or malloc
  internals as a result, leading to things like segfaults, and even sneaky ways
  to mutate the Python bytecode stream.  It's extremely unlikely that a non-
  malicious program could bump into these.

+ Horrid hackery went into 2.3's version to cater to broken extension modules
  that called PyMem functions without holding the GIL.  2.1's may not be
  as thread-safe in these cases.

+ 2.1's only fields requests up to 64 bytes, 2.3's up to 256 bytes.  Changes in
  the dict implementation, and new-style classes, for 2.3 made it a pragmatic
  necessity to boost the limit for 2.3.

> (we're having serious memory fragmentation problems on a 2.1.3 system,
> and while I can patch/rebuild the interpreter if necessary, we cannot update
> the system right now...)

I'd give it a shot -- pymalloc has always been very effective at
handling large numbers of small objects gracefully.  The meaning of
"small" got 4x bigger since 2.1, which appeared to be a pure win, but
64 bytes was enough under 2.1 that most small instance dicts fit.