[Python-Dev] pymalloc killer

[Tim Peters]

[martin@v.loewis.de]
> Are we discussing using PyMalloc even when not holding the GIL? That
> can't possibly work: pymalloc maintains linked lists of free memory;
> if two allocate or deallocate objects in the same pool simultaneously,
> bad things will happen.

I checked in obmalloc changes sufficient so that PyObject_XXX *could* be
changed to use pymalloc again, and not break if a PyObject "get memory"
function is mixed with a PyMem "free memory" function.  This requires (in
part) that all the latter be redirected to use the (new, as of a couple
hours ago) pymalloc free.

This leaves one known hole for backward compatibility:  some people call a
PyMem "get memory" routine when not holding the GIL.  That's still no
problem, because the PyMem "get memory" routines would still call the system
malloc without pymalloc getting involved.  Some people also call a PyMem
"free memory" routine when not holding the GIL, but only on memory that was
obtained from a PyMem "get memory" routine.  That's the potential problem.
The PyMem "free memory" routines would invoke pymalloc's free, and the mere
act of *checking* the address passed to PyMem free() is vulnerable if the
GIL isn't held.

It's not obvious how it's vulnerable, so I'll spell it out:  Some thread
that does hold the GIL and simultaneously calls a PyObject "get memory"
routine ends up in the pymalloc malloc, and *if* that needs to allocate a
new arena, and *if* that in turn requires growing the vector of arena base
addresses, then there appears to be no way (short of slopping locks on
everything) to guarantee that the *other* thread (in pymalloc free) won't
pick up an address for the arena-base-address vector that's invalid by the
time it gets around to indexing into it.  It's an extremely small hole (just
a couple machine instructions), and requires an extremely unlikely set of
circumstances to make it possible to fall into it, but it is a real hole all
the same.

There wouldn't be a hole if the vector of arena base addresses were
statically allocated.