[Python-Dev] Activating pymalloc

[Tim Peters]

Suppose we were to take the idea of "enforcing the rules" seriously.  That
means we'd first have to know what the rules are.  I count 40 primary (just
"get", "resize", and "free") memory APIs now:

malloc, realloc, free
PyMem_{Malloc, Realloc, Free}
PyMem_{MALLOC, REALLOC, FREE}
PyMem_{New, Resize, Del}
PyMem_{NEW, RESIZE, DEL}
PyObject_{Malloc, Realloc, Free}
PyObject_{MALLOC, REALLOC, FREE}
PyObject_{New, NewVar, Del}
PyObject_{NEW, NEWVAR, DEL}
PyObject_GC_{New, NewVar, Resize, Del}
PyMalloc_{New, NewVar, Del}
_PyMalloc_{Malloc, Realloc, Free}
_PyMalloc_{MALLOC, REALLOC, FREE}

The last 6 were recently added, and made part of the private API to avoid
complicating the public API even more.

I don't understand what "API family" means, and none of the prose I've seen
made the notion clearer.  The only way it *can* mean something intelligible
to me is via a table like the following, spelling out exactly which "get
memory" spellings are legal with which "release memory" spellings (counting
realloc/resize as a form of release).  Catering to Martin's belief that
functions and macros are necessarily in distinct families, yet catering also
to the real-life examples Neil found, this is the most restrictive set of
families I think we have a chance of enforcing without massive breakage:

Memory released by      Must have been originally allocated by
one of these            one of these
----------------------- --------------------------------------
realloc,                malloc
free
----------------------- --------------------------------------
PyObject_GC_Del         PyObject_GC_New,
                        PyObject_GC_NewVar
----------------------- --------------------------------------
PyObject_GC_Resize      PyObject_GC_NewVar
----------------------- --------------------------------------
PyMalloc_Del            PyMalloc_New,
                        PyMalloc_NewVar
----------------------- --------------------------------------
_PyMalloc_Realloc,      _PyMalloc_Malloc
_PyMalloc_Free
----------------------- --------------------------------------
_PyMalloc_REALLOC,      _PyMalloc_MALLOC
_PyMalloc_FREE
----------------------- --------------------------------------
PyObject_Del            PyObject_New,
                        PyObject_NewVar
----------------------- --------------------------------------
PyObject_DEL            PyObject_NEW,
                        PyObject_NEWVAR
----------------------- --------------------------------------
PyObject_Realloc,       PyObject_Malloc
PyObject_Free
----------------------- --------------------------------------
PyObject_REALLOC,       PyObject_MALLOC
PyObject_FREE
----------------------- --------------------------------------
PyMem_Del,              PyMem_New
PyMem_Resize
----------------------- --------------------------------------
PyMem_RESIZE            PyMem_NEW
----------------------- --------------------------------------
PyMem_Realloc           PyMem_Malloc
----------------------- --------------------------------------
PyMem_FREE,             PyMem_MALLOC
PyMem_REALLOC
----------------------- --------------------------------------
PyMem_DEL,              PyMem_NEW,
PyMem_Free              PyObject_NEW
----------------------- --------------------------------------

The last block caters to the frequenct mixes of PyObject_NEW with PyMem_DEL
and PyMem_Free Neil found in real life.

I always found the distinction between "free" and "del" spellings to be
pointlessly pedantic (there's no difference under the covers, and never
was -- and backward compatibility ensures there never will be).

The distincion between Mem and Object has also become pointlessly pedantic.

The distinction between macro and function spellings also doesn't make much
sense to me, despite Martin's defense (you shouldn't use a macro spelling in
an extension simply because you're in an extension, and macros can change
across releases; it's not because we're reserving the right for the function
and macro spellings to do incompatible things in a single release).

The distinctions between raw and typed memory, and between var and non-var
objects, still make good sense.

Putting all those together could cut the # of groupings in half:

Memory released by      Must have been originally allocated by
one of these            one of these
----------------------- --------------------------------------
realloc,                malloc
free
----------------------- --------------------------------------
PyObject_GC_Del         PyObject_GC_New,
                        PyObject_GC_NewVar
----------------------- --------------------------------------
PyObject_GC_Resize      PyObject_GC_NewVar
----------------------- --------------------------------------
PyMalloc_Del            PyMalloc_New,
                        PyMalloc_NewVar
----------------------- --------------------------------------
PyMem_Del,              PyMem_New,
PyMem_DEL,              PyMem_NEW,
PyMem_Free,             PyMem_Malloc,
PyMem_FREE,             PyMem_MALLOC,
PyObject_Del,           PyObject_New,
PyObject_DEL,           PyObject_NEW,
PyObject_Free           PyObject_Malloc,
PyObject_FREE           PyObject_MALLOC
                        PyObject_NewVar,
                        PyObject_NEWVAR
----------------------- --------------------------------------
PyMem_Realloc,          PyMem_Malloc,
PyMem_REALLOC,          PyMem_MALLOC,
PyObject_Realloc,       PyObject_Malloc,
PyObject_REALLOC        PyObject_MALLOC
----------------------- --------------------------------------
PyMem_Resize,           PyMem_New,
PyMem_RESIZE            PyMem_NEW
----------------------- --------------------------------------
_PyMalloc_Realloc,      _PyMalloc_Malloc
_PyMalloc_Free,         _PyMalloc_MALLOC
_PyMalloc_REALLOC,
_PyMalloc_FREE
----------------------- --------------------------------------

I don't yet know whether we can actually enforce anything here, but we have
to explain the rules regardless.  What kind of table would you make up?  I
expect the last table above matches what most extension authors think,
although some probably lump free/malloc into the block with the other 8
historical ways to spell free.