[Python-Dev] Suggested memory API rules for 2.3

Tim Peters tim.one@comcast.net
Wed, 03 Apr 2002 17:59:46 -0500


>> "Programs" does not include the Python core.  We can do anything we
>> need to do in the core, based on our complete <heh> knowledge of
>> each release's internal implementation details.

[Guido]
> However, since many people use the core as an example, we may have to
> be careful here.  Certainly xxmodule.c, xxobject.c and xxsubtype.c
> should do the recommended thing;

I'll make sure that they do.

> but if too many of the built-in objects use the macros despite those
> being in danger of deprecation, that might still perpetuate use of
> those macros.

It's hard to imagine that the macro names will often be cheaper than the
function names, given the new requirement that they have identical I/O
behavior.  So, yes, I expect a lot of mindless source edits to replace core
use of macro names with function names (and I'm good at massive mindless
source edits <wink>).  There may be still be cases where the core wants to
exploit that, e.g., it *knows* a malloc argument is > 0, and so skip the
_PyMem_EXTRA business; if so, I expect we'll grow an internal API spelling
for that.

>>     After memory has been obtained via one of these functions, it
>>     should be resized and freed only by a function from the same line,
>>     except that PyMem_Free may be used freely in place of PyMem_Del,
>>     and PyObject_Free in place of PyObject_Del.

> I actually had a slightly different rule in mind:

I did too at first.

> mixing and matching between two adjacent lines that only differ by case
> is also permitted.  Except of course that the macro lines will be
> deprecated, so maybe it doesn't matter to spell this out.

Exactly so.  Since programs should not use macro names at all anymore, text
devoted to drawing fine distinctions among macro name use is pretty much a
waste.  Note there is no "shall" or "shall not" text *forbidding* mixing
corresponding macro and function names:  the text above is only "should be
... from the same line".  The later blurbs of the form

    Memory obtained by PyMem_{Malloc, MALLOC} shall be resized only
    by PyMem_{Realloc, REALLOC}.

are *not* intended not to be read as forbidding mixing PyMem_Malloc with
PyMem_REALLOC.  Perhaps that would be clearer said

    Memory obtained by either of PyMem_{Malloc, MALLOC} shall be
    resized only by either of PyMem_{Realloc, REALLOC}.

The exhaustive tables I posted earlier are much clearer for stuff like this,
but, again, I'm decreasingly keen to spell out elaborate details for
spellings that are discouraged.

[on whether private-API spellings for raw malloc etc should be introduced]
> IMO, PyMem_Malloc() and PyMem_Free() should be good enough.

I was expecting this would be an objective matter resolved by exhaustive
study of the code base when renaming the macro uses.  I'm happy to settle it
arbitrarily, though <wink>.

>>     Relationship to Platform Allocator
>>
>>     All names in lines A, B, C and D ultimately invoke the platform
>>     C malloc, realloc, or free.  However, programs shall not mix any
>>     of these names with direct calls to the platform malloc, calloc,
>>     realloc, or free referencing the same base memory addresses, as
>>     Python may need to perform bookkeeping of its own around its calls
>>     to the platform allocator.

> Really?  Why not just say these are wrappers around malloc and free?

I'm specifically trying to leave the door open for the PyMalloc_DebugXYZ
routines to capture all uses of memory API functions in PYMALLOC_DEBUG mode.
It also reads better as-is than to say, well, ya, OK, I suppose you *can*
mix raw calls to libc with these particular memory API families, but the
PyObject_GC_xyz family is entirely different in this respect.  I don't want
to draw distinctions just because the current implementation makes it
possible to draw them; users shall <wink> consider malloc/realloc/free as a
distinct API.

> (On platforms where it matters, they will be guaranteed to use the
> same heap as the core uses -- this is apparently an issue with Windows
> DLLs.)

But that's another reason (if it's real <wink>) to say "no mixing" here:  if
this is a use they care about, then, e.g., they're dead meat if they mix
PyMem_Malloc() with raw free() (the former would use the malloc in the
Python DLL, the latter the free in the user's DLL).