[Python-Dev] Moving forward on the object memory API

[Tim Peters]

[Neil Schemenauer]
> It looks like Tim managed to work some magic and make pymalloc's free
> handle memory allocated by either the system allocator or pymalloc
> itself.

Yes, I believe that's so now.  We'll know better after we try it <wink -- I
did some light contrived testing of mixed uses>.  One more:  provided that
an address was obtained via the system allocator (malloc or realloc), it
should be safe to free it via the pymalloc free even when the GIL is not
held.

> That means we can make PyMem_DEL use the pymalloc's 'free' and
> make PyObject_NEW use pymalloc's 'malloc'.

It means we have a *lot* more freedom in mixing and matching now.
Unfortunately, this comes down to spelling out what "the rules" actually
are, and there's been no progress on that.  The only feedback I've gotten on
the two exhaustive tables I posted earlier was along the lines of "geez,
this really is a mess -- I'm not sure what to do".

> Here's my plan for making this happen:
>
>     #define PyMem_MALLOC(n)         malloc(n)
>     #define PyMem_REALLOC(p, n)     realloc((void *)(p), (n))
>     #define PyMem_FREE(p)           free((void *)(p))

Is there a reason not to make these the simpler

#define PyMem_MALLOC malloc

etc?

> I think making PyMem_FREE call free() is safe.  I haven't seen any
> extension modules allocate with PyObject_NEW and free with PyMem_FREE.

What about PyMem_Free()?  You reported that MySQL-python-0.3.5 and crng-1.1
mix PyObject_NEW with PyMem_Free.

Note that Guido recently reported:

    The memory thus allocated is returned by PyOS_Readline() (which calls
    either function through a hook pointer), and both its callers (the
    tokenizer and raw_input()) free the result using PyMem_DEL or
    PyMem_FREE (these two seem to be used synonymically).

If that's so, PyMem_DEL and PyMem_FREE should be the same (but that grates
on me too, FWIW).

> We deprecate the PyMem_* functions.

This may hit two snags:  the docs *currently* say (and have for some time)
that all of

   PyMem_{MALLOC, REALLOC, FREE, NEW, RESIZE, DEL}

are "deprecated in extension modules".  The docs encourage using the PyMem
function spellings instead (or perhaps the docs insist on that -- it depends
on what the docs mean by "deprecated"; IMO deprecation is without
consequence in any context where we can't deliver warning msgs for at least
a release cycle -- else we'll never actually get rid of "deprecated" stuff).
The rationale has to do with cross-release binary compatibility:  if an
extension spells a thing PyMem_Malloc, they have "a right" to expect that
their extension won't need to be recompiled across releases just because
Python changes what PyMem_MALLOC does.

> There's no need for them, IMHO:
>
>     #define PyMem_Malloc PyMem_MALLOC
>     #define PyMem_New PyMem_NEW
>     #define PyMem_Resize PyMem_RESIZE
>     #define PyMem_Free PyMem_DEL
>     #define PyMem_Del PyMem_DEL
>     /* See comment near MALLOC_ZERO_RETURNS_NULL in pyport.h. */
>     #define PyMem_Realloc(p, n) \
>             do { \
>                     size_t _n = n; \
>                     PyMem_REALLOC(p, _n ? _n : 1); \
>             } while (0)

Note that PyMem_Realloc needs to return a result (else, e.g., the user can't
know whether it failed).  In any case, I predict Guido will say that the
function spellings must indeed be functions, and are still the recommended
way, and that the macro spellings are still deprecated for use in extensions
(but not in the core), and that "deprecated" in this context will continue
to mean nothing stronger than "should, but there's no actual consequence if
you don't, apart from opening yourself up to possible x-release binary
incompatibility".

> Next, we base PyObject_{MALLOC,REALLOC,FREE} on pymalloc.

No problem there:  if we don't, I wasted my last few days.

>  Basically:
>
>     #ifdef WITH_PYMALLOC
>     #define PyObject_MALLOC(n)      _PyMalloc_Malloc(n)
>     #define PyObject_REALLOC(op, n) _PyMalloc_Realloc((void *)(op), (n))
>     #define PyObject_FREE(op)       _PyMalloc_Free((void *)(op))
>     #else
>     #define PyObject_MALLOC(n)      PyMem_MALLOC(n)
>     #define PyObject_REALLOC(op, n) PyMem_REALLOC((void *)(op), (n))
>     #define PyObject_FREE(op)       PyMem_FREE((void *)(op))
>     #endif

Again I wonder why these aren't simply name substitutions ("#define
PyObject_MALLOC _PyMalloc_Malloc", etc).

> PyMem_DEL and PyMem_Free need to call pymalloc:
>
>     #define PyMem_DEL(op) PyObject_FREE(op)

At least those two, yes (see above for questions about whether PyMem_FREE
also needs to play along; but I'd sure like *some* way for an extension
author to say "give me the same damn platform free() Python uses, without
any damn wrappers").

> We go through the source and replace all the PyMem_* function calls with
> the equivalent PyMem_* macro calls and replace PyMem_DEL with
> PyMem_FREE.

As above, I expect Guido will resist at least part of this part.

> The recommended API for extension modules writers would be
> PyMem_{MALLOC,REALLOC,NEW,RESIZE,FREE},

Ditto, but more so, since it's a 180 degree reversal of what the docs have
(at least) recommended.

> PyObject_{New,NewVar,Del}, and PyObject_GC_{New,NewVar,Del}.

This part should be non-controversial so far as it goes.  But the docs don't
currently give any cautions about using the macro versions of PyObject_XXX,
and in addition to those we're also missing PyObject_{Malloc, MALLOC,
Realloc, REALLOC, Free, FREE} and PyObject_GC_Resize in this account.
Straightening all that out again requires agreeing on what the rules really
are.  I think ultimately has to come from Guido, but the best way to get
that to happen is to provoke him with schemes he doesn't like <wink>.