[Python-Dev] Activating pymalloc

[Tim Peters]

[Tim]
>> Are "the right" alloc/dealloc macros documented anywhere?  I can't
>> find it, if they are.

[Martin]
> http://www.python.org/doc/current/api/memoryInterface.html
>
> is as precise and comprehensive as anything I would have written
> myself.

Then no wonder extension authors can't get this right <0.9 wink>.

> The Examples even give explicit examples what cases to avoid.

Heh -- I never noticed the examples before.  It's curious that they claim

    PyMem_Del(buf3);  /* Wrong -- should be PyMem_Free() */

when the docs just finished saying

    void PyMem_Del(void *p)
        Same as PyMem_Free().

Either the example or the doc's "same as" is incomprehensible.

It's also curious that the examples say

    free(buf1);       /* Fatal -- should be PyMem_Del()  */

when the docs said no such thing (and this may be the #1 error in practice).
The Examples page (but not the docs) says

    Indeed, it is required to use the same memory API family for a given
    memory block,

but the crucial phrase "the same memory API family" is left undefined.  At a
minimum, it needs to identify "the memory API families" by explicit,
exhaustive enumeration.  The mess in practice is a fact, and proves people
don't understand what the docs are *trying* to say here.  I'm not sure
myself.

Like, are the functions and macros interchangeable?  For example, if you
know something was allocated via PyMem_Malloc, is it OK to release it via
PyMem_FREE?  I simply can't guess from what the docs say, again because what
constitutes "a family" is left undefined.

> For PyObject, the equivalent documentation is at
>
> http://www.python.org/doc/current/api/allocating-objects.html

I'll be charitable and guess this defines a different family <wink>.

Note that users also have to know about

http://www.python.org/doc/current/api/supporting-cycle-detection.html

in order to get at

    PyObject_GC_New / PyObject_GC_NewVar / PyObject_GC_Del

So far we've covered more than two dozen spellings (even without plain
"malloc" and "free"), spread over at least 4 manual pages.  One compact
table listing all of them in "legal" pairs would be an enormous help.

> ...
> Do you think you would understand the documentation that is currently
> there?

Not well enough to use with confidence, no.  I've seen the docs before,
although I did miss the Examples section previously.  I have a fancy editor
with an integrated symbol database, and in practice I chase down the macros
and functions until I see what they *do* at the leaves.  The endless layers
of indirection macros make this quite a pain (and I see you resorted to
looking at post-preprocessor listings).

> BTW, I found the documentation by grepping for PyMem_ in all .tex
> files, because I could not be bothered to read the online version,
> either.

They're easy to find via the index in the C API manual.

> It leaves out a good many details (i.e. additional API), but I think
> this is deliberate - you are not supposed to ever use this other API,
> anyway.

Which API?

> I wonder whether one could design a script that analyses Python code
> for asymmetric usage of memory management functions, e.g. trying to
> match
>
>    expression->FIELD = ALLOCATOR;
>    DEALLOCATOR(other_expression->FIELD);
>
> This would complain if there is a FIELD that has no symmetric usage,
> or if a certain allocator has no counterpart at all in a source file.
>
> I then wonder if that script would have found the errors you cite.

Well, to a first approximation, just searching for "free(" is valuable!  In
the Python source, the thread implementations use malloc and free on
purpose, but it also turns up a curious

		    free((char *)Py_FileSystemDefaultEncoding);

in _localmodule.c.  If not an outright bug, that's at least begging for a
comment, as the only obvious definition is in bltinmodule.c:

#if defined(MS_WIN32) && defined(HAVE_USABLE_WCHAR_T)
const char *Py_FileSystemDefaultEncoding = "mbcs";
#else
const char *Py_FileSystemDefaultEncoding = NULL; /* use default */
#endif

Does that mean that, on Windows, we may free() a static char*?!