[Python-Dev] RFC: malloc cleanup

Vladimir Marangozov Vladimir.Marangozov@inrialpes.fr
Thu, 17 Feb 2000 22:03:06 +0100 (CET)


Since nobody has had a spontaneous comment on the malloc problem
I was referring to in my previous message, here is a concrete proposal.

BTW, I was thinking on how to deal with *all* malloc problems and solve
them definitively starting from <as soon as possible -- 1.6/7> by staying
backwards compatible on top of that!

Please comment on it, so that I could proceed on working on a patch
(which involves lots of files in the distribution, so it'll take me
some time verifying the files once again, and document the changes).

---------------

I'll start with the public malloc interfaces, exported by libpython.a
(and Python.h) for third party development.

As a second step, I'll attack the "core".

The "core" for me are the files in Parser/*, Python/* and Objects/*.
I'm considering the files in Modules/* as extensions to the core.
(although as long as they are in the standard distribution, one may want
to see them as part of the core, but this is a secondary issue).

--------------------------------------------------------------------------

Public Malloc (Object) Interface
================================

Functions:

1) PyMem_Malloc / Realloc / Free -- existing wrappers around the Python core
                                    malloc - don't call anything on failure.

2) Py_Malloc / Realloc / Free    -- existing wrappers around the Python core
                                    malloc, calling PyErr_NoMemory on failure

3) _PyObject_New                 -- object creation, uses Python's core malloc

Macros:

4) PyMem_NEW / PyMem_RESIZE      -- the existing type-oriented macros
                                    *but* implemented in terms of 1).
                                     (PyMem_Malloc & PyMem_Realloc)
                                    
5) PyObject_NEW                  -- existing, implemented in terms of 3)

6) PyMem_DEL/XDEL                -- existing, *but* pointing to PyMem_Free.
                                    PyMem_XDEL is deprecated, but is left
                                    for backwards compatibility.

Note: everything is implemented on top of 1), the PyMem_Malloc wrappers.

      This will ensure that compiled C extensions can be used "as is" by
      new versions of the core (even if the latter has another malloc)
      ==> no need to recompile them.

Questions:

   a) What about PyMem_DEL? Should it really be deprecated as a "handy alias"
      for PyMem_Free? Strictly speaking, I'd probably keep it so that
      PyMem_NEW/DEL form a pair.

   b) What about introducing PyObject_DEL as the buddy of PyObject_NEW?
      (also an alias for PyMem_Free). This is really an interface issue,
      so please adopt an "end-user" point of view.

      Personally, I'd feel more comfortable writing code like this one,
      voluntarily made to use the different interfaces (checks omitted).

      Object initialization:

      o = PyObject_NEW(spam_struct, &spam_type); /* new spam object */
      o->foo = 1;
      o->bar = PyMem_Malloc(10);     /* requesting Python memory */
      o->my_buff = malloc(20);       /* for personal needs  */

      Finalization:

      free(o->my_buff);              /* release private mem */
      PyMem_Free(o->bar);            /* release Python mem */
      PyObject_DEL(o);               /* release the object */


      That is, as a programmer, I don't really want to know what's behind
      the macros, as long as they come in pairs. I want things to be easier
      for me as much as possible.

      Note that as long as I allocate/release my objects with
      PyObject_NEW/DEL, I can use _any_ malloc I want for private purposes.

---------------------------------------------------------------------------

Given the above, the malloc interfaces used by the core (the private ones)
should be easier to understand.

Basically, we'll have the the same macros (with the same questions a) & b)
but everything here is implemented on top of PyMem_MALLOC and friends:

Private/Core Malloc (Object) Interface
=======================================

1) PyMem_MALLOC / REALLOC / FREE      - raw memory interface, core malloc.

2) _PyObject_New                      - uses 1)

3) PyMem_NEW / RESIZE / DEL(?)        - use 1)  PyMem_XDEL is out here.

4) PyObject_NEW / DEL (?)             - use 2) & 1)

--------------------------------------------------------------------------

The thing is: the public interfaces are always redirected through
the wrappers, while the private ones are used (are visible) only
by the core, and are based on a raw malloc.

This is, IMO, the right approach for avoiding all troubles (past & future)

Implementation issue
====================

Solution 1)

If the exported and the core malloc interfaces preserve the same
(macro) names, we need to introduce some preprocessor magic such that
the same macros have different definitions depending on the type of
the interface they belong to (public or core).

Solution 2)

The alternative is to avoid the name clashes by renaming the macros of
the _core_ interface (the exported one cannot be changed for backwards
compatibility reasons). That is, rename 3) and 4).
(Note: 1) is new, 2) is a function)

Question: which one is better?

(assuming that both of them require the same amount of work ;-)

------------------------------------------------------------------------

That's it. Don't hesitate to ask me for clarifications.

All undecided (not approved) parts of the proposal have been marked
as such --> Questions.

If we manage to agree on the differents questions, I believe that
the implementation of this proposal would solve the issue permanently.

Thanks for your patience!  I'm sick either <0.5 wink>

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252