[Python-checkins] CVS: python/dist/src/Doc/api api.tex,1.56.2.12,1.56.2.13

Fred L. Drake fdrake@weyr.cnri.reston.va.us
Tue, 29 Feb 2000 10:53:06 -0500 (EST)


Update of /projects/cvsroot/python/dist/src/Doc/api
In directory weyr:/home/fdrake/projects/python/Doc-152p2/api

Modified Files:
      Tag: release152p1-patches
	api.tex 
Log Message:

Documentation for the memory management APIs, from Vladimir Marangozov
<marangoz@python.inrialpes.fr>.


Index: api.tex
===================================================================
RCS file: /projects/cvsroot/python/dist/src/Doc/api/api.tex,v
retrieving revision 1.56.2.12
retrieving revision 1.56.2.13
diff -C2 -r1.56.2.12 -r1.56.2.13
*** api.tex	2000/02/21 18:10:09	1.56.2.12
--- api.tex	2000/02/29 15:53:03	1.56.2.13
***************
*** 709,717 ****
  well as the global variable \cdata{_Py_RefTotal}.
  
- XXX Should mention Py_Malloc(), Py_Realloc(), Py_Free(),
- PyMem_Malloc(), PyMem_Realloc(), PyMem_Free(), PyMem_NEW(),
- PyMem_RESIZE(), PyMem_DEL(), PyMem_XDEL().
  
- 
  \chapter{Exception Handling \label{exceptionHandling}}
  
--- 709,713 ----
***************
*** 3248,3251 ****
--- 3244,3462 ----
  must be held.
  \end{cfuncdesc}
+ 
+ 
+ \chapter{Memory Management \label{memory}}
+ 
+ \section{Overview \label{memoryOverview}}
+ 
+ Memory management in Python involves a private heap containing all
+ Python objects and data structures. The management of this private
+ heap is ensured internally by the \emph{Python memory manager}.  The
+ Python memory manager has different components which deal with various
+ dynamic storage management aspects, like sharing, segmentation,
+ preallocation or caching.
+ 
+ At the lowest level, a raw memory allocator ensures that there is
+ enough room in the private heap for storing all Python-related data,
+ by interacting with the memory manager of the operating system. On top
+ of the raw memory allocator, several object-specific allocators
+ operate on the same heap and implement distinct memory management
+ policies adapted to the peculiarities of every object type. For
+ example, integer objects are managed differently within the heap than
+ strings, tuples or dictionaries because integers imply different
+ storage requirements and speed/space tradeoffs. The Python memory
+ manager thus delegates some of the work to the object-specific
+ allocators, but ensures that the latter operate within the bounds of
+ the private heap.
+ 
+ It is important to understand that the management of the Python heap
+ is performed by the interpreter itself and that the user has no
+ control on it, even if she manipulates regularly object pointers to
+ memory blocks inside that heap. The allocation of heap space for
+ Python objects and other internal buffers is performed on demand by
+ the Python memory manager through the Python/C API functions listed in
+ this document.
+ 
+ To avoid memory corruption, extension writers should never try to
+ operate on Python objects with the functions exported by the C
+ library: \cfunction{malloc()}, \cfunction{calloc()},
+ \cfunction{realloc()} and \cfunction{free()}. This will result in
+ mixed calls between the C allocator and the Python memory manager
+ with fatal consequences, because they implement different algorithms
+ and operate on different heaps. However, one may safely allocate and
+ release memory blocks with the C library allocator for individual
+ purposes, as shown in the following example:
+ 
+ \begin{verbatim}
+ {
+     PyObject *res;
+     char *buf = (char *) malloc(BUFSIZ); /* for I/O */
+     if (buf == NULL)
+         return PyErr_NoMemory();
+     ...Do some I/O operation involving buf...
+     res = PyString_FromString(buf);
+     free(buf); /* malloc'ed */
+     return res;
+ }
+ \end{verbatim}
+ 
+ In this example, the memory request for the I/O buffer is handled by
+ the C library allocator. The Python memory manager is involved only
+ in the allocation of the string object returned as a result.
+ 
+ In most situations, however, it is recommended to allocate memory from
+ the Python heap specifically because the latter is under control of
+ the Python memory manager. For example, this is required when the
+ interpreter is extended with new object types written in C. Another
+ reason for using the Python heap is the desire to \emph{inform} the
+ Python memory manager about the memory needs of the extension module.
+ Even when the requested memory is used exclusively for internal,
+ highly-specific purposes, delegating all memory requests to the Python
+ memory manager causes the interpreter to have a more accurate image of
+ its memory footprint as a whole. Consequently, under certain
+ circumstances, the Python memory manager may or may not trigger
+ appropriate actions, like garbage collection, memory compaction or
+ other preventive procedures. Note that by using the C library
+ allocator as shown in the previous example, the allocated memory for
+ the I/O buffer escapes completely the Python memory manager.
+ 
+ \section{Memory Interface \label{memoryInterface}}
+ 
+ The following function sets, modeled after the ANSI-C standard, are
+ available for allocating and releasing memory from the Python heap:
+ 
+ \begin{cfuncdesc}{ANY*}{PyMem_Malloc}{size_t n}
+ Allocates \var{n} bytes and returns a pointer of type \ctype{ANY*} to
+ the allocated memory, or \NULL{} if the request fails. Requesting zero
+ bytes returns a non-\NULL{} pointer.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{ANY*}{PyMem_Realloc}{ANY *p, size_t n}
+ Resizes the memory block pointed to by \var{p} to \var{n} bytes. The
+ contents will be unchanged to the minimum of the old and the new
+ sizes. If \var{p} is \NULL{}, the call is equivalent to
+ \cfunction{PyMem_Malloc(\var{n})}; if \var{n} is equal to zero, the memory block
+ is resized but is not freed, and the returned pointer is non-\NULL{}.
+ Unless \var{p} is \NULL{}, it must have been returned by a previous
+ call to \cfunction{PyMem_Malloc()} or \cfunction{PyMem_Realloc()}.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{void}{PyMem_Free}{ANY *p}
+ Frees the memory block pointed to by \var{p}, which must have been
+ returned by a previous call to \cfunction{PyMem_Malloc()} or
+ \cfunction{PyMem_Realloc()}.  Otherwise, or if
+ \cfunction{PyMem_Free(p)} has been called before, undefined behaviour
+ occurs. If \var{p} is \NULL{}, no operation is performed.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{ANY*}{Py_Malloc}{size_t n}
+ Same as \cfunction{PyMem_Malloc()}, but calls
+ \cfunction{PyErr_NoMemory()} on failure.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{ANY*}{Py_Realloc}{ANY *p, size_t n}
+ Same as \cfunction{PyMem_Realloc()}, but calls
+ \cfunction{PyErr_NoMemory()} on failure.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{void}{Py_Free}{ANY *p}
+ Same as \cfunction{PyMem_Free()}.
+ \end{cfuncdesc}
+ 
+ The following type-oriented macros are provided for convenience:
+ 
+ \begin{cfuncdesc}{TYPE*}{PyMem_NEW}{TYPE, size_t n}
+ Same as \cfunction{PyMem_Malloc()}, but allocates \var{(n *
+ sizeof(TYPE))} bytes of memory. Returns a pointer casted to
+ \ctype{TYPE*}.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{TYPE*}{PyMem_RESIZE}{ANY *p, TYPE, size_t n}
+ Same as \cfunction{PyMem_Realloc()}, but the memory block is resized
+ to \var{(n * sizeof(TYPE))} bytes. Returns a pointer casted to
+ \ctype{TYPE*}.
+ \end{cfuncdesc}
+ 
+ \begin{cfuncdesc}{void}{PyMem_DEL}{ANY *p}
+ Same as \cfunction{PyMem_Free()}.
+ \end{cfuncdesc}
+ 
+ 
+ \section{Examples \label{memoryExamples}}
+ 
+ Here is the example from section 1, rewritten so that the I/O buffer is
+ allocated from the Python heap by using the first function set:
+ 
+ \begin{verbatim}
+ {
+     PyObject *res;
+     char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */
+     if (buf == NULL)
+         return PyErr_NoMemory();
+     ...Do some I/O operation involving buf...
+     res = PyString_FromString(buf);
+     PyMem_Free(buf); /* allocated with PyMem_Malloc */
+     return res;
+ }
+ \end{verbatim}
+ 
+ With the second function set, the need to call
+ \cfunction{PyErr_NoMemory()} is obviated:
+ 
+ \begin{verbatim}
+ {
+     PyObject *res;
+     char *buf = (char *) Py_Malloc(BUFSIZ); /* for I/O */
+     if (buf == NULL)
+         return NULL;
+     ...Do some I/O operation involving buf...
+     res = PyString_FromString(buf);
+     Py_Free(buf); /* allocated with Py_Malloc */
+     return res;
+ }
+ \end{verbatim}
+ 
+ The same code using the macro set:
+ 
+ \begin{verbatim}
+ {
+     PyObject *res;
+     char *buf = PyMem_NEW(char, BUFSIZ); /* for I/O */
+     if (buf == NULL)
+         return PyErr_NoMemory();
+     ...Do some I/O operation involving buf...
+     res = PyString_FromString(buf);
+     PyMem_DEL(buf); /* allocated with PyMem_NEW */
+     return res;
+ }
+ \end{verbatim}
+ 
+ Note that in the three examples above, the buffer is always
+ manipulated via functions/macros belonging to the same set. Indeed, it
+ is highly recommended to use the same memory API family for a given
+ memory block, so that the risk of mixing different allocators is
+ reduced to a minimum. The following code sequence contains two errors,
+ one of which is labeled as \emph{fatal} because it mixes two different
+ allocators operating on different heaps.
+ 
+ \begin{verbatim}
+ char *buf1 = PyMem_NEW(char, BUFSIZ);
+ char *buf2 = (char *) malloc(BUFSIZ);
+ char *buf3 = (char *) PyMem_Malloc(BUFSIZ);
+ ...
+ PyMem_DEL(buf3);  /* Wrong -- should be PyMem_Free() */
+ free(buf2);       /* Right -- allocated via malloc() */
+ free(buf1);       /* Fatal -- should be PyMem_DEL()  */
+ \end{verbatim}
+ 
+ In addition to the functions aimed at handling raw memory blocks from
+ the Python heap, objects in Python are allocated and released with
+ \cfunction{_PyObject_New()}, \cfunction{_PyObject_NewVar()},
+ \cfunction{_PyObject_Del()}, or with their corresponding macros
+ \cfunction{PyObject_NEW()}, \cfunction{PyObject_NEW_VAR()},
+ \cfunction{PyObject_DEL()}.
+ 
+ % These will be explained in the next chapter on defining and
+ % implementing new object types in C.