[Python-checkins] peps: PEP 445: more background

[victor.stinner]

http://hg.python.org/peps/rev/5e26a6276514
changeset:   4932:5e26a6276514
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Sun Jun 16 03:49:29 2013 +0200
summary:
  PEP 445: more background

files:
  pep-0445.txt |  59 +++++++++++++++++++++++++++++++++++++++-
  1 files changed, 58 insertions(+), 1 deletions(-)


diff --git a/pep-0445.txt b/pep-0445.txt
--- a/pep-0445.txt
+++ b/pep-0445.txt
@@ -151,13 +151,65 @@
 XXX To be done (Kristján Valur Jónsson) XXX
 
 
+External libraries
+==================
+
+* glib: `g_mem_set_vtable()
+  <http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable>`_
+
+
+Memory allocators
+=================
+
+The C standard library provides the well known ``malloc()`` function. Its
+implementation depends on the platform and of the C library. The GNU C library
+uses a modified ptmalloc2, based on "Doug Lea's Malloc" (dlmalloc). FreeBSD
+uses `jemalloc <http://www.canonware.com/jemalloc/>`_. Google provides
+tcmalloc which is part of `gperftools <http://code.google.com/p/gperftools/>`_.
+
+``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
+mappings are usually used for large allocations (ex: larger than 256 KB),
+whereas the heap is used for small allocations.
+
+The heap is handled by ``brk()`` and ``sbrk()`` system calls on Linux, and is
+contiguous.  Memory mappings are handled by ``mmap()`` on UNIX and
+``VirtualAlloc()`` on Windows, they are discontiguous. Releasing a memory
+mapping gives back the memory immediatly to the system. For the heap, memory is
+only gave back to the system if it is at the end of the heap. Otherwise, the
+memory will only gave back to the system when all the memory located after the
+released memory are also released. This limitation causes an issue called the
+"memory fragmentation": the memory usage seen by the system may be much higher
+than real usage.
+
+Windows provides a `Low-fragmentation Heap
+<http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%29.aspx>`_.
+
+The Linux kernel uses `slab allocation
+<http://en.wikipedia.org/wiki/Slab_allocation>`_.
+
+The glib library has a `Memory Slice API
+<https://developer.gnome.org/glib/unstable/glib-Memory-Slices.html>`_:
+efficient way to allocate groups of equal-sized chunks of memory
+
+
 Links
 =====
 
-Memory allocators:
+CPython issues related to memory allocation:
 
 * `Issue #3329: Add new APIs to customize memory allocators
   <http://bugs.python.org/issue3329>`_
+* `Issue #13483: Use VirtualAlloc to allocate memory arenas
+  <http://bugs.python.org/issue13483>`_
+* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
+  isn't thread safe <http://bugs.python.org/issue16742>`_
+* `Issue #18203: Replace calls to malloc() with PyMem_Malloc()
+  <http://bugs.python.org/issue18203>`_
+* `Issue #18227: Use Python memory allocators in external libraries like zlib
+  or OpenSSL <http://bugs.python.org/issue18227>`_
+
+Projects analyzing the memory usage of Python applications:
+
 * `pytracemalloc
   <https://pypi.python.org/pypi/pytracemalloc>`_
 * `Meliae: Python Memory Usage Analyzer
@@ -167,6 +219,11 @@
 * `PySizer (developed for Python 2.4)
   <http://pysizer.8325.org/>`_
 
+APIs to set a custom memory allocator and/or hook memory allocators:
+
+* `GNU libc: Memory Allocation Hooks
+  <http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_
+
 Other:
 
 * `Python benchmark suite

-- 
Repository URL: http://hg.python.org/peps