[Python-checkins] peps: PEP 445: cleanup
victor.stinner
python-checkins at python.org
Mon Jul 1 22:29:26 CEST 2013
http://hg.python.org/peps/rev/4de91c027ae2
changeset: 4969:4de91c027ae2
user: Victor Stinner <victor.stinner at gmail.com>
date: Mon Jul 01 22:29:08 2013 +0200
summary:
PEP 445: cleanup
Avoid "should", "may" and "might". Rephrase some sentences
files:
pep-0445.txt | 218 ++++++++++++++++++++------------------
1 files changed, 114 insertions(+), 104 deletions(-)
diff --git a/pep-0445.txt b/pep-0445.txt
--- a/pep-0445.txt
+++ b/pep-0445.txt
@@ -12,7 +12,8 @@
Abstract
========
-Add new APIs to customize Python memory allocators.
+Add new Application Programming Interfaces (API) to customize Python
+memory allocators.
Rationale
@@ -20,21 +21,22 @@
Use cases:
-* Application embedding Python may want to isolate Python memory from
- the memory of the application, or may want to use a different memory
+* Applications embedding Python which want to isolate Python memory from
+ the memory of the application, or want to use a different memory
allocator optimized for its Python usage
* Python running on embedded devices with low memory and slow CPU.
- A custom memory allocator may be required to use efficiently the
- memory and/or to be able to use all the memory of the device.
-* Debug tool to:
+ A custom memory allocator can be used for efficiency and/or to get
+ access all the memory of the device.
+* Debug tools for memory allocators:
- - track the memory usage (memory leaks)
- - get the Python filename and line number where an object was
- allocated
- - detect buffer underflow, buffer overflow and detect misuse of Python
- allocator APIs (builtin Python debug hooks)
- - force allocation to fail to test handling of ``MemoryError``
- exception
+ - track the memory usage (find memory leaks)
+ - get the location of a memory allocation: Python filename and line
+ number, and the size of a memory block
+ - detect buffer underflow, buffer overflow and misuse of Python
+ allocator APIs (see `Redesign Debug Checks on Memory Block
+ Allocators as Hooks`_)
+ - force memory allocations to fail to test handling of the
+ ``MemoryError`` exception
Proposal
@@ -56,8 +58,7 @@
* Add a new ``PyMemAllocator`` structure::
typedef struct {
- /* user context passed as the first argument
- to the 3 functions */
+ /* user context passed as the first argument to the 3 functions */
void *ctx;
/* allocate a memory block */
@@ -82,7 +83,7 @@
- ``PYMEM_DOMAIN_OBJ``: ``PyObject_Malloc()``, ``PyObject_Realloc()``
and ``PyObject_Free()``
-* Add new functions to get and set memory allocators:
+* Add new functions to get and set memory block allocators:
- ``void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
- ``void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)``
@@ -94,8 +95,7 @@
* Add a new ``PyObjectArenaAllocator`` structure::
typedef struct {
- /* user context passed as the first argument
- to the 2 functions */
+ /* user context passed as the first argument to the 2 functions */
void *ctx;
/* allocate an arena */
@@ -111,18 +111,17 @@
- ``void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)``
- ``void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)``
-* Add a new function to setup the debug checks on memory allocators when
- a memory allocator is replaced:
+* Add a new function to reinstall the debug checks on memory allocators when
+ a memory allocator is replaced with ``PyMem_SetAllocator()``:
- ``void PyMem_SetupDebugHooks(void)``
- - Install the debug hook on all memory block allocators. The function
- can be called more than once, hooks are not reinstalled if they
- were already installed.
- - The function does nothing is Python is not compiled in debug mode
+ - Install the debug hooks on all memory block allocators. The function can be
+ called more than once, hooks are only installed once.
+ - The function does nothing is Python is not compiled in debug mode.
-* Memory allocators always returns *NULL* if size is greater than
- ``PY_SSIZE_T_MAX``. The check is done before calling the
- inner function.
+* Memory block allocators always return *NULL* if *size* is greater than
+ ``PY_SSIZE_T_MAX``. The check is done before calling the inner
+ function.
The *pymalloc* allocator is optimized for objects smaller than 512 bytes
with a short lifetime. It uses memory mappings with a fixed size of 256
@@ -140,8 +139,8 @@
and ``free()``
-Redesign Debug Checks on Memory Allocators as Hooks
-----------------------------------------------------
+Redesign Debug Checks on Memory Block Allocators as Hooks
+---------------------------------------------------------
Since Python 2.3, Python implements different checks on memory
allocators in debug mode:
@@ -157,7 +156,8 @@
``PyMem_Realloc()``, ``PyMem_Free()``, ``PyObject_Malloc()``,
``PyObject_Realloc()`` and ``PyObject_Free()`` using macros. The new
allocator allocates a larger buffer and write a pattern to detect buffer
-underflow and overflow. It uses the original ``PyObject_Malloc()``
+underflow, buffer overflow and use after free (fill the buffer with the
+pattern ``0xDB``). It uses the original ``PyObject_Malloc()``
function to allocate memory. So ``PyMem_Malloc()`` and
``PyMem_Realloc()`` call indirectly ``PyObject_Malloc()`` and
``PyObject_Realloc()``.
@@ -178,13 +178,14 @@
* ``PyObject_Free()`` => ``_PyMem_DebugFree()``
=> ``_PyObject_Free()``
-As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call
-``malloc()`` and ``realloc()``, instead of calling ``PyObject_Malloc()``
-and ``PyObject_Realloc()`` in debug mode.
+As a result, ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now call
+``malloc()`` and ``realloc()`` in release mode and in debug mode,
+instead of calling ``PyObject_Malloc()`` and ``PyObject_Realloc()`` in
+debug mode.
When at least one memory allocator is replaced with
``PyMem_SetAllocator()``, the ``PyMem_SetupDebugHooks()`` function must
-be called to install the debug hooks on top on the new allocator.
+be called to reinstall the debug hooks on top on the new allocator.
Don't call malloc() directly anymore
@@ -195,7 +196,7 @@
``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
``realloc()``
-Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
+Direct calls to ``malloc()`` are replaced with ``PyMem_Malloc()``, or
``PyMem_RawMalloc()`` if the GIL is not held.
Configure external libraries like zlib or OpenSSL to allocate memory
@@ -205,22 +206,22 @@
For the "track memory usage" use case, it is important to track memory
allocated in external libraries to have accurate reports, because these
-allocations may be large.
+allocations can be large (can raise a ``MemoryError`` exception).
If an hook is used to the track memory usage, the memory allocated by
-``malloc()`` will not be tracked. Remaining ``malloc()`` in external
-libraries like OpenSSL or bz2 may allocate large memory blocks and so
-would be missed in memory usage reports.
+direct calls to ``malloc()`` will not be tracked. Remaining ``malloc()``
+in external libraries like OpenSSL or bz2 can allocate large memory
+blocks and so would be missed in memory usage reports.
Examples
========
-Use case 1: Replace Memory Allocator, keep pymalloc
+Use case 1: Replace Memory Allocators, keep pymalloc
----------------------------------------------------
Dummy example wasting 2 bytes per memory block,
-and 10 bytes per memory mapping::
+and 10 bytes per *pymalloc* arena::
#include <stdlib.h>
@@ -267,6 +268,7 @@
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
+ /* leave PYMEM_DOMAIN_OBJ unchanged, use pymalloc */
arena.ctx = &arena_padding;
arena.alloc = my_alloc_arena;
@@ -277,7 +279,7 @@
}
-Use case 2: Replace Memory Allocator, override pymalloc
+Use case 2: Replace Memory Allocators, override pymalloc
--------------------------------------------------------
If your allocator is optimized for allocations of objects smaller than
@@ -322,11 +324,14 @@
PyMem_SetupDebugHooks();
}
+The *pymalloc* arena does not need to be replaced, because it is no more
+used by the new allocator.
-Use case 3: Setup Allocator Hooks
----------------------------------
-Example to setup hooks on all memory allocators::
+Use case 3: Setup Hooks On Memory Block Allocators
+--------------------------------------------------
+
+Example to setup hooks on all memory block allocators::
struct {
PyMemAllocator raw;
@@ -390,22 +395,23 @@
}
.. note::
- ``PyMem_SetupDebugHooks()`` does not need to be called because the
- allocator is not replaced: Python debug hooks are installed
- automatically at startup.
+ ``PyMem_SetupDebugHooks()`` does not need to be called because
+ memory allocator are not replaced: the debug checks on memory
+ block allocators are installed automatically at startup.
Performances
============
+The implementation of this PEP (issue #3329) has no visible overhead on
+the Python benchmark suite.
+
Results of the `Python benchmarks suite
<http://hg.python.org/benchmarks>`_ (-b 2n3): some tests are 1.04x
-faster, some tests are 1.04 slower, significant is between 115 and -191.
+faster, some tests are 1.04 slower. Results of pybench microbenchmark:
+"+0.1%" slower globally (diff between -4.9% and +5.6%).
-Results of pybench benchmark: "+0.1%" slower globally (diff between
--4.9% and +5.6%).
-
-The full reports are attached to the issue #3329.
+The full output of benchmarks is attached to the issue #3329.
Rejected Alternatives
@@ -428,8 +434,9 @@
* ``void PyMem_SetAllocator(PyMemAllocator *allocator)``
* ``void PyObject_SetAllocator(PyMemAllocator *allocator)``
-With more specific functions, it becomes more difficult to write generic
-code, like reusing the same code for different allocator domains.
+This alternative was rejected because it is not possible to write
+generic code with more specific functions: code must be duplicated for
+each memory allocator domain.
Make PyMem_Malloc() reuse PyMem_RawMalloc() by default
@@ -439,25 +446,25 @@
calling ``PyMem_SetAllocator(PYMEM_DOMAIN_RAW, alloc)`` would also also
patch ``PyMem_Malloc()`` indirectly.
-This option was rejected because ``PyMem_SetAllocator()`` would have a
-different behaviour depending on the domain. Always having the same
-behaviour is less error-prone.
+This alternative was rejected because ``PyMem_SetAllocator()`` would
+have a different behaviour depending on the domain. Always having the
+same behaviour is less error-prone.
Add a new PYDEBUGMALLOC environment variable
--------------------------------------------
-To be able to use the Python builtin debug hooks even when a custom
-memory allocator replaces the default Python allocator, an environment
-variable ``PYDEBUGMALLOC`` can be added to setup these debug function
-hooks, instead of adding the new function ``PyMem_SetupDebugHooks()``.
-If the environment variable is present, ``PyMem_SetRawAllocator()``,
-``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()`` will reinstall
-automatically the hook on top of the new allocator.
+Add a new ``PYDEBUGMALLOC`` environment variable to enable debug checks
+on memory block allocators. The environment variable replaces the new
+function ``PyMem_SetupDebugHooks()`` which is not needed anymore.
+Another advantage is to allow to enable debug checks even in release
+mode: debug checks are always compiled, but only enabled when the
+environment variable is present and non-empty.
-A new environment variable would make the Python initialization even
-more complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_
-tries to simply the CPython startup sequence.
+This alternative was rejected because a new environment variable would
+make the Python initialization even more complex. The `PEP 432
+<http://www.python.org/dev/peps/pep-0432/>`_ tries to simply the CPython
+startup sequence.
Use macros to get customizable allocators
@@ -503,7 +510,7 @@
void* _PyMem_MallocTrace(const char *filename, int lineno,
size_t size);
- /* need also a function for the Python stable ABI */
+ /* the function is still needed for the Python stable ABI */
void* PyMem_Malloc(size_t size);
#define PyMem_Malloc(size) \
@@ -527,19 +534,19 @@
calls indirectly ``PyObject_Malloc()`` which requires the GIL to be
held. That's why ``PyMem_Malloc()`` must be called with the GIL held.
-This PEP proposes changes ``PyMem_Malloc()``: it now always call
-``malloc()``. The "GIL must be held" restriction can be removed from
+This PEP changes ``PyMem_Malloc()``: it now always call ``malloc()``.
+The "GIL must be held" restriction could be removed from
``PyMem_Malloc()``.
This alternative was rejected because allowing to call
-``PyMem_Malloc()`` without holding the GIL might break applications
+``PyMem_Malloc()`` without holding the GIL can break applications
which setup their own allocators or allocator hooks. Holding the GIL is
convinient to develop a custom allocator: no need to care of other
threads. It is also convinient for a debug allocator hook: Python
internal objects can be safetly inspected.
Calling ``PyGILState_Ensure()`` in
-a memory allocator may have unexpected behaviour, especially at Python
+a memory allocator has unexpected behaviour, especially at Python
startup and at creation of a new Python thread state.
@@ -552,13 +559,14 @@
The ``PyMem_Malloc()`` is used without the GIL held in some Python
functions. For example, the ``main()`` and ``Py_Main()`` functions of
Python call ``PyMem_Malloc()`` whereas the GIL do not exist yet. In this
-case, ``PyMem_Malloc()`` should be replaced with ``malloc()`` (or
+case, ``PyMem_Malloc()`` would be replaced with ``malloc()`` (or
``PyMem_RawMalloc()``).
-If an hook is used to the track memory usage, the memory allocated by
-direct calls to ``malloc()`` will not be tracked. External libraries
-like OpenSSL or bz2 should not call ``malloc()`` directly, so large
-allocated will be included in memory usage reports.
+This alternative was rejected because ``PyMem_RawMalloc()`` is required
+for accurate reports of the memory usage. When a debug hook is used to
+track the memory usage, the memory allocated by direct calls to
+``malloc()`` cannot be tracked. ``PyMem_RawMalloc()`` can be hooked and
+so all the memory allocated by Python can be tracked.
Use existing debug tools to analyze the memory
@@ -571,11 +579,11 @@
<http://www.nongnu.org/failmalloc/>`_, etc.
The problem is to retrieve the Python object related to a memory pointer
-to read its type and/or content. Another issue is to retrieve the
+to read its type and/or its content. Another issue is to retrieve the
location of the memory allocation: the C backtrace is usually useless
-(same reasoning than macros using ``__FILE__`` and ``__LINE__``), the
-Python filename and line number (or even the Python traceback) is more
-useful.
+(same reasoning than macros using ``__FILE__`` and ``__LINE__``, see
+`Pass the C filename and line number`_), the Python filename and line
+number (or even the Python traceback) is more useful.
This alternative was rejected because classic tools are unable to
introspect Python internals to collect such information. Being able to
@@ -586,8 +594,8 @@
Add a msize() function
----------------------
-Add another field to ``PyMemAllocator`` and ``PyObjectArenaAllocator``
-structures::
+Add another function to ``PyMemAllocator`` and
+``PyObjectArenaAllocator`` structures::
size_t msize(void *ptr);
@@ -607,8 +615,8 @@
platforms implement it. For example, Linux with the GNU libc does not
provide a function to get the size of a memory block. ``msize()`` is not
currently used in the Python source code. The function is only used to
-track the memory usage, but makes the API more complex. A debug hook can
-implemente the function internally, there is no need to add it to
+track the memory usage, and makes the API more complex. A debug hook can
+implement the function internally, there is no need to add it to
``PyMemAllocator`` and ``PyObjectArenaAllocator`` structures.
@@ -653,17 +661,19 @@
* lzma: `LZMA SDK - How to Use
<http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html>`_,
pass an opaque pointer
-* lipmpdec doesn't have this extra *ctx* parameter
+* lipmpdec: no opaque pointer (classic malloc API)
Other libraries:
* glib: `g_mem_set_vtable()
<http://developer.gnome.org/glib/unstable/glib-Memory-Allocation.html#g-mem-set-vtable>`_
-* libxml2: `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
+* libxml2:
+ `xmlGcMemSetup() <http://xmlsoft.org/html/libxml-xmlmemory.html>`_,
global
* Oracle's OCI: `Oracle Call Interface Programmer's Guide,
Release 2 (9.2)
- <http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_
+ <http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm>`_,
+ pass an opaque pointer
See also the `GNU libc: Memory Allocation Hooks
<http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html>`_.
@@ -676,30 +686,30 @@
Its implementation depends on the platform and of the C library. The GNU
C library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
(dlmalloc). FreeBSD uses `jemalloc
-<http://www.canonware.com/jemalloc/>`_. Google provides tcmalloc which
+<http://www.canonware.com/jemalloc/>`_. Google provides *tcmalloc* which
is part of `gperftools <http://code.google.com/p/gperftools/>`_.
``malloc()`` uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256
KB), whereas the heap is used for small allocations.
-On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls on
-Linux, and it is contiguous. On Windows, the heap is handled by
-``HeapAlloc()`` and may be discontiguous. Memory mappings are handled by
-``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they may be
+On UNIX, the heap is handled by ``brk()`` and ``sbrk()`` system calls,
+and it is contiguous. On Windows, the heap is handled by
+``HeapAlloc()`` and can be discontiguous. Memory mappings are handled by
+``mmap()`` on UNIX and ``VirtualAlloc()`` on Windows, they can be
discontiguous.
Releasing a memory mapping gives back immediatly the memory to the
-system. On UNIX, heap memory is only given back to the system if it is
-at the end of the heap. Otherwise, the memory will only be given back to
-the system when all the memory located after the released memory are
-also released.
+system. On UNIX, the heap memory is only given back to the system if the
+released block is located at the end of the heap. Otherwise, the memory
+will only be given back to the system when all the memory located after
+the released memory is also released.
-To allocate memory in the heap, the allocator tries to reuse free space.
-If there is no contiguous space big enough, the heap must be increased,
-even if we have more free space than required size. This issue is
+To allocate memory on the heap, an allocator tries to reuse free space.
+If there is no contiguous space big enough, the heap must be enlarged,
+even if there is more free space than required size. This issue is
called the "memory fragmentation": the memory usage seen by the system
-may be much higher than real usage. On Windows, ``HeapAlloc()`` creates
+is higher than real usage. On Windows, ``HeapAlloc()`` creates
a new memory mapping with ``VirtualAlloc()`` if there is not enough free
contiguous memory.
@@ -730,8 +740,8 @@
<http://bugs.python.org/issue3329>`_
* `Issue #13483: Use VirtualAlloc to allocate memory arenas
<http://bugs.python.org/issue13483>`_
-* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
- isn't thread safe <http://bugs.python.org/issue16742>`_
+* `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline,
+ which isn't thread safe <http://bugs.python.org/issue16742>`_
* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
PyMem_RawMalloc() <http://bugs.python.org/issue18203>`_
* `Issue #18227: Use Python memory allocators in external libraries like
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list