[Python-checkins] peps (merge default -> default): merge
christian.heimes
python-checkins at python.org
Thu Oct 3 22:29:32 CEST 2013
http://hg.python.org/peps/rev/b44fe7b71a00
changeset: 5168:b44fe7b71a00
parent: 5167:f7da0866e6dc
parent: 5165:6af6b504c635
user: Christian Heimes <christian at cheimes.de>
date: Thu Oct 03 22:29:22 2013 +0200
summary:
merge
files:
pep-0454.txt | 986 +++++++++++++++++++++++++-------------
1 files changed, 645 insertions(+), 341 deletions(-)
diff --git a/pep-0454.txt b/pep-0454.txt
--- a/pep-0454.txt
+++ b/pep-0454.txt
@@ -21,17 +21,16 @@
=========
Common debug tools tracing memory allocations read the C filename and
-number. Using such tool to analyze Python memory allocations does not
-help because most memory block are allocated in the same C function,
+line number. Using such tool to analyze Python memory allocations does
+not help because most memory block are allocated in the same C function,
in ``PyMem_Malloc()`` for example.
There are debug tools dedicated to the Python language like ``Heapy``
-and ``PySizer``. These projects analyze objects type and/or content.
-These tools are useful when most memory leaks are instances of the
-same type and this type is only instancied in a few functions. The
-problem is when the object type is very common like ``str`` or
-``tuple``, and it is hard to identify where these objects are
-instancied.
+and ``PySizer``. These tools analyze objects type and/or content. They
+are useful when most memory leaks are instances of the same type and
+this type is only instantiated in a few functions. The problem is when
+the object type is very common like ``str`` or ``tuple``, and it is hard
+to identify where these objects are instantiated.
Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools cannot
@@ -43,18 +42,17 @@
========
Using the PEP 445, it becomes easy to setup an hook on Python memory
-allocators. The hook can inspect the current Python frame to get the
-Python filename and line number.
+allocators. A hook can inspect Python internals to retrieve the Python
+tracebacks.
This PEP proposes to add a new ``tracemalloc`` module. It is a debug
-tool to trace memory allocations made by Python. The module provides the
+tool to trace memory blocks allocated by Python. The module provides the
following information:
* Compute the differences between two snapshots to detect memory leaks
-* Statistics on allocated memory blocks per filename and per line number:
- total size, number and average size of allocated memory blocks
-* For each allocated memory block: its size and the traceback where the block
- was allocated
+* Statistics on allocated memory blocks per filename and per line
+ number: total size, number and average size of allocated memory blocks
+* Traceback where a memory block was allocated
The API of the tracemalloc module is similar to the API of the
faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()``
@@ -72,17 +70,161 @@
===
To trace most memory blocks allocated by Python, the module should be
-enabled as early as possible by calling ``tracemalloc.enable()``
-function, by setting the ``PYTHONTRACEMALLOC`` environment variable to
-``1``, or by using ``-X tracemalloc`` command line option.
+enabled as early as possible by setting the ``PYTHONTRACEMALLOC``
+environment variable to ``1``, or by using ``-X tracemalloc`` command
+line option. The ``tracemalloc.enable()`` function can also be called to
+start tracing Python memory allocations.
-By default, the ``Trace.traceback`` attribute only stores one ``Frame``
-instance per allocated memory block. Use ``set_traceback_limit()`` to
-store more frames.
+By default, a trace of an allocated memory block only stores one frame.
+Use the ``set_traceback_limit()`` function to store more frames.
+Python memory blocks allocated in the ``tracemalloc`` module are also
+traced by default. Use ``add_exclude_filter(tracemalloc.__file__)`` to
+ignore these these memory allocations.
-Functions
----------
+At fork, the module is automatically disabled in the child process.
+
+
+Main Functions
+--------------
+
+``cancel_tasks()`` function:
+
+ Cancel all scheduled tasks.
+
+ See also the ``get_tasks()`` function.
+
+
+``clear_traces()`` function:
+
+ Clear all traces and statistics on Python memory allocations, and
+ reset the ``get_arena_size()`` and ``get_traced_memory()`` counters.
+
+
+``disable()`` function:
+
+ Stop tracing Python memory allocations and cancel scheduled tasks.
+
+ See also ``enable()`` and ``is_enabled()`` functions.
+
+
+``enable()`` function:
+
+ Start tracing Python memory allocations.
+
+ At fork, the module is automatically disabled in the child process.
+
+ See also ``disable()`` and ``is_enabled()`` functions.
+
+
+``get_stats()`` function:
+
+ Get statistics on traced Python memory blocks as a dictionary
+ ``{filename (str): {line_number (int): stats}}`` where *stats* in a
+ ``(size: int, count: int)`` tuple, *filename* and *line_number* can
+ be ``None``.
+
+ Return an empty dictionary if the ``tracemalloc`` module is
+ disabled.
+
+ See also the ``get_traces()`` function.
+
+
+``get_tasks()`` function:
+
+ Get the list of scheduled tasks, list of ``Task`` instances.
+
+
+``is_enabled()`` function:
+
+ ``True`` if the ``tracemalloc`` module is tracing Python memory
+ allocations, ``False`` otherwise.
+
+ See also ``enable()`` and ``disable()`` functions.
+
+
+Trace Functions
+---------------
+
+``get_traceback_limit()`` function:
+
+ Get the maximum number of frames stored in the traceback of a trace
+ of a memory block.
+
+ Use the ``set_traceback_limit()`` function to change the limit.
+
+
+``get_object_address(obj)`` function:
+
+ Get the address of the memory block of the specified Python object.
+
+ A Python object can be composed by multiple memory blocks, the
+ function only returns the address of the main memory block.
+
+ See also ``get_object_trace()`` and ``gc.get_referrers()`` functions.
+
+
+``get_object_trace(obj)`` function:
+
+ Get the trace of a Python object *obj* as a ``(size: int,
+ traceback)`` tuple where *traceback* is a tuple of ``(filename: str,
+ lineno: int)`` tuples, *filename* and *lineno* can be ``None``.
+
+ The function only returns the trace of the main memory block of the
+ object. The *size* of the trace is smaller than the total size of
+ the object if the object is composed by more than one memory block.
+
+ Return ``None`` if the ``tracemalloc`` module did not trace the
+ allocation of the object.
+
+ See also ``get_object_address()``, ``get_trace()``,
+ ``get_traces()``, ``gc.get_referrers()`` and ``sys.getsizeof()``
+ functions.
+
+
+``get_trace(address)`` function:
+
+ Get the trace of a memory block as a ``(size: int, traceback)``
+ tuple where *traceback* is a tuple of ``(filename: str, lineno:
+ int)`` tuples, *filename* and *lineno* can be ``None``.
+
+ Return ``None`` if the ``tracemalloc`` module did not trace the
+ allocation of the memory block.
+
+ See also ``get_object_trace()``, ``get_stats()`` and
+ ``get_traces()`` functions.
+
+
+``get_traces()`` function:
+
+ Get all traces of Python memory allocations as a dictionary
+ ``{address (int): trace}`` where *trace* is a ``(size: int,
+ traceback)`` and *traceback* is a list of ``(filename: str, lineno:
+ int)``. *traceback* can be empty, *filename* and *lineno* can be
+ None.
+
+ Return an empty dictionary if the ``tracemalloc`` module is
+ disabled.
+
+ See also ``get_object_trace()``, ``get_stats()`` and ``get_trace()``
+ functions.
+
+
+``set_traceback_limit(nframe: int)`` function:
+
+ Set the maximum number of frames stored in the traceback of a trace
+ of a memory block.
+
+ Storing the traceback of each memory allocation has an important
+ overhead on the memory usage. Use the ``get_tracemalloc_memory()``
+ function to measure the overhead and the ``add_filter()`` function
+ to select which memory allocations are traced.
+
+ Use the ``get_traceback_limit()`` function to get the current limit.
+
+
+Filter Functions
+----------------
``add_filter(filter)`` function:
@@ -94,14 +236,15 @@
allocation is ignored if at least one exclusive filter matchs its
trace.
- The new filter is not applied on already collected traces. Use
- ``clear_traces()`` to ensure that all traces match the new filter.
-
+ The new filter is not applied on already collected traces. Use the
+ ``clear_traces()`` function to ensure that all traces match the new
+ filter.
``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
- Add an inclusive filter: helper for ``add_filter()`` creating a
- ``Filter`` instance with ``include`` attribute set to ``True``.
+ Add an inclusive filter: helper for the ``add_filter()`` method
+ creating a ``Filter`` instance with the ``Filter.include`` attribute
+ set to ``True``.
Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)``
only includes memory blocks allocated by the ``tracemalloc`` module.
@@ -109,8 +252,9 @@
``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function:
- Add an exclusive filter: helper for ``add_filter()`` creating a
- ``Filter`` instance with ``include`` attribute set to ``False``.
+ Add an exclusive filter: helper for the ``add_filter()`` method
+ creating a ``Filter`` instance with the ``Filter.include`` attribute
+ set to ``False``.
Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)``
ignores memory blocks allocated by the ``tracemalloc`` module.
@@ -120,26 +264,7 @@
Reset the filter list.
-
-``clear_traces()`` function:
-
- Clear all traces and statistics on Python memory allocations, and
- reset the ``get_traced_memory()`` counter.
-
-
-``disable()`` function:
-
- Stop tracing Python memory allocations and stop the timer started by
- ``start_timer()``.
-
- See also ``enable()`` and ``is_enabled()`` functions.
-
-
-``enable()`` function:
-
- Start tracing Python memory allocations.
-
- See also ``disable()`` and ``is_enabled()`` functions.
+ See also the ``get_filters()`` function.
``get_filters()`` function:
@@ -147,131 +272,127 @@
Get the filters on Python memory allocations as list of ``Filter``
instances.
+ See also the ``clear_filters()`` function.
-``get_traceback_limit()`` function:
- Get the maximum number of ``Frame`` instances stored in the
- ``traceback`` attribute of a ``Trace`` instance.
+Metric Functions
+----------------
- Use ``set_traceback_limit()`` to change the limit.
+The following functions can be used to add metrics to a snapshot, see
+the ``Snapshot.add_metric()`` method.
+``get_allocated_blocks()`` function:
-``get_object_address(obj)`` function:
+ Get the current number of allocated memory blocks.
- Get the address of the memory block of the specified Python object.
+``get_arena_size()`` function:
-``get_object_trace(obj)`` function:
+ Get the size in bytes of traced arenas.
- Get the trace of a Python object *obj* as a ``Trace`` instance.
-
- The function only returns the trace of the memory block directly
- holding to object. The ``size`` attribute of the trace is smaller
- than the total size of the object if the object is composed of more
- than one memory block.
-
- Return ``None`` if the ``tracemalloc`` module did not trace the
- allocation of the object.
-
- See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.
+ See also the ``get_pymalloc_stats()`` function.
``get_process_memory()`` function:
- Get the memory usage of the current process as a meminfo namedtuple
- with two attributes:
-
- * ``rss``: Resident Set Size in bytes
- * ``vms``: size of the virtual memory in bytes
+ Get the memory usage of the current process as a ``(rss: int, vms:
+ int)`` tuple, *rss* is the "Resident Set Size" in bytes and *vms* is
+ the size of the virtual memory in bytes
Return ``None`` if the platform is not supported.
-``get_stats()`` function:
+``get_pymalloc_stats()`` function:
- Get statistics on traced Python memory blocks as a dictionary
- ``{filename (str): {line_number (int): stats}}`` where *stats* in a
- ``TraceStats`` instance, *filename* and *line_number* can be
- ``None``.
+ Get statistics on the ``pymalloc`` allocator as a dictionary.
- Return an empty dictionary if the ``tracemalloc`` module is
- disabled.
+ +---------------------+-------------------------------------------------------+
+ | Key | Description |
+ +=====================+=======================================================+
+ | ``alignment`` | Alignment of addresses returned to the user. |
+ +---------------------+-------------------------------------------------------+
+ | ``threshold`` | Small block threshold in bytes: pymalloc uses |
+ | | PyMem_RawMalloc() for allocation greater than |
+ | | threshold. |
+ +---------------------+-------------------------------------------------------+
+ | ``nalloc`` | Number of times object malloc called |
+ +---------------------+-------------------------------------------------------+
+ | ``arena_size`` | Arena size in bytes |
+ +---------------------+-------------------------------------------------------+
+ | ``total_arenas`` | Number of calls to new_arena(): total number of |
+ | | allocated arenas, including released arenas |
+ +---------------------+-------------------------------------------------------+
+ | ``max_arenas`` | Maximum number of arenas |
+ +---------------------+-------------------------------------------------------+
+ | ``arenas`` | Number of arenas currently allocated |
+ +---------------------+-------------------------------------------------------+
+ | ``allocated_bytes`` | Number of bytes in allocated blocks |
+ +---------------------+-------------------------------------------------------+
+ | ``available_bytes`` | Number of bytes in available blocks in used pools |
+ +---------------------+-------------------------------------------------------+
+ | ``pool_size`` | Pool size in bytes |
+ +---------------------+-------------------------------------------------------+
+ | ``free_pools`` | Number of unused pools |
+ +---------------------+-------------------------------------------------------+
+ | ``pool_headers`` | Number of bytes wasted in pool headers |
+ +---------------------+-------------------------------------------------------+
+ | ``quantization`` | Number of bytes in used and full pools wasted due to |
+ | | quantization, i.e. the necessarily leftover space at |
+ | | the ends of used and full pools. |
+ +---------------------+-------------------------------------------------------+
+ | ``arena_alignment`` | Number of bytes for arena alignment padding |
+ +---------------------+-------------------------------------------------------+
+
+ The function is not available if Python is compiled without ``pymalloc``.
+
+ See also ``get_arena_size()`` and ``sys._debugmallocstats()`` functions.
``get_traced_memory()`` function:
- Get the total size of all traced memory blocks allocated by Python.
+ Get the current size and maximum size of memory blocks traced by the
+ ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.
-``get_tracemalloc_size()`` function:
+``get_tracemalloc_memory()`` function:
- Get the memory usage in bytes of the ``tracemalloc`` module.
+ Get the memory usage in bytes of the ``tracemalloc`` module as a
+ tuple: ``(size: int, free: int)``.
+ * *size*: total size of bytes allocated by the module,
+ including *free* bytes
+ * *free*: number of free bytes available to store data
-``get_traces(obj)`` function:
- Get all traces of Python memory allocations as a dictionary
- ``{address (int): trace}`` where *trace* is a ``Trace`` instance.
+``get_unicode_interned()`` function:
- Return an empty dictionary if the ``tracemalloc`` module is
- disabled.
+ Get the size in bytes and the length of the dictionary of Unicode
+ interned strings as a ``(size: int, length: int)`` tuple.
+ The size is the size of the dictionary, excluding the size of
+ strings.
-``is_enabled()`` function:
- ``True`` if the ``tracemalloc`` module is tracing Python memory
- allocations, ``False`` otherwise.
-
- See also ``enable()`` and ``disable()`` functions.
-
-
-``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function:
-
- Start a timer calling ``func(*args, **kwargs)`` every *delay*
- seconds. Enable the ``tracemalloc`` module if it is disabled. The
- timer is based on the Python memory allocator, it is not real time.
- *func* is called after at least *delay* seconds, it is not called
- exactly after *delay* seconds if no Python memory allocation
- occurred. The timer has a resolution of 1 second.
-
- If the ``start_timer()`` function is called twice, previous
- parameters are replaced. Call the ``stop_timer()`` function to stop
- the timer.
-
- The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods
- use the ``start_timer()`` function to run regulary a task.
-
-
-``set_traceback_limit(limit: int)`` function:
-
- Set the maximum number of ``Frame`` instances stored in the
- ``traceback`` attribute of a ``Trace`` instance. Clear all traces
- and statistics on Python memory allocations if the ``tracemalloc``
- module is enabled,
-
- Storing the traceback of each memory allocation has an important
- overhead on the memory usage. Example with the Python test suite:
- tracing all memory allocations increases the memory usage by
- ``+50%`` when storing only 1 frame and ``+150%`` when storing 10
- frames. Use ``get_tracemalloc_size()`` to measure the overhead and
- ``add_filter()`` to select which memory allocations are traced.
-
- Use ``get_traceback_limit()`` to get the current limit.
-
-
-``stop_timer()`` function:
-
- Stop the timer started by ``start_timer()``.
-
-
-DisplayTop class
-----------------
+DisplayTop
+----------
``DisplayTop()`` class:
- Display the top of allocated memory blocks.
+ Display the top of allocated memory blocks.
-``display_snapshot(snapshot, count=10, group_by="filename_lineno", cumulative=False, file=None)`` method:
+``display(count=10, group_by="line", cumulative=False, file=None, callback=None)`` method:
+
+ Take a snapshot and display the top *count* biggest allocated memory
+ blocks grouped by *group_by*.
+
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the
+ newly created snapshot instance. Use the ``Snapshot.add_metric()``
+ method to add new metric.
+
+ Return the snapshot, a ``Snapshot`` instance.
+
+``display_snapshot(snapshot, count=10, group_by="line", cumulative=False, file=None)`` method:
Display a snapshot of memory blocks allocated by Python, *snapshot*
is a ``Snapshot`` instance.
@@ -284,65 +405,78 @@
``display_top_stats(top_stats, count=10, file=None)`` method:
Display the top of allocated memory blocks grouped by the
- ``group_by`` attribute of *top_stats*, *top_stats* is a
+ ``GroupedStats.group_by`` attribute of *top_stats*, *top_stats* is a
``GroupedStats`` instance.
+``average`` attribute:
+
+ If ``True`` (default value), display the average size of memory
+ blocks.
+
``color`` attribute:
If ``True``, always use colors. If ``False``, never use colors. The
default value is ``None``: use colors if the *file* parameter is a
TTY device.
-``compare_with_previous`` attribute:
+``compare_to_previous`` attribute:
- If ``True`` (default value), compare with the previous snapshot. If
- ``False``, compare with the first snapshot.
+ If ``True`` (default value), compare to the previous snapshot. If
+ ``False``, compare to the first snapshot.
``filename_parts`` attribute:
Number of displayed filename parts (int, default: ``3``). Extra
parts are replaced with ``'...'``.
-``show_average`` attribute:
+``metrics`` attribute:
- If ``True`` (default value), display the average size of memory blocks.
+ If ``True`` (default value), display metrics: see
+ ``Snapshot.metrics``.
-``show_count`` attribute:
+``previous_top_stats`` attribute:
- If ``True`` (default value), display the number of allocated memory
- blocks.
+ Previous ``GroupedStats`` instance, or first ``GroupedStats``
+ instance if ``compare_to_previous`` is ``False``, used to display
+ the differences between two snapshots.
-``show_size`` attribute:
+``size`` attribute:
If ``True`` (default value), display the size of memory blocks.
-DisplayTopTask class
---------------------
+DisplayTopTask
+--------------
-``DisplayTopTask(count=10, group_by="filename_lineno", cumulative=False, file=sys.stdout, user_data_callback=None)`` class:
+``DisplayTopTask(count=10, group_by="line", cumulative=False, file=sys.stdout, callback=None)`` class:
- Task taking temporary snapshots and displaying the top *count*
- memory allocations grouped by *group_by*.
+ Task taking temporary snapshots and displaying the top *count* memory
+ allocations grouped by *group_by*.
- Call the ``start()`` method to start the task.
+ ``DisplayTopTask`` is based on the ``Task`` class and so inherit
+ all attributes and methods, especially:
+
+ * ``Task.cancel()``
+ * ``Task.schedule()``
+ * ``Task.set_delay()``
+ * ``Task.set_memory_threshold()``
+
+ Modify the ``display_top`` attribute to customize the display.
``display()`` method:
- Take a snapshot and display the top *count* biggest allocated memory
- blocks grouped by *group_by* using the ``display_top`` attribute.
+ Take a snapshot and display the top ``count`` biggest allocated
+ memory blocks grouped by ``group_by`` using the ``display_top``
+ attribute.
Return the snapshot, a ``Snapshot`` instance.
-``start(delay: int)`` method:
+``callback`` attribute:
- Start a task using the ``start_timer()`` function calling the
- ``display()`` method every *delay* seconds.
-
-``stop()`` method:
-
- Stop the task started by the ``start()`` method using the
- ``stop_timer()`` function.
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the
+ newly created snapshot instance. Use the ``Snapshot.add_metric()``
+ method to add new metric.
``count`` attribute:
@@ -351,11 +485,11 @@
``cumulative`` attribute:
If ``True``, cumulate size and count of memory blocks of all frames
- of each ``Trace`` instance, not only the most recent frame. The
- default value is ``False``.
+ of each trace, not only the most recent frame. The default value is
+ ``False``.
- The option is ignored if the traceback limit is ``1``, see the
- ``get_traceback_limit()`` function.
+ The option is ignored if the traceback limit is less than ``2``, see
+ the ``get_traceback_limit()`` function.
``display_top`` attribute:
@@ -368,190 +502,279 @@
``group_by`` attribute:
Determine how memory allocations are grouped: see
- ``Snapshot.top_by`` for the available values.
+ ``Snapshot.top_by()`` for the available values.
-``user_data_callback`` attribute:
- Optional callback collecting user data (callable, default:
- ``None``). See ``Snapshot.create()``.
-
-
-Filter class
-------------
+Filter
+------
``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class:
- Filter to select which memory allocations are traced. Filters can be
- used to reduce the memory usage of the ``tracemalloc`` module, which
- can be read using ``get_tracemalloc_size()``.
-
-``match_trace(trace)`` method:
-
- Return ``True`` if the ``Trace`` instance must be kept according to
- the filter, ``False`` otherwise.
+ Filter to select which memory allocations are traced. Filters can be
+ used to reduce the memory usage of the ``tracemalloc`` module, which
+ can be read using the ``get_tracemalloc_memory()`` function.
``match(filename: str, lineno: int)`` method:
- Return ``True`` if the filename and line number must be kept
- according to the filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the filename and line number,
+ ``False`` otherwise.
``match_filename(filename: str)`` method:
- Return ``True`` if the filename must be kept according to the
- filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the filename, ``False``
+ otherwise.
``match_lineno(lineno: int)`` method:
- Return ``True`` if the line number must be kept according to the
- filter, ``False`` otherwise.
+ Return ``True`` if the filter matchs the line number, ``False``
+ otherwise.
+
+``match_traceback(traceback)`` method:
+
+ Return ``True`` if the filter matchs the *traceback*, ``False``
+ otherwise.
+
+ *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples.
``include`` attribute:
If *include* is ``True``, only trace memory blocks allocated in a
file with a name matching filename ``pattern`` at line number
- ``lineno``. If *include* is ``False``, ignore memory blocks
- allocated in a file with a name matching filename :attr`pattern` at
- line number ``lineno``.
+ ``lineno``.
+
+ If *include* is ``False``, ignore memory blocks allocated in a file
+ with a name matching filename :attr`pattern` at line number
+ ``lineno``.
+
+``lineno`` attribute:
+
+ Line number (``int``). If is is ``None`` or less than ``1``, it
+ matches any line number.
``pattern`` attribute:
The filename *pattern* can contain one or many ``*`` joker
characters which match any substring, including an empty string. The
- ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On
+ ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On
Windows, the comparison is case insensitive and the alternative
separator ``/`` is replaced with the standard separator ``\``.
-``lineno`` attribute:
-
- Line number (``int``). If is is ``None`` or lesser than ``1``, it
- matches any line number.
-
``traceback`` attribute:
- If *traceback* is ``True``, all frames of the ``traceback``
- attribute of ``Trace`` instances are checked. If *traceback* is
- ``False``, only the most recent frame is checked.
+ If *traceback* is ``True``, all frames of the traceback are checked.
+ If *traceback* is ``False``, only the most recent frame is checked.
- This attribute only has an effect on the ``match_trace()`` method
- and only if the traceback limit is greater than ``1``. See the
- ``get_traceback_limit()`` function.
+ This attribute is ignored if the traceback limit is less than ``2``.
+ See the ``get_traceback_limit()`` function.
-Frame class
------------
+GroupedStats
+------------
-``Frame`` class:
+``GroupedStats(timestamp: datetime.datetime, stats: dict, group_by: str, cumulative=False, metrics: dict=None)`` class:
- Trace of a Python frame, used by ``Trace.traceback`` attribute.
+ Top of allocated memory blocks grouped by *group_by* as a dictionary.
-``filename`` attribute:
-
- Python filename, ``None`` if unknown.
-
-``lineno`` attribute:
-
- Python line number, ``None`` if unknown.
-
-
-GroupedStats class
-------------------
-
-``GroupedStats(stats: dict, group_by: str, cumulative=False, timestamp=None, process_memory=None, tracemalloc_size=None)`` class:
-
- Top of allocated memory blocks grouped by on *group_by* as a
- dictionary.
-
- The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance.
+ The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance.
``compare_to(old_stats: GroupedStats=None)`` method:
Compare to an older ``GroupedStats`` instance. Return a
``StatsDiff`` instance.
+ The ``StatsDiff.differences`` list is not sorted: call the
+ ``StatsDiff.sort`` method to sort the list.
+
+ ``None`` values are replaced with an empty string for filenames or
+ zero for line numbers, because ``str`` and ``int`` cannot be
+ compared to ``None``.
+
``cumulative`` attribute:
If ``True``, cumulate size and count of memory blocks of all frames
- of ``Trace``, not only the most recent frame.
+ of the traceback of a trace, not only the most recent frame.
+
+``metrics`` attribute:
+
+ Dictionary storing metrics read when the snapshot was created:
+ ``{name (str): metric}`` where *metric* type is ``Metric``.
``group_by`` attribute:
- Determine how memory allocations were grouped. The type of ``stats``
- keys depends on *group_by*:
-
- ===================== ======================== ==============
- group_by description key type
- ===================== ======================== ==============
- ``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
- ``'address'`` memory block address ``int``
- ===================== ======================== ==============
-
- See the *group_by* parameter of the ``Snapshot.top_by()`` method.
+ Determine how memory allocations were grouped: see
+ ``Snapshot.top_by()`` for the available values.
``stats`` attribute:
Dictionary ``{key: stats}`` where the *key* type depends on the
- ``group_by`` attribute and *stats* type is ``TraceStats``.
+ ``group_by`` attribute and *stats* is a ``(size: int, count: int)``
+ tuple.
-``process_memory`` attribute:
-
- Result of the ``get_process_memory()`` function, can be ``None``.
+ See the ``Snapshot.top_by()`` method.
``timestamp`` attribute:
Creation date and time of the snapshot, ``datetime.datetime``
instance.
-``tracemalloc_size`` attribute:
- The memory usage in bytes of the ``tracemalloc`` module, result of
- the ``get_tracemalloc_size()`` function.
+Metric
+------
+``Metric(name: str, value: int, format: str)`` class:
-Snapshot class
---------------
+ Value of a metric when a snapshot is created.
-``Snapshot`` class:
+``name`` attribute:
- Snapshot of memory blocks allocated by Python.
+ Name of the metric.
- Use ``TakeSnapshot`` to take regulary snapshots.
+``value`` attribute:
+
+ Value of the metric.
+
+``format`` attribute:
+
+ Format of the metric:
+
+ * ``'int'``: a number
+ * ``'percent'``: percentage, ``1.0`` means ``100%``
+ * ``'size'``: a size in bytes
+
+
+Snapshot
+--------
+
+``Snapshot(timestamp: datetime.datetime, pid: int, traces: dict=None, stats: dict=None, metrics: dict=None)`` class:
+
+ Snapshot of traces and statistics on memory blocks allocated by
+ Python.
+
+ Use ``TakeSnapshotTask`` to take regulary snapshots.
+
+``add_gc_metrics()`` method:
+
+ Add a metric on garbage collector:
+
+ * ``gc.objects``: total number of Python objects
+
+ See the ``gc`` module.
+
+
+``add_metric(name: str, value: int, format: str)`` method:
+
+ Helper to add a ``Metric`` instance to ``Snapshot.metrics``. Return
+ the newly created ``Metric`` instance.
+
+ Raise an exception if the name is already present in
+ ``Snapshot.metrics``.
+
+
+``add_process_memory_metrics()`` method:
+
+ Add metrics on the process memory:
+
+ * ``process_memory.rss``: Resident Set Size
+ * ``process_memory.vms``: Virtual Memory Size
+
+ These metrics are only available if the ``get_process_memory()``
+ function is available on the platform.
+
+
+``add_pymalloc_metrics()`` method:
+
+ Add metrics on the Python memory allocator (``pymalloc``):
+
+ * ``pymalloc.blocks``: number of allocated memory blocks
+ * ``pymalloc.size``: size of ``pymalloc`` arenas
+ * ``pymalloc.max_size``: maximum size of ``pymalloc`` arenas
+ * ``pymalloc.allocated``: number of allocated bytes
+ * ``pymalloc.free``: number of free bytes
+ * ``pymalloc.fragmentation``: fragmentation percentage of the arenas
+
+ These metrics are only available if Python is compiled in debug
+ mode, except ``pymalloc.blocks`` which is always available.
+
+
+``add_tracemalloc_metrics()`` method:
+
+ Add metrics on the ``tracemalloc`` module:
+
+ * ``tracemalloc.traced.size``: size of memory blocks traced by the
+ ``tracemalloc`` module
+ * ``tracemalloc.traced.max_size``: maximum size of memory blocks
+ traced by the ``tracemalloc`` module
+ * ``tracemalloc.traces``: number of traces of Python memory blocks
+ * ``tracemalloc.module.size``: total size of bytes allocated by the
+ ``tracemalloc`` module, including free bytes
+ * ``tracemalloc.module.free``: number of free bytes available for
+ the ``tracemalloc`` module
+ * ``tracemalloc.module.fragmentation``: percentage of fragmentation
+ of the memory allocated by the ``tracemalloc`` module
+ * ``tracemalloc.arena_size``: size of traced arenas
+
+ ``tracemalloc.traces`` metric is only present if the snapshot was
+ created with traces.
+
+
+``add_unicode_metrics()`` method:
+
+ Add metrics on the Unicode interned strings:
+
+ * ``unicode_interned.size``: size of the dictionary, excluding size
+ of strings
+ * ``unicode_interned.len``: length of the dictionary
+
``apply_filters(filters)`` method:
- Apply a list filters on the ``traces`` and ``stats`` dictionaries,
+ Apply filters on the ``traces`` and ``stats`` dictionaries,
*filters* is a list of ``Filter`` instances.
-``create(\*, with_traces=False, with_stats=True, user_data_callback=None)`` classmethod:
+
+``create(traces=False, metrics=True)`` classmethod:
Take a snapshot of traces and/or statistics of allocated memory
blocks.
- If *with_traces* is ``True``, ``get_traces()`` is called and its
- result is stored in the ``traces`` attribute. This attribute
- contains more information than ``stats`` and uses more memory and
- more disk space. If *with_traces* is ``False``, ``traces`` is set to
- ``None``.
+ If *traces* is ``True``, ``get_traces`` is called and its result is
+ stored in the ``Snapshot.traces`` attribute. This attribute contains
+ more information than ``Snapshot.stats`` and uses more memory and
+ more disk space. If *traces* is ``False``, ``Snapshot.traces`` is
+ set to ``None``.
- If *with_stats* is ``True``, ``get_stats()`` is called and its
- result is stored in the ``Snapshot.stats`` attribute. If
- *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``.
+ If *metrics* is ``True``, fill ``Snapshot.metrics`` with metrics
+ using the following methods:
- *with_traces* and *with_stats* cannot be ``False`` at the same time.
+ * ``add_gc_metrics``
+ * ``add_process_memory_metrics``
+ * ``add_pymalloc_metrics``
+ * ``add_tracemalloc_metrics``
+ * ``add_unicode_metrics``
- *user_data_callback* is an optional callable object. Its result
- should be serializable by the ``pickle`` module, or
- ``Snapshot.write()`` would fail. If *user_data_callback* is set, it
- is called and the result is stored in the ``Snapshot.user_data``
- attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``.
+ If *metrics* is ``False``, ``Snapshot.metrics`` is set to an empty
+ dictionary.
+
+ Tracebacks of traces are limited to ``traceback_limit`` frames. Call
+ ``set_traceback_limit()`` before calling ``Snapshot.create()`` to
+ store more frames.
The ``tracemalloc`` module must be enabled to take a snapshot. See
- the ``enable()`` function.
+ the the ``enable()`` function.
-``load(filename)`` classmethod:
+``get_metric(name, default=None)`` method:
+
+ Get the value of the metric called *name*. Return *default* if the
+ metric does not exist.
+
+
+``load(filename, traces=True)`` classmethod:
Load a snapshot from a file.
+ If *traces* is ``False``, don't load traces.
+
+
``top_by(group_by: str, cumulative: bool=False)`` method:
Compute top statistics grouped by *group_by* as a ``GroupedStats``
@@ -561,99 +784,75 @@
group_by description key type
===================== ======================== ==============
``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
+ ``'line'`` filename and line number ``(str, int)``
``'address'`` memory block address ``int``
===================== ======================== ==============
If *cumulative* is ``True``, cumulate size and count of memory
- blocks of all frames of each ``Trace`` instance, not only the most
- recent frame. The *cumulative* parameter is ignored if *group_by* is
- ``'address'`` or if the traceback limit is ``1``. See the
- ``traceback_limit`` attribute.
+ blocks of all frames of the traceback of a trace, not only the most
+ recent frame. The *cumulative* parameter is ignored if *group_by*
+ is ``'address'`` or if the traceback limit is less than ``2``.
+
``write(filename)`` method:
- Write the snapshot into a file.
+ Write the snapshot into a file.
+
+
+``metrics`` attribute:
+
+ Dictionary storing metrics read when the snapshot was created:
+ ``{name (str): metric}`` where *metric* type is ``Metric``.
``pid`` attribute:
Identifier of the process which created the snapshot, result of
``os.getpid()``.
-``process_memory`` attribute:
-
- Memory usage of the current process, result of the
- ``get_process_memory()`` function. It can be ``None``.
-
``stats`` attribute:
Statistics on traced Python memory, result of the ``get_stats()``
- function, if ``create()`` was called with *with_stats* equals to
- ``True``, ``None`` otherwise.
-
-``tracemalloc_size`` attribute:
-
- The memory usage in bytes of the ``tracemalloc`` module, result of
- the ``get_tracemalloc_size()`` function.
+ function.
``traceback_limit`` attribute:
- The maximum number of frames stored in the ``traceback`` attribute
- of a ``Trace``, result of the ``get_traceback_limit()`` function.
+ Maximum number of frames stored in a trace of a memory block
+ allocated by Python.
``traces`` attribute:
Traces of Python memory allocations, result of the ``get_traces()``
- function, if ``create()`` was called with *with_traces* equals to
- ``True``, ``None`` otherwise.
-
- The ``traceback`` attribute of each ``Trace`` instance is limited to
- ``traceback_limit`` frames.
+ function, can be ``None``.
``timestamp`` attribute:
Creation date and time of the snapshot, ``datetime.datetime``
instance.
-``user_data`` attribute:
- Result of *user_data_callback* called in ``Snapshot.create()``
- (default: ``None``).
-
-
-StatsDiff class
----------------
+StatsDiff
+---------
``StatsDiff(differences, old_stats, new_stats)`` class:
- Differences between two ``GroupedStats`` instances. By default, the
- ``differences`` list is unsorted: call ``sort()`` to sort it.
+ Differences between two ``GroupedStats`` instances.
- The ``GroupedStats.compare_to()`` method creates a ``StatsDiff``
+ The ``GroupedStats.compare_to`` method creates a ``StatsDiff``
instance.
``sort()`` method:
- Sort the ``differences`` list from the biggest allocation to the
- smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and
- then by *key*.
+ Sort the ``differences`` list from the biggest difference to the
+ smallest difference. Sort by ``abs(size_diff)``, *size*,
+ ``abs(count_diff)``, *count* and then by *key*.
``differences`` attribute:
Differences between ``old_stats`` and ``new_stats`` as a list of
``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*,
*size*, *count_diff* and *count* are ``int``. The key type depends
- on the ``group_by`` attribute of ``new_stats``:
-
- ===================== ======================== ==============
- group_by description key type
- ===================== ======================== ==============
- ``'filename'`` filename ``str``
- ``'filename_lineno'`` filename and line number ``(str, str)``
- ``'address'`` memory block address ``int``
- ===================== ======================== ==============
-
- See the ``group_by`` attribute of the ``GroupedStats`` class.
+ on the ``GroupedStats.group_by`` attribute of ``new_stats``: see the
+ ``Snapshot.top_by()`` method.
``old_stats`` attribute:
@@ -664,43 +863,148 @@
New ``GroupedStats`` instance.
-Trace class
------------
+Task
+----
-``Trace`` class:
+``Task(func, *args, **kw)`` class:
- Debug information of a memory block allocated by Python.
+ Task calling ``func(*args, **kw)``. When scheduled, the task is
+ called when the traced memory is increased or decreased by more than
+ *threshold* bytes, or after *delay* seconds.
-``size`` attribute:
+``call()`` method:
- Size in bytes of the memory block.
+ Call ``func(*args, **kw)`` and return the result.
-``traceback`` attribute:
- Traceback where the memory block was allocated as a list of
- ``Frame`` instances, most recent first.
+``cancel()`` method:
- The list can be empty or incomplete if the ``tracemalloc`` module
- was unable to retrieve the full traceback.
+ Cancel the task.
- The traceback is limited to ``get_traceback_limit()`` frames. Use
- ``set_traceback_limit()`` to store more frames.
+ Do nothing if the task is not scheduled.
-TraceStats class
+``get_delay()`` method:
+
+ Get the delay in seconds. If the delay is ``None``, the timer is
+ disabled.
+
+
+``get_memory_threshold()`` method:
+
+ Get the threshold of the traced memory. When scheduled, the task is
+ called when the traced memory is increased or decreased by more than
+ *threshold* bytes. The memory threshold is disabled if *threshold*
+ is ``None``.
+
+ See also the ``set_memory_threshold()`` method and the
+ ``get_traced_memory()`` function.
+
+
+``schedule(repeat: int=None)`` method:
+
+ Schedule the task *repeat* times. If *repeat* is ``None``, the task
+ is rescheduled after each call until it is cancelled.
+
+ If the method is called twice, the task is rescheduled with the new
+ *repeat* parameter.
+
+ The task must have a memory threshold or a delay: see
+ ``set_delay()`` and ``set_memory_threshold()`` methods. The
+ ``tracemalloc`` must be enabled to schedule a task: see the
+ ``enable`` function.
+
+ The task is cancelled if the ``call()`` method raises an exception.
+ The task can be cancelled using the ``cancel()`` method or the
+ ``cancel_tasks()`` function.
+
+
+``set_delay(seconds: int)`` method:
+
+ Set the delay in seconds before the task will be called. Set the
+ delay to ``None`` to disable the timer.
+
+ The timer is based on the Python memory allocator, it is not real
+ time. The task is called after at least *delay* seconds, it is not
+ called exactly after *delay* seconds if no Python memory allocation
+ occurred. The timer has a resolution of 1 second.
+
+ The task is rescheduled if it was scheduled.
+
+
+``set_memory_threshold(size: int)`` method:
+
+ Set the threshold of the traced memory. When scheduled, the task is
+ called when the traced memory is increased or decreased by more than
+ *threshold* bytes. Set the threshold to ``None`` to disable it.
+
+ The task is rescheduled if it was scheduled.
+
+ See also the ``get_memory_threshold()`` method and the
+ ``get_traced_memory()`` function.
+
+
+``func`` attribute:
+
+ Function, callable object.
+
+``func_args`` attribute:
+
+ Function arguments, ``tuple``.
+
+``func_kwargs`` attribute:
+
+ Function keyword arguments, ``dict``. It can be ``None``.
+
+
+TakeSnapshotTask
----------------
-``TraceStats`` class:
+``TakeSnapshotTask(filename_template: str="tracemalloc-$counter.pickle", traces: bool=False, metrics: bool=True, callback: callable=None)`` class:
- Statistics on Python memory allocations.
+ Task taking snapshots of Python memory allocations and writing them
+ into files.
-``size`` attribute:
+ ``TakeSnapshotTask`` is based on the ``Task`` class and so inherit
+ all attributes and methods, especially:
- Total size in bytes of allocated memory blocks.
+ * ``Task.cancel()``
+ * ``Task.schedule()``
+ * ``Task.set_delay()``
+ * ``Task.set_memory_threshold()``
-``count`` attribute:
+``take_snapshot()`` method:
- Number of allocated memory blocks.
+ Take a snapshot and write it into a file. Return ``(snapshot,
+ filename)`` where *snapshot* is a ``Snapshot`` instance and filename
+ type is ``str``.
+
+``callback`` attribute:
+
+ *callback* is an optional callable object which can be used to add
+ metrics to a snapshot. It is called with only one parameter: the
+ newly created snapshot instance. Use the ``Snapshot.add_metric()``
+ method to add new metric.
+
+``filename_template`` attribute:
+
+ Template to create a filename. The template supports the following
+ variables:
+
+ * ``$pid``: identifier of the current process
+ * ``$timestamp``: current date and time
+ * ``$counter``: counter starting at 1 and incremented at each snapshot,
+ formatted as 4 decimal digits
+
+ The default template is ``'tracemalloc-$counter.pickle'``.
+
+``metrics`` attribute:
+
+ Parameter passed to the ``Snapshot.create()`` function.
+
+``traces`` attribute:
+
+ Parameter passed to the ``Snapshot.create()`` function.
Links
@@ -722,8 +1026,8 @@
* `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4
* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_
* `pympler <http://code.google.com/p/pympler/>`_
-* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of
- the CherryPy memory leak debugger
+* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version
+ of the CherryPy memory leak debugger
* `objgraph <http://mg.pov.lt/objgraph/>`_
* `caulk <https://github.com/smartfile/caulk/>`_
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list