Hi, Thanks to the early remarks on the PEP 454, I redesigned and enhanced the API of the new tracemalloc module. Changes: * it is now possibility to record more than 1 frame per memory allocation * add filters on filename and line number * new GroupedStats and StatsDiff class to generate and compare statistics * cumulative statistics * display the traceback of a memory block * almost all the code has unit tests * better documentation HTML version of the documentation: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html The documentation contains output examples and a short tutorial ("Usage"), but also the documentation of the command line, which are not included in the PEP. HTML version of the PEP: http://www.python.org/dev/peps/pep-0454/ Issue tracking the implementation: http://bugs.python.org/issue18874 PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== Add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Common debug tools tracing memory allocations read the C filename and number. Using such tool to analyze Python memory allocations does not help because most memory block are allocated in the same C function, in ``PyMem_Malloc()`` for example. There are debug tools dedicated to the Python language like ``Heapy`` and ``PySizer``. These projects analyze objects type and/or content. These tools are useful when most memory leaks are instances of the same type and this type is only instancied in a few functions. The problem is when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instancied. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the PEP 445, it becomes easy to setup an hook on Python memory allocators. The hook can inspect the current Python frame to get the Python filename and line number. This PEP proposes to add a new ``tracemalloc`` module. It is a debug tool to trace memory allocations made by Python. The module provides the following information: * Compute the differences between two snapshots to detect memory leaks * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * For each allocated memory block: its size and the traceback where the block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/dev/library/faulthandler.html>`_. The tracemalloc module has been written for CPython. Other implementations of Python may not provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by calling ``tracemalloc.enable()`` function, by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. By default, the ``Trace.traceback`` attribute only stores one ``Frame`` instance per allocated memory block. Use ``set_traceback_limit()`` to store more frames. Functions --------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filter match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use ``clear_traces()`` to ensure that all traces match the new filter. ``add_include_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for ``add_filter()`` creating a ``Filter`` instance with ``include`` attribute set to ``True``. Example: ``tracemalloc.add_include_filter(tracemalloc.__file__)`` only includes memory blocks allocated by the ``tracemalloc`` module. ``add_exclude_filter(filename: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for ``add_filter()`` creating a ``Filter`` instance with ``include`` attribute set to ``False``. Example: ``tracemalloc.add_exclude_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Reset the filter list. ``clear_traces()`` function: Clear all traces and statistics on Python memory allocations, and reset the ``get_traced_memory()`` counter. ``disable()`` function: Stop tracing Python memory allocations and stop the timer started by ``start_timer()``. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_filters()`` function: Get the filters on Python memory allocations as list of ``Filter`` instances. ``get_traceback_limit()`` function: Get the maximum number of ``Frame`` instances stored in the ``traceback`` attribute of a ``Trace`` instance. Use ``set_traceback_limit()`` to change the limit. ``get_object_address(obj)`` function: Get the address of the memory block of the specified Python object. ``get_object_trace(obj)`` function: Get the trace of a Python object *obj* as a ``Trace`` instance. The function only returns the trace of the memory block directly holding to object. The ``size`` attribute of the trace is smaller than the total size of the object if the object is composed of more than one memory block. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_process_memory()`` function: Get the memory usage of the current process as a meminfo namedtuple with two attributes: * ``rss``: Resident Set Size in bytes * ``vms``: size of the virtual memory in bytes Return ``None`` if the platform is not supported. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``TraceStats`` instance, *filename* and *line_number* can be ``None``. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``get_traced_memory()`` function: Get the total size of all traced memory blocks allocated by Python. ``get_tracemalloc_size()`` function: Get the memory usage in bytes of the ``tracemalloc`` module. ``get_traces(obj)`` function: Get all traces of Python memory allocations as a dictionary ``{address (int): trace}`` where *trace* is a ``Trace`` instance. Return an empty dictionary if the ``tracemalloc`` module is disabled. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``enable()`` and ``disable()`` functions. ``start_timer(delay: int, func: callable, args: tuple=(), kwargs: dict={})`` function: Start a timer calling ``func(*args, **kwargs)`` every *delay* seconds. Enable the ``tracemalloc`` module if it is disabled. The timer is based on the Python memory allocator, it is not real time. *func* is called after at least *delay* seconds, it is not called exactly after *delay* seconds if no Python memory allocation occurred. The timer has a resolution of 1 second. If the ``start_timer()`` function is called twice, previous parameters are replaced. Call the ``stop_timer()`` function to stop the timer. The ``DisplayTopTask.start()`` and ``TakeSnapshot.start()`` methods use the ``start_timer()`` function to run regulary a task. ``set_traceback_limit(limit: int)`` function: Set the maximum number of ``Frame`` instances stored in the ``traceback`` attribute of a ``Trace`` instance. Clear all traces and statistics on Python memory allocations if the ``tracemalloc`` module is enabled, Storing the traceback of each memory allocation has an important overhead on the memory usage. Example with the Python test suite: tracing all memory allocations increases the memory usage by ``+50%`` when storing only 1 frame and ``+150%`` when storing 10 frames. Use ``get_tracemalloc_size()`` to measure the overhead and ``add_filter()`` to select which memory allocations are traced. Use ``get_traceback_limit()`` to get the current limit. ``stop_timer()`` function: Stop the timer started by ``start_timer()``. DisplayTop class ---------------- ``DisplayTop()`` class: Display the top of allocated memory blocks. ``display_snapshot(snapshot, count=10, group_by="filename_lineno", cumulative=False, file=None)`` method: Display a snapshot of memory blocks allocated by Python, *snapshot* is a ``Snapshot`` instance. ``display_top_diff(top_diff, count=10, file=None)`` method: Display differences between two ``GroupedStats`` instances, *top_diff* is a ``StatsDiff`` instance. ``display_top_stats(top_stats, count=10, file=None)`` method: Display the top of allocated memory blocks grouped by the ``group_by`` attribute of *top_stats*, *top_stats* is a ``GroupedStats`` instance. ``color`` attribute: If ``True``, always use colors. If ``False``, never use colors. The default value is ``None``: use colors if the *file* parameter is a TTY device. ``compare_with_previous`` attribute: If ``True`` (default value), compare with the previous snapshot. If ``False``, compare with the first snapshot. ``filename_parts`` attribute: Number of displayed filename parts (int, default: ``3``). Extra parts are replaced with ``'...'``. ``show_average`` attribute: If ``True`` (default value), display the average size of memory blocks. ``show_count`` attribute: If ``True`` (default value), display the number of allocated memory blocks. ``show_size`` attribute: If ``True`` (default value), display the size of memory blocks. DisplayTopTask class -------------------- ``DisplayTopTask(count=10, group_by="filename_lineno", cumulative=False, file=sys.stdout, user_data_callback=None)`` class: Task taking temporary snapshots and displaying the top *count* memory allocations grouped by *group_by*. Call the ``start()`` method to start the task. ``display()`` method: Take a snapshot and display the top *count* biggest allocated memory blocks grouped by *group_by* using the ``display_top`` attribute. Return the snapshot, a ``Snapshot`` instance. ``start(delay: int)`` method: Start a task using the ``start_timer()`` function calling the ``display()`` method every *delay* seconds. ``stop()`` method: Stop the task started by the ``start()`` method using the ``stop_timer()`` function. ``count`` attribute: Maximum number of displayed memory blocks. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of each ``Trace`` instance, not only the most recent frame. The default value is ``False``. The option is ignored if the traceback limit is ``1``, see the ``get_traceback_limit()`` function. ``display_top`` attribute: Instance of ``DisplayTop``. ``file`` attribute: The top is written into *file*. ``group_by`` attribute: Determine how memory allocations are grouped: see ``Snapshot.top_by`` for the available values. ``user_data_callback`` attribute: Optional callback collecting user data (callable, default: ``None``). See ``Snapshot.create()``. Filter class ------------ ``Filter(include: bool, pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using ``get_tracemalloc_size()``. ``match_trace(trace)`` method: Return ``True`` if the ``Trace`` instance must be kept according to the filter, ``False`` otherwise. ``match(filename: str, lineno: int)`` method: Return ``True`` if the filename and line number must be kept according to the filter, ``False`` otherwise. ``match_filename(filename: str)`` method: Return ``True`` if the filename must be kept according to the filter, ``False`` otherwise. ``match_lineno(lineno: int)`` method: Return ``True`` if the line number must be kept according to the filter, ``False`` otherwise. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching filename ``pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching filename :attr`pattern` at line number ``lineno``. ``pattern`` attribute: The filename *pattern* can contain one or many ``*`` joker characters which match any substring, including an empty string. The ``.pyc`` and ``.pyo`` suffixes are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``lineno`` attribute: Line number (``int``). If is is ``None`` or lesser than ``1``, it matches any line number. ``traceback`` attribute: If *traceback* is ``True``, all frames of the ``traceback`` attribute of ``Trace`` instances are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute only has an effect on the ``match_trace()`` method and only if the traceback limit is greater than ``1``. See the ``get_traceback_limit()`` function. Frame class ----------- ``Frame`` class: Trace of a Python frame, used by ``Trace.traceback`` attribute. ``filename`` attribute: Python filename, ``None`` if unknown. ``lineno`` attribute: Python line number, ``None`` if unknown. GroupedStats class ------------------ ``GroupedStats(stats: dict, group_by: str, cumulative=False, timestamp=None, process_memory=None, tracemalloc_size=None)`` class: Top of allocated memory blocks grouped by on *group_by* as a dictionary. The ``Snapshot.top_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats=None)`` method: Compare to an older ``GroupedStats`` instance. Return a ``StatsDiff`` instance. ``cumulative`` attribute: If ``True``, cumulate size and count of memory blocks of all frames of ``Trace``, not only the most recent frame. ``group_by`` attribute: Determine how memory allocations were grouped. The type of ``stats`` keys depends on *group_by*: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== See the *group_by* parameter of the ``Snapshot.top_by()`` method. ``stats`` attribute: Dictionary ``{key: stats}`` where the *key* type depends on the ``group_by`` attribute and *stats* type is ``TraceStats``. ``process_memory`` attribute: Result of the ``get_process_memory()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``tracemalloc_size`` attribute: The memory usage in bytes of the ``tracemalloc`` module, result of the ``get_tracemalloc_size()`` function. Snapshot class -------------- ``Snapshot`` class: Snapshot of memory blocks allocated by Python. Use ``TakeSnapshot`` to take regulary snapshots. ``apply_filters(filters)`` method: Apply a list filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(\*, with_traces=False, with_stats=True, user_data_callback=None)`` classmethod: Take a snapshot of traces and/or statistics of allocated memory blocks. If *with_traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``traces`` attribute. This attribute contains more information than ``stats`` and uses more memory and more disk space. If *with_traces* is ``False``, ``traces`` is set to ``None``. If *with_stats* is ``True``, ``get_stats()`` is called and its result is stored in the ``Snapshot.stats`` attribute. If *with_stats* is ``False``, ``Snapshot.stats`` is set to ``None``. *with_traces* and *with_stats* cannot be ``False`` at the same time. *user_data_callback* is an optional callable object. Its result should be serializable by the ``pickle`` module, or ``Snapshot.write()`` would fail. If *user_data_callback* is set, it is called and the result is stored in the ``Snapshot.user_data`` attribute. Otherwise, ``Snapshot.user_data`` is set to ``None``. The ``tracemalloc`` module must be enabled to take a snapshot. See the ``enable()`` function. ``load(filename)`` classmethod: Load a snapshot from a file. ``top_by(group_by: str, cumulative: bool=False)`` method: Compute top statistics grouped by *group_by* as a ``GroupedStats`` instance: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of each ``Trace`` instance, not only the most recent frame. The *cumulative* parameter is ignored if *group_by* is ``'address'`` or if the traceback limit is ``1``. See the ``traceback_limit`` attribute. ``write(filename)`` method: Write the snapshot into a file. ``pid`` attribute: Identifier of the process which created the snapshot, result of ``os.getpid()``. ``process_memory`` attribute: Memory usage of the current process, result of the ``get_process_memory()`` function. It can be ``None``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function, if ``create()`` was called with *with_stats* equals to ``True``, ``None`` otherwise. ``tracemalloc_size`` attribute: The memory usage in bytes of the ``tracemalloc`` module, result of the ``get_tracemalloc_size()`` function. ``traceback_limit`` attribute: The maximum number of frames stored in the ``traceback`` attribute of a ``Trace``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, if ``create()`` was called with *with_traces* equals to ``True``, ``None`` otherwise. The ``traceback`` attribute of each ``Trace`` instance is limited to ``traceback_limit`` frames. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. ``user_data`` attribute: Result of *user_data_callback* called in ``Snapshot.create()`` (default: ``None``). StatsDiff class --------------- ``StatsDiff(differences, old_stats, new_stats)`` class: Differences between two ``GroupedStats`` instances. By default, the ``differences`` list is unsorted: call ``sort()`` to sort it. The ``GroupedStats.compare_to()`` method creates a ``StatsDiff`` instance. ``sort()`` method: Sort the ``differences`` list from the biggest allocation to the smallest. Sort by *size_diff*, *size*, *count_diff*, *count* and then by *key*. ``differences`` attribute: Differences between ``old_stats`` and ``new_stats`` as a list of ``(size_diff, size, count_diff, count, key)`` tuples. *size_diff*, *size*, *count_diff* and *count* are ``int``. The key type depends on the ``group_by`` attribute of ``new_stats``: ===================== ======================== ============== group_by description key type ===================== ======================== ============== ``'filename'`` filename ``str`` ``'filename_lineno'`` filename and line number ``(str, str)`` ``'address'`` memory block address ``int`` ===================== ======================== ============== See the ``group_by`` attribute of the ``GroupedStats`` class. ``old_stats`` attribute: Old ``GroupedStats`` instance, can be ``None``. ``new_stats`` attribute: New ``GroupedStats`` instance. Trace class ----------- ``Trace`` class: Debug information of a memory block allocated by Python. ``size`` attribute: Size in bytes of the memory block. ``traceback`` attribute: Traceback where the memory block was allocated as a list of ``Frame`` instances, most recent first. The list can be empty or incomplete if the ``tracemalloc`` module was unable to retrieve the full traceback. The traceback is limited to ``get_traceback_limit()`` frames. Use ``set_traceback_limit()`` to store more frames. TraceStats class ---------------- ``TraceStats`` class: Statistics on Python memory allocations. ``size`` attribute: Total size in bytes of allocated memory blocks. ``count`` attribute: Number of allocated memory blocks. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Similar projects: * `Meliae: Python Memory Usage Analyzer <https://pypi.python.org/pypi/meliae>`_ * `Guppy-PE: umbrella package combining Heapy and GSL <http://guppy-pe.sourceforge.net/>`_ * `PySizer <http://pysizer.8325.org/>`_: developed for Python 2.4 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_ * `pympler <http://code.google.com/p/pympler/>`_ * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger * `objgraph <http://mg.pov.lt/objgraph/>`_ * `caulk <https://github.com/smartfile/caulk/>`_ Copyright ========= This document has been placed into the public domain. Victor