Python-Dev October 2013

python-dev@python.org

112 participants
109 discussions

Updated PEP 454 (tracemalloc): no more metrics!
by Victor Stinner 31 Oct '13

31 Oct '13

Hi, I was at the restaurant with Charles-François and Antoine yesterday to discuss the PEP 454 (tracemalloc). They gave me a lot of advices to improve the PEP. Most remarks were request to remove code :-) I also improved surprising/strange APIs (like the infamous GroupedStats.compate_to(None)). HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the implementation, especially examples: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#examples Major changes: * GroupedStats.compare_to()/statistics() now returns a list of Statistic instances instead of a tuple with 5 items * StatsDiff class has been removed * Metrics have been removed * Remove Filter.match*() methods * Replace get_object_trace() function with get_object_traceback() * More complete list of prior work. There are 11 Python projects to debug memory leaks! I mentioned that PySizer implemented something similar to tracemalloc 8 years ago. I also rewrote the Rationale section * Rename some classes, attributes and functions Mercurial log of the PEP: http://hg.python.org/peps/log/f851d4a1622a/pep-0454.txt PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Classic generic tools like Valgrind can get the C traceback where a memory block was allocated. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python has an allocator for small object called "pymalloc" which keeps free blocks for efficiency. This is not well handled by these tools. There are debug tools dedicated to the Python language like ``Heapy`` ``Pympler`` and ``Meliae`` which lists all live objects using the garbage module (functions like ``gc.get_objects()``, ``gc.get_referrers()`` and ``gc.get_referents()``), compute their size (ex: using ``sys.getsizeof()``) and group objects by type. These tools provide a better estimation of the memory usage of an application. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Computed differences between two snapshots to detect memory leaks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_. The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds a overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attachs a traceback to the underlying layer, to memory blocks, and has no overhead when the module is disabled. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the ``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X tracemalloc=25`` command line option. The ``set_traceback_limit()`` function can be used at runtime to set the limit. By default, Python memory blocks allocated in the ``tracemalloc`` module are ignored using a filter. Use ``clear_filters()`` to trace also these memory allocations. Main Functions -------------- ``reset()`` function: Clear traces and statistics on Python memory allocations. See also ``disable()``. ``disable()`` function: Stop tracing Python memory allocations and clear traces and statistics. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. *size* is the total size in bytes of all memory blocks allocated on the line, or *count* is the number of memory blocks allocated on the line. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module used internally to trace memory allocations. Return an ``int``. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``disable()`` and ``enable()`` functions. Trace Functions --------------- When Python allocates a memory block, ``tracemalloc`` attachs a "trace" to it to store information on it: its size in bytes and the traceback where the allocation occured. The following functions give access to these traces. A trace is a ``(size: int, traceback)`` tuple. *size* is the size of the memory block in bytes. *traceback* is a tuple of frames sorted from the most recent to the oldest frame, limited to ``get_traceback_limit()`` frames. A frame is a ``(filename: str, lineno: int)`` tuple where *filename* and *lineno* can be ``None``. Example of trace: ``(32, (('x.py', 7), ('x.py', 11)))``. The memory block has a size of 32 bytes and was allocated at ``x.py:7``, line called from line ``x.py:11``. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. For example, items of ``dict`` and ``set`` containers are stored in a second memory block. See also ``get_object_traceback()`` and ``gc.get_referrers()`` functions. .. note:: The builtin function ``id()`` returns a different address for objects tracked by the garbage collector, because ``id()`` returns the address after the garbage collector header. ``get_object_traceback(obj)`` function: Get the traceback where the Python object *obj* was allocated. Return a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block allocated by Python. Return a tuple: ``(size: int, traceback)``, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_traceback()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace. By default, a trace of an allocated memory block only stores the most recent frame: the limit is ``1``. This limit is enough to get statistics using ``get_stats()``. Use the ``set_traceback_limit()`` function to change the limit. ``get_traces()`` function: Get traces of all memory blocks allocated by Python. Return a dictionary: ``{address (int): trace}``, *trace* is a ``(size: int, traceback)`` tuple, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_traceback()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. The ``PYTHONTRACEMALLOC`` environment variable and the ``-X`` ``tracemalloc=NFRAME`` command line option can be used to set a limit at startup. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``reset()`` function to ensure that all traces match the new filter. ``add_inclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_inclusive_filter(subprocess.__file__)`` only includes memory blocks allocated by the ``subprocess`` module. ``add_exclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_exclusive_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Clear the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations. Return a list of ``Filter`` instances. By default, there is one exclusive filter to ignore Python memory blocks allocated by the ``tracemalloc`` module. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, filename_pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``) of the filter. If *lineno* is is ``None`` or less than ``1``, the filter matches any line number. ``filename_pattern`` attribute: Filename pattern (``str``) of the filter. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, traceback_limit: int, stats: dict, key_type: str, cumulative: bool)`` class: Top of allocated memory blocks grouped by *key_type* as a dictionary. The ``Snapshot.group_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats, sort=True)`` method: Compare statistics to an older ``GroupedStats`` instance. Return a list of ``Statistic`` instances. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``statistics()`` method. ``statistics(sort=True)`` method: Get statistics as a list of ``Statistic`` instances. ``Statistic.size_diff`` and ``Statistic.count_diff`` attributes are set to zero. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``compare_to()`` method. ``cumulative`` attribute: If ``True``, size and count of memory blocks of all frames of the traceback of a trace were cumulated, not only the most recent frame. ``key_type`` attribute: Determine how memory allocations were grouped: see ``Snapshot.group_by()()`` for the available values. ``stats`` attribute: Dictionary ``{key: (size: int, count: int)}`` where the type of *key* depends on the ``key_type`` attribute. See the ``Snapshot.group_by()`` method. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traceback_limit: int, stats: dict=None, traces: dict=None)`` class: Snapshot of statistics and traces of memory blocks allocated by Python. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of statistics and traces of memory blocks allocated by Python. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot, see the the ``enable()`` function. ``dump(filename)`` method: Write the snapshot into a file. Use ``load()`` to reload the snapshot. ``load(filename)`` classmethod: Load a snapshot from a file. See also ``dump()``. ``group_by(key_type: str, cumulative: bool=False)`` method: Group statistics by *key_type* as a ``GroupedStats`` instance: ===================== =================================== ================================ key_type description type ===================== =================================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` memory block address with traceback ``(address: int, traceback)`` ===================== =================================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is set to ``False`` if *key_type* is ``'address'``, or if the traceback limit is less than ``2``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Statistic --------- ``Statistic(key, size, size_diff, count, count_diff)`` class: Statistic on memory allocations. ``GroupedStats.compare_to()`` and ``GroupedStats.statistics()`` return a list of ``Statistic`` instances. ``key`` attribute: Key identifying the statistic. The key type depends on ``GroupedStats.key_type``, see the ``Snapshot.group_by()`` method. ``count`` attribute: Number of memory blocks (``int``). ``count_diff`` attribute: Difference of number of memory blocks (``int``). ``size`` attribute: Total size of memory blocks in bytes (``int``). ``size_diff`` attribute: Difference of total size of memory blocks in bytes (``int``). Prior Work ========== * `Python Memory Validator <http://www.softwareverify.com/python/memory/index.html>`_ (2005-2013): commercial Python memory validator developed by Software Verification. It uses the Python Reflection API. * `PySizer <http://pysizer.8325.org/>`_: Google Summer of Code 2005 project by Nick Smallbone. * `Heapy <http://guppy-pe.sourceforge.net/>`_ (2006-2013): part of the Guppy-PE project written by Sverker Nilsson. * Draft PEP: `Support Tracking Low-Level Memory Usage in CPython <http://svn.python.org/projects/python/branches/bcannon-sandboxing/PEP.txt>`_ (Brett Canon, 2006) * Muppy: project developed in 2008 by Robert Schuppenies. * `asizeof <http://code.activestate.com/recipes/546530/>`_: a pure Python module to estimate the size of objects by Jean Brouwers (2008). * `Heapmonitor <http://www.scons.org/wiki/LudwigHaehne/HeapMonitor>`_: It provides facilities to size individual objects and can track all objects of certain classes. It was developed in 2008 by Ludwig Haehne. * `Pympler <http://code.google.com/p/pympler/>`_ (2008-2011): project based on asizeof, muppy and HeapMonitor * `objgraph <http://mg.pov.lt/objgraph/>`_ (2008-2012) * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013) * `Meliae <https://pypi.python.org/pypi/meliae>`_: Python Memory Usage Analyzer developed by John A Meinel since 2009 * `caulk <https://github.com/smartfile/caulk/>`_: written by Ben Timby in 2012 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_: written by Fabian Pedregosa (2011-2013) See also `Pympler Related Work <http://pythonhosted.org/Pympler/related.html>`_. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Copyright ========= This document has been placed in the public domain.

5 18

PEP 455: TransformDict
by Antoine Pitrou 30 Oct '13

30 Oct '13

Hello, Following the python-dev discussion, I've written a PEP to recap the proposal and the various arguments. It's inlined below, and it will probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. Regards Antoine. PEP: 455 Title: Adding a key-transforming dictionary to collections Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Sep-2013 Python-Version: 3.4 Post-History: Abstract ======== This PEP proposes a new data structure for the ``collections`` module, called "TransformDict" in this PEP. This structure is a mutable mapping which transforms the key using a given function when doing a lookup, but retains the original key when reading. Rationale ========= Numerous specialized versions of this pattern exist. The most common is a case-insensitive case-preserving dict, i.e. a dict-like container which matches keys in a case-insensitive fashion but retains the original casing. It is a very common need in network programming, as many protocols feature some arrays of "key / value" properties in their messages, where the keys are textual strings whose casing isn't relevant. Another common request is an identity dict, where keys are matched according to their respective id()s instead of normal matching. Both are instances of a more general pattern, where a given transformation function is applied to keys when looking them up: that function being ``str.lower`` in the former example and the built-in ``id`` function in the latter. (it can be said that the pattern *projects* keys from the user-visible set onto the internal lookup set, hence this PEP's title) Semantics ========= TransformDict is a ``MutableMapping`` implementation: it faithfully implements the well-known API of mutable mappings, as ``dict`` itself and other dict-like classes in the standard library. Therefore, this PEP won't rehash the semantics of most TransformDict methods. The transformation function needn't be bijective, it can be strictly surjective as in the case-insensitive example:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 5 >>> d['somekey'] 5 >>> d['SOMEKEY'] 5 TransformDict retains the first key used when creating an entry:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 1 >>> d['somekey'] = 2 >>> list(d.items()) [('SomeKey', 2)] The original keys needn't be hashable, as long as the transformation function returns a hashable one:: >>> d = TransformDict(id) >>> l = [None] >>> d[l] = 5 >>> l in d True Constructor ----------- As shown in the example aboves, creating a TransformDict requires passing the key transformation function as the first argument (much like creating a ``defaultdict`` requires passing the factory function as first argument). The constructor also takes other optional arguments which can be used to initialize the TransformDict with certain key-value pairs. Those optional arguments are the same as in the ``dict`` and ``defaultdict`` constructors:: >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) >>> sorted(d.items()) [('Bar', 2), ('Foo', 1)] Alternative proposals and questions =================================== Retaining the last original key ------------------------------- Most python-dev respondents found retaining the first user-supplied key more intuitive than retaining the last. Also, it matches the dict object's own behaviour when using different but equal keys:: >>> d = {} >>> d[1] = 'hello' >>> d[1.0] = 'world' >>> d {1: 'world'} Furthermore, explicitly retaining the last key in a first-key-retaining scheme is still possible using the following approach:: d.pop(key, None) d[key] = value while the converse (retaining the first key in a last-key-retaining scheme) doesn't look possible without rewriting part of the container's code. Using an encoder / decoder pair ------------------------------- Using a function pair isn't necessary, since the original key is retained by the container. Moreover, an encoder / decoder pair would require the transformation to be bijective, which prevents important use cases like case-insensitive matching. Providing a transformation function for values ---------------------------------------------- Dictionary values are not used for lookup, their semantics are totally irrelevant to the container's operation. Therefore, there is no point in having both an "original" and a "transformed" value: the transformed value wouldn't be used for anything. Providing a specialized container, not generic ---------------------------------------------- It was asked why we would provide the generic TransformDict construct rather than a specialized case-insensitive dict variant. The answer is that it's nearly as cheap (code-wise and performance-wise) to provide the generic construct, and it can fill more use cases. Implementation ============== A patch for the collections module is tracked on the bug tracker at http://bugs.python.org/issue18986. Existing work ============= Case-insensitive dicts are a popular request: * http://twistedmatrix.com/documents/current/api/twisted.python.util.Insensit… * https://mail.python.org/pipermail/python-list/2013-May/647243.html * https://mail.python.org/pipermail/python-list/2005-April/296208.html * https://mail.python.org/pipermail/python-list/2004-June/241748.html * http://bugs.python.org/msg197376 * http://stackoverflow.com/a/2082169 * http://stackoverflow.com/a/3296782 * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ * https://gist.github.com/babakness/3901174 * http://www.wikier.org/blog/key-insensitive-dictionary-in-python * http://en.sharejs.com/python/14534 * http://www.voidspace.org.uk/python/archive.shtml#caseless Identity dicts have been requested too: * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html * http://www.gossamer-threads.com/lists/python/python/209527 Python's own pickle module uses identity lookups for object memoization: http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

22 101

RELEASED: Python 2.6.9 final
by Barry Warsaw 29 Oct '13

29 Oct '13

Hello Pythoneers and Pythonistas, Five years ago this month, we released Python 2.6. It was an important version in Python's evolution, as it was developed in tandem with Python 3.0 and had the express intent of starting the transition toward Python 3. Amazingly, here we are five years later, and I am happy (and a little sad) to announce the Python 2.6.9 final release. Python 2.6.9 is a security-only source-only release, and with this, I officially retire the Python 2.6 branch. All maintenance of Python 2.6, including for security issues, has now ended. So too has my latest stint as Python Release Manager. Over the 19 years I have been involved with Python, I've been honored, challenged, stressed, and immeasurably rewarded by managing several Python releases. I wish I could thank each of you individually for all the support you've generously given me in this role. You deserve most of the credit for all these great releases; I'm just a monkey pushing buttons. :) Here then are some release notes about 2.6.9. For ongoing maintenance, please see Python 2.7. Future issues with Python 2.6 may still be tracked in the Python bug tracker, and you may wish to visit specific issues to see if unofficial patches are available for 2.6. http://bugs.python.org The source can be downloaded from: http://www.python.org/download/releases/2.6.9/ You can also see what's changed since Python 2.6.8: http://www.python.org/download/releases/2.6.9/NEWS.txt Users on OS X 10.9 (Mavericks) please note that issue 18458, which can crash the interactive interpreter, is *not* fixed in 2.6.9. If this issue affects you, please review the tracker for possible options: http://bugs.python.org/issue18458 Enjoy, -Barry (on behalf of the Python development community)

2 1

tracemalloc.get_object_traceback() and "unclosed" ResourceWarning
by Victor Stinner 29 Oct '13

29 Oct '13

Hi, Since Python 3.3, when a file or socket object is destroyed before being closed, a ResourceWarning is emitted. The warning is emitted where the object is destroyed: it's common to see "gc.collect()" location in the Python test suite for example. Tarek Ziadé asked on Twitter if there is a tool to track the origin of the object: "I wonder is there's a tool to trace back objects declarations that raise ResourceWarning in Py" https://twitter.com/tarek_ziade/status/394006754779877377 Nick Coghlan ‏replied that tracemalloc may help: "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ … - not accepted yet) will likely make it possible in 3.4, dunno about earlier versions." https://twitter.com/ncoghlan_dev/status/394010756242350080 I checked the code and it's not so simple. FileIO destructor emits the warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not possible to "hook" this function. I tried something with weakref, but I failed to set my hook. Does anyone see how to reuse tracemalloc (which is optional and disabled by default) in FileIO destructor? The file/object destructor may retrieve somehow the most recent frame of the object and call PyErr_WarnExplicitObject() instead of PyErr_WarnFormat() to pass the filename and lineno of the object. But how would Python decide to use tracemalloc or not? Add a new genric "get object traceback" callback which would be set by tracemalloc when the module is enabled? A frame is maybe not enough, it would be better to get the whole traceback! Do you know other features where it would be nice to know the origin of a Python object? Victor

2 2

Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
by Georg Brandl 29 Oct '13

29 Oct '13

Am 29.10.2013 01:46, schrieb victor.stinner: > http://hg.python.org/cpython/rev/e1d51c42e5a1 > changeset: 86716:e1d51c42e5a1 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Oct 29 01:28:23 2013 +0100 > summary: > Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on > memory allocation failure > > files: > Objects/unicodeobject.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -3766,6 +3766,7 @@ > return NULL; > _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1); > if (_PyUnicode_UTF8(unicode) == NULL) { > + PyErr_NoMemory(); > Py_DECREF(bytes); > return NULL; > } Shouldn't this (and related commits from #18408) have been committed to the 3.3 branch? Georg

3 3

Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
by Georg Brandl 29 Oct '13

29 Oct '13

Am 29.10.2013 01:19, schrieb victor.stinner: > http://hg.python.org/cpython/rev/4ef4578db38a > changeset: 86715:4ef4578db38a > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Oct 29 01:19:37 2013 +0100 > summary: > Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle > exceptions when merging fast locals into f_locals of a frame. > PyEval_GetLocals() now raises an exception and return NULL on failure. You'll have to either make this private or document it. Georg

3 3

Block module import during Python finalization?
by Victor Stinner 29 Oct '13

29 Oct '13

Hi, While working on a fix, I got an assertion error during Python finalization because Python tried to import the "io" module whereas the module was just unloaded. Python tried to import the io module to display a warning and display the Python line where the warning was emitted. See the following issue for detail: http://bugs.python.org/issue19421 I propose to block importing new modules (or import again unloaded modules) during Python finalization to avoid such tricky bugs. What do you think? Does anyone rely on the ability of importing Python modules at exit? :-) At least, the whole test suite pass with my import_finalization.patch which block import during Python finalization. Victor

4 3

Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting
by Tae Wong 28 Oct '13

28 Oct '13

Here's the codeaccess.txt file. https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt All committer names have (No): Python; check the sixth line. Merging the names of committers in the sixth line to the Misc/ACKS file might be good.

4 6

[RELEASED] Python 3.3.3 release candidate 1
by Georg Brandl 27 Oct '13

27 Oct '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm quite happy to announce the Python 3.3.3 release candidate 1. Python 3.3.3 includes several security fixes and over 150 bug fixes compared to the Python 3.3.2 release. This release fully supports OS X 10.9 Mavericks. In particular, this release fixes an issue that could cause previous versions of Python to crash when typing in interactive mode on OS X 10.9. Python 3.3.3 also contains a new batteries-included feature for OS X users of IDLE and other Tkinter-based programs. The python.org Mac OS X 64-bit/32-bit x86-64/i386 Installer for OS X 10.6+ now includes its own builtin version of Tcl/Tk 8.5. It is no longer necessary to install a third-party version of Tcl/Tk 8.5 to workaround the problematic system versions. See http://www.python.org/download/mac/tcltk/ for more information. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html and for the detailed changelog of 3.3.3, see http://docs.python.org/3.3/whatsnew/changelog.html To download Python 3.3.3 rc1 visit: http://www.python.org/download/releases/3.3.3/ This is a preview release, please report any bugs to http://bugs.python.org/ The final version is scheduled to be released in two weeks' time, on or about the 10th of November. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJte8kACgkQN9GcIYhpnLDx8wCgqtabbC8DaoW+Vy03AYGWyLhw sWcAoIK5jQeXDAxHayG+VWH/rRF1+qHC =yOed -----END PGP SIGNATURE-----

1 0

pathlib (PEP 428) status
by Charles-François Natali 27 Oct '13

27 Oct '13

Hi, What's the current status of pathlib? Is it targeted for 3.4? It would be a really nice addition, and AFAICT it has already been maturing a while on pypi, and discussed several times here. If I remember correctly, the only remaining issue was stat()'s result caching. cf

5 4

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev October 2013