Mailman 3 October 2013 - Python-Dev

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

More compact dictionaries with faster iteration
by Raymond Hettinger Jan. 3, 2015

Jan. 3, 2015

The current memory layout for dictionaries is unnecessarily inefficient. It has a sparse table of 24-byte entries containing the hash value, key pointer, and value pointer. Instead, the 24-byte entries should be stored in a dense table referenced by a sparse table of indices. For example, the dictionary: d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'} is currently stored as: entries = [['--', '--', '--'], [-8522787127447073495, 'barry', 'green'], ['--', '--', '--'], ['--', '--', '--'], ['--', '--', '--'], [-9092791511155847987, 'timmy', 'red'], ['--', '--', '--'], [-6480567542315338377, 'guido', 'blue']] Instead, the data should be organized as follows: indices = [None, 1, None, None, None, 0, None, 2] entries = [[-9092791511155847987, 'timmy', 'red'], [-8522787127447073495, 'barry', 'green'], [-6480567542315338377, 'guido', 'blue']] Only the data layout needs to change. The hash table algorithms would stay the same. All of the current optimizations would be kept, including key-sharing dicts and custom lookup functions for string-only dicts. There is no change to the hash functions, the table search order, or collision statistics. The memory savings are significant (from 30% to 95% compression depending on the how full the table is). Small dicts (size 0, 1, or 2) get the most benefit. For a sparse table of size t with n entries, the sizes are: curr_size = 24 * t new_size = 24 * n + sizeof(index) * t In the above timmy/barry/guido example, the current size is 192 bytes (eight 24-byte entries) and the new size is 80 bytes (three 24-byte entries plus eight 1-byte indices). That gives 58% compression. Note, the sizeof(index) can be as small as a single byte for small dicts, two bytes for bigger dicts and up to sizeof(Py_ssize_t) for huge dict. In addition to space savings, the new memory layout makes iteration faster. Currently, keys(), values, and items() loop over the sparse table, skipping-over free slots in the hash table. Now, keys/values/items can loop directly over the dense table, using fewer memory accesses. Another benefit is that resizing is faster and touches fewer pieces of memory. Currently, every hash/key/value entry is moved or copied during a resize. In the new layout, only the indices are updated. For the most part, the hash/key/value entries never move (except for an occasional swap to fill a hole left by a deletion). With the reduced memory footprint, we can also expect better cache utilization. For those wanting to experiment with the design, there is a pure Python proof-of-concept here: http://code.activestate.com/recipes/578375 YMMV: Keep in mind that the above size statics assume a build with 64-bit Py_ssize_t and 64-bit pointers. The space savings percentages are a bit different on other builds. Also, note that in many applications, the size of the data dominates the size of the container (i.e. the weight of a bucket of water is mostly the water, not the bucket). Raymond

27 70

Semi-official read-only Github mirror of the CPython Mercurial repository
by Eli Bendersky Oct. 15, 2014

Oct. 15, 2014

Hi all, https://github.com/python/cpython is now live as a semi-official, *read only* Github mirror of the CPython Mercurial repository. Let me know if you have any problems/concerns. I still haven't decided how often to update it (considering either just N times a day, or maybe use a Hg hook for batching). Suggestions are welcome. The methodology I used to create it is via hg-fast-export. I also tried to pack and gc the git repo as much as possible before the initial Github push - it went down from almost ~2GB to ~200MB (so this is the size of a fresh clone right now). Eli P.S. thanks Jesse for the keys to https://github.com/python

4 8

Reviving restricted mode?
by Guido van Rossum Aug. 14, 2014

Aug. 14, 2014

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

19 35

CLA link from bugs.python.org
by Tim Delaney Feb. 16, 2014

Feb. 16, 2014

It appears there's no obvious link from bugs.python.org to the contributor agreement - you need to go via the unintuitive link Foundation -> Contribution Forms (and from what I've read, you're prompted when you add a patch to the tracker). I'd suggest that if the "Contributor Form Received" field is "No" in user details, there be a link to http://www.python.org/psf/contrib/. Tim Delaney

3 4

cffi in stdlib
by Maciej Fijalkowski Dec. 19, 2013

Dec. 19, 2013

Hello. I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. This is a project Armin Rigo has been working for a while, with some input from other developers. It seems that the main reason why people would prefer ctypes over cffi these days is "because it's included in stdlib", which is not generally the reason I would like to hear. Our calls to not use C extensions and to use an FFI instead has seen very limited success with ctypes and quite a lot more since cffi got released. The API is fairly stable right now with minor changes going in and it'll definitely stablize until Python 3.4 release. Notable projects using it: * pypycore - gevent main loop ported to cffi * pgsql2cffi * sdl-cffi bindings * tls-cffi bindings * lxml-cffi port * pyzmq * cairo-cffi * a bunch of others So relatively a lot given that the project is not even a year old (it got 0.1 release in June). As per documentation, the advantages over ctypes: * The goal is to call C code from Python. You should be able to do so without learning a 3rd language: every alternative requires you to learn their own language (Cython, SWIG) or API (ctypes). So we tried to assume that you know Python and C and minimize the extra bits of API that you need to learn. * Keep all the Python-related logic in Python so that you don’t need to write much C code (unlike CPython native C extensions). * Work either at the level of the ABI (Application Binary Interface) or the API (Application Programming Interface). Usually, C libraries have a specified C API but often not an ABI (e.g. they may document a “struct” as having at least these fields, but maybe more). (ctypes works at the ABI level, whereas Cython and native C extensions work at the API level.) * We try to be complete. For now some C99 constructs are not supported, but all C89 should be, including macros (and including macro “abuses”, which you can manually wrap in saner-looking C functions). * We attempt to support both PyPy and CPython, with a reasonable path for other Python implementations like IronPython and Jython. * Note that this project is not about embedding executable C code in Python, unlike Weave. This is about calling existing C libraries from Python. so among other things, making a cffi extension gives you the same level of security as writing C (and unlike ctypes) and brings quite a bit more flexibility (API vs ABI issue) that let's you wrap arbitrary libraries, even those full of macros. Cheers, fijal .. [1]: http://cffi.readthedocs.org/en/release-0.5/

22 75

bugs.python.org not reachable in IPv6?
by Victor Stinner Nov. 4, 2013

Nov. 4, 2013

Hi, The DNS server of python.org announce the IP address 2a01:4f8:131:2480::3: $ host -t AAAA bugs.python.org bugs.python.org has IPv6 address 2a01:4f8:131:2480::3 The problem is that I cannot connect to this IP address: $ ping6 -c 4 2a01:4f8:131:2480::3 PING 2a01:4f8:131:2480::3(2a01:4f8:131:2480::3) 56 data bytes --- 2a01:4f8:131:2480::3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms Do you have a the same issue, or is it just me? Victor

5 11

PEP 451 update
by Eric Snow Nov. 2, 2013

Nov. 2, 2013

I've had some offline discussion with Brett and Nick about PEP 451 which has led to some meaningful clarifications in the PEP. In the interest of pulling further discussions back onto this (archived/public) list, here's an update of what we'd discussed and where things are at. :) * path entry finders indicate that they found part of a possible namespace package by returning a spec with no loader set (but with submodule_search_locations set). Brett wanted some clarification on this. * The name/path signature and attributes of file-based finders in importlib will no longer be changing. Brett had some suggestions on the proposed change and it became clear that the the change was actually pointless. * I've asserted that there shouldn't be much difficulty in adjusting pkgutil and other modules to work with ModuleSpec. * Brett asked for clarification on whether the "load()" example from the PEP would be realized implicitly by the import machinery or explicitly as a method on ModuleSpec. This has bearing on the ability of finders to return instances of ModuleSpec subclasses or even ModuleSpec-like objects (a la duck typing). The answer is the it will not be a method on ModuleSpec, so it is effectively just part of the general import system implementation. Finders may return any object that provides the attributes of ModuleSpec. I will be updating the PEP to make these points clear. * Nick suggested writing a draft patch for the language reference changes (the import page). Such a patch will be a pretty good indicator of the impact of PEP 451 on the import system and should highlight any design flaws in the API. This is on my to-do list (hopefully by tomorrow). * Nick also suggested moving all ModuleSpec methods to a separate class that will simply make use of a separate, existing ModuleSpec instance. This will help address several issues, particularly by relaxing the constraints on what finders can return, but also by avoiding the unnecessary exposure of the methods via every module.__spec__. I plan on going with this, but currently am trying out the change to see if there are any problems I've missed. Once I feel good about it I'll update the PEP. That about sums up our discussions. I have a couple of outstanding updates to the PEP to make when I get a chance, as well as putting up a language reference patch for review. -eric

5 45

Compiler security
by MRAB Oct. 31, 2013

Oct. 31, 2013

Has anybody here heard about this, and, if so, is it anything we should be thinking about: How your compiler may be compromising application security http://www.itworld.com/security/380406/how-your-compiler-may-be-compromisin…

6 6

PEP 454 (tracemalloc) disable ==> clear?
by Jim Jewett Oct. 31, 2013

Oct. 31, 2013

reset() function: Clear traces of memory blocks allocated by Python. Does this do anything besides clear? If not, why not just re-use the 'clear' name from dicts? disable() function: Stop tracing Python memory allocations and clear traces of memory blocks allocated by Python. I would disable to stop tracing, but I would not expect it to clear out the traces it had already captured. If it has to do that, please put in some sample code showing how to save the current traces before disabling. -jJ

7 12

Updated PEP 454 (tracemalloc): no more metrics!
by Victor Stinner Oct. 30, 2013

Oct. 30, 2013

Hi, I was at the restaurant with Charles-François and Antoine yesterday to discuss the PEP 454 (tracemalloc). They gave me a lot of advices to improve the PEP. Most remarks were request to remove code :-) I also improved surprising/strange APIs (like the infamous GroupedStats.compate_to(None)). HTML version: http://www.python.org/dev/peps/pep-0454/ See also the documentation of the implementation, especially examples: http://www.haypocalc.com/tmp/tracemalloc/library/tracemalloc.html#examples Major changes: * GroupedStats.compare_to()/statistics() now returns a list of Statistic instances instead of a tuple with 5 items * StatsDiff class has been removed * Metrics have been removed * Remove Filter.match*() methods * Replace get_object_trace() function with get_object_traceback() * More complete list of prior work. There are 11 Python projects to debug memory leaks! I mentioned that PySizer implemented something similar to tracemalloc 8 years ago. I also rewrote the Rationale section * Rename some classes, attributes and functions Mercurial log of the PEP: http://hg.python.org/peps/log/f851d4a1622a/pep-0454.txt PEP: 454 Title: Add a new tracemalloc module to trace Python memory allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-September-2013 Python-Version: 3.4 Abstract ======== This PEP proposes to add a new ``tracemalloc`` module to trace memory blocks allocated by Python. Rationale ========= Classic generic tools like Valgrind can get the C traceback where a memory block was allocated. Using such tools to analyze Python memory allocations does not help because most memory blocks are allocated in the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python has an allocator for small object called "pymalloc" which keeps free blocks for efficiency. This is not well handled by these tools. There are debug tools dedicated to the Python language like ``Heapy`` ``Pympler`` and ``Meliae`` which lists all live objects using the garbage module (functions like ``gc.get_objects()``, ``gc.get_referrers()`` and ``gc.get_referents()``), compute their size (ex: using ``sys.getsizeof()``) and group objects by type. These tools provide a better estimation of the memory usage of an application. They are useful when most memory leaks are instances of the same type and this type is only instantiated in a few functions. Problems arise when the object type is very common like ``str`` or ``tuple``, and it is hard to identify where these objects are instantiated. Finding reference cycles is also a difficult problem. There are different tools to draw a diagram of all references. These tools cannot be used on large applications with thousands of objects because the diagram is too huge to be analyzed manually. Proposal ======== Using the customized allocation API from PEP 445, it becomes easy to set up a hook on Python memory allocators. A hook can inspect Python internals to retrieve Python tracebacks. The idea of getting the current traceback comes from the faulthandler module. The faulthandler dumps the traceback of all Python threads on a crash, here is the idea is to get the traceback of the current Python thread when a memory block is allocated by Python. This PEP proposes to add a new ``tracemalloc`` module, as a debug tool to trace memory blocks allocated by Python. The module provides the following information: * Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks * Computed differences between two snapshots to detect memory leaks * Traceback where a memory block was allocated The API of the tracemalloc module is similar to the API of the faulthandler module: ``enable()``, ``disable()`` and ``is_enabled()`` functions, an environment variable (``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_. The idea of tracing memory allocations is not new. It was first implemented in the PySizer project in 2005. PySizer was implemented differently: the traceback was stored in frame objects and some Python types were linked the trace with the name of object type. PySizer patch on CPython adds a overhead on performances and memory footprint, even if the PySizer was not used. tracemalloc attachs a traceback to the underlying layer, to memory blocks, and has no overhead when the module is disabled. The tracemalloc module has been written for CPython. Other implementations of Python may not be able to provide it. API === To trace most memory blocks allocated by Python, the module should be enabled as early as possible by setting the ``PYTHONTRACEMALLOC`` environment variable to ``1``, or by using ``-X tracemalloc`` command line option. The ``tracemalloc.enable()`` function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the ``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X tracemalloc=25`` command line option. The ``set_traceback_limit()`` function can be used at runtime to set the limit. By default, Python memory blocks allocated in the ``tracemalloc`` module are ignored using a filter. Use ``clear_filters()`` to trace also these memory allocations. Main Functions -------------- ``reset()`` function: Clear traces and statistics on Python memory allocations. See also ``disable()``. ``disable()`` function: Stop tracing Python memory allocations and clear traces and statistics. See also ``enable()`` and ``is_enabled()`` functions. ``enable()`` function: Start tracing Python memory allocations. See also ``disable()`` and ``is_enabled()`` functions. ``get_stats()`` function: Get statistics on traced Python memory blocks as a dictionary ``{filename (str): {line_number (int): stats}}`` where *stats* in a ``(size: int, count: int)`` tuple, *filename* and *line_number* can be ``None``. *size* is the total size in bytes of all memory blocks allocated on the line, or *count* is the number of memory blocks allocated on the line. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also the ``get_traces()`` function. ``get_traced_memory()`` function: Get the current size and maximum size of memory blocks traced by the ``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``. ``get_tracemalloc_memory()`` function: Get the memory usage in bytes of the ``tracemalloc`` module used internally to trace memory allocations. Return an ``int``. ``is_enabled()`` function: ``True`` if the ``tracemalloc`` module is tracing Python memory allocations, ``False`` otherwise. See also ``disable()`` and ``enable()`` functions. Trace Functions --------------- When Python allocates a memory block, ``tracemalloc`` attachs a "trace" to it to store information on it: its size in bytes and the traceback where the allocation occured. The following functions give access to these traces. A trace is a ``(size: int, traceback)`` tuple. *size* is the size of the memory block in bytes. *traceback* is a tuple of frames sorted from the most recent to the oldest frame, limited to ``get_traceback_limit()`` frames. A frame is a ``(filename: str, lineno: int)`` tuple where *filename* and *lineno* can be ``None``. Example of trace: ``(32, (('x.py', 7), ('x.py', 11)))``. The memory block has a size of 32 bytes and was allocated at ``x.py:7``, line called from line ``x.py:11``. ``get_object_address(obj)`` function: Get the address of the main memory block of the specified Python object. A Python object can be composed by multiple memory blocks, the function only returns the address of the main memory block. For example, items of ``dict`` and ``set`` containers are stored in a second memory block. See also ``get_object_traceback()`` and ``gc.get_referrers()`` functions. .. note:: The builtin function ``id()`` returns a different address for objects tracked by the garbage collector, because ``id()`` returns the address after the garbage collector header. ``get_object_traceback(obj)`` function: Get the traceback where the Python object *obj* was allocated. Return a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the object. See also ``get_object_address()``, ``gc.get_referrers()`` and ``sys.getsizeof()`` functions. ``get_trace(address)`` function: Get the trace of a memory block allocated by Python. Return a tuple: ``(size: int, traceback)``, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. Return ``None`` if the ``tracemalloc`` module did not trace the allocation of the memory block. See also ``get_object_traceback()``, ``get_stats()`` and ``get_traces()`` functions. ``get_traceback_limit()`` function: Get the maximum number of frames stored in the traceback of a trace. By default, a trace of an allocated memory block only stores the most recent frame: the limit is ``1``. This limit is enough to get statistics using ``get_stats()``. Use the ``set_traceback_limit()`` function to change the limit. ``get_traces()`` function: Get traces of all memory blocks allocated by Python. Return a dictionary: ``{address (int): trace}``, *trace* is a ``(size: int, traceback)`` tuple, *traceback* is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be None. Return an empty dictionary if the ``tracemalloc`` module is disabled. See also ``get_object_traceback()``, ``get_stats()`` and ``get_trace()`` functions. ``set_traceback_limit(nframe: int)`` function: Set the maximum number of frames stored in the traceback of a trace. Storing the traceback of each memory allocation has an important overhead on the memory usage. Use the ``get_tracemalloc_memory()`` function to measure the overhead and the ``add_filter()`` function to select which memory allocations are traced. Use the ``get_traceback_limit()`` function to get the current limit. The ``PYTHONTRACEMALLOC`` environment variable and the ``-X`` ``tracemalloc=NFRAME`` command line option can be used to set a limit at startup. Filter Functions ---------------- ``add_filter(filter)`` function: Add a new filter on Python memory allocations, *filter* is a ``Filter`` instance. All inclusive filters are applied at once, a memory allocation is only ignored if no inclusive filters match its trace. A memory allocation is ignored if at least one exclusive filter matchs its trace. The new filter is not applied on already collected traces. Use the ``reset()`` function to ensure that all traces match the new filter. ``add_inclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an inclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``True``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_inclusive_filter(subprocess.__file__)`` only includes memory blocks allocated by the ``subprocess`` module. ``add_exclusive_filter(filename_pattern: str, lineno: int=None, traceback: bool=False)`` function: Add an exclusive filter: helper for the ``add_filter()`` function creating a ``Filter`` instance with the ``Filter.include`` attribute set to ``False``. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. Example: ``tracemalloc.add_exclusive_filter(tracemalloc.__file__)`` ignores memory blocks allocated by the ``tracemalloc`` module. ``clear_filters()`` function: Clear the filter list. See also the ``get_filters()`` function. ``get_filters()`` function: Get the filters on Python memory allocations. Return a list of ``Filter`` instances. By default, there is one exclusive filter to ignore Python memory blocks allocated by the ``tracemalloc`` module. See also the ``clear_filters()`` function. Filter ------ ``Filter(include: bool, filename_pattern: str, lineno: int=None, traceback: bool=False)`` class: Filter to select which memory allocations are traced. Filters can be used to reduce the memory usage of the ``tracemalloc`` module, which can be read using the ``get_tracemalloc_memory()`` function. The ``*`` joker character can be used in *filename_pattern* to match any substring, including empty string. The ``.pyc`` and ``.pyo`` file extensions are replaced with ``.py``. On Windows, the comparison is case insensitive and the alternative separator ``/`` is replaced with the standard separator ``\``. ``include`` attribute: If *include* is ``True``, only trace memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. If *include* is ``False``, ignore memory blocks allocated in a file with a name matching ``filename_pattern`` at line number ``lineno``. ``lineno`` attribute: Line number (``int``) of the filter. If *lineno* is is ``None`` or less than ``1``, the filter matches any line number. ``filename_pattern`` attribute: Filename pattern (``str``) of the filter. ``traceback`` attribute: If *traceback* is ``True``, all frames of the traceback are checked. If *traceback* is ``False``, only the most recent frame is checked. This attribute is ignored if the traceback limit is less than ``2``. See the ``get_traceback_limit()`` function. GroupedStats ------------ ``GroupedStats(timestamp: datetime.datetime, traceback_limit: int, stats: dict, key_type: str, cumulative: bool)`` class: Top of allocated memory blocks grouped by *key_type* as a dictionary. The ``Snapshot.group_by()`` method creates a ``GroupedStats`` instance. ``compare_to(old_stats: GroupedStats, sort=True)`` method: Compare statistics to an older ``GroupedStats`` instance. Return a list of ``Statistic`` instances. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``statistics()`` method. ``statistics(sort=True)`` method: Get statistics as a list of ``Statistic`` instances. ``Statistic.size_diff`` and ``Statistic.count_diff`` attributes are set to zero. The result is sorted in the biggest to the smallest by ``abs(size_diff)``, *size*, ``abs(count_diff)``, *count* and then by *key*. Set the *sort* parameter to ``False`` to get the list unsorted. ``None`` values in keys are replaced with an empty string for filenames or zero for line numbers, because ``str`` and ``int`` cannot be compared to ``None``. See also the ``compare_to()`` method. ``cumulative`` attribute: If ``True``, size and count of memory blocks of all frames of the traceback of a trace were cumulated, not only the most recent frame. ``key_type`` attribute: Determine how memory allocations were grouped: see ``Snapshot.group_by()()`` for the available values. ``stats`` attribute: Dictionary ``{key: (size: int, count: int)}`` where the type of *key* depends on the ``key_type`` attribute. See the ``Snapshot.group_by()`` method. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Snapshot -------- ``Snapshot(timestamp: datetime.datetime, traceback_limit: int, stats: dict=None, traces: dict=None)`` class: Snapshot of statistics and traces of memory blocks allocated by Python. ``apply_filters(filters)`` method: Apply filters on the ``traces`` and ``stats`` dictionaries, *filters* is a list of ``Filter`` instances. ``create(traces=False)`` classmethod: Take a snapshot of statistics and traces of memory blocks allocated by Python. If *traces* is ``True``, ``get_traces()`` is called and its result is stored in the ``Snapshot.traces`` attribute. This attribute contains more information than ``Snapshot.stats`` and uses more memory and more disk space. If *traces* is ``False``, ``Snapshot.traces`` is set to ``None``. Tracebacks of traces are limited to ``traceback_limit`` frames. Call ``set_traceback_limit()`` before calling ``Snapshot.create()`` to store more frames. The ``tracemalloc`` module must be enabled to take a snapshot, see the the ``enable()`` function. ``dump(filename)`` method: Write the snapshot into a file. Use ``load()`` to reload the snapshot. ``load(filename)`` classmethod: Load a snapshot from a file. See also ``dump()``. ``group_by(key_type: str, cumulative: bool=False)`` method: Group statistics by *key_type* as a ``GroupedStats`` instance: ===================== =================================== ================================ key_type description type ===================== =================================== ================================ ``'filename'`` filename ``str`` ``'line'`` filename and line number ``(filename: str, lineno: int)`` ``'address'`` memory block address ``int`` ``'traceback'`` memory block address with traceback ``(address: int, traceback)`` ===================== =================================== ================================ The ``traceback`` type is a tuple of ``(filename: str, lineno: int)`` tuples, *filename* and *lineno* can be ``None``. If *cumulative* is ``True``, cumulate size and count of memory blocks of all frames of the traceback of a trace, not only the most recent frame. The *cumulative* parameter is set to ``False`` if *key_type* is ``'address'``, or if the traceback limit is less than ``2``. ``stats`` attribute: Statistics on traced Python memory, result of the ``get_stats()`` function. ``traceback_limit`` attribute: Maximum number of frames stored in the traceback of ``traces``, result of the ``get_traceback_limit()`` function. ``traces`` attribute: Traces of Python memory allocations, result of the ``get_traces()`` function, can be ``None``. ``timestamp`` attribute: Creation date and time of the snapshot, ``datetime.datetime`` instance. Statistic --------- ``Statistic(key, size, size_diff, count, count_diff)`` class: Statistic on memory allocations. ``GroupedStats.compare_to()`` and ``GroupedStats.statistics()`` return a list of ``Statistic`` instances. ``key`` attribute: Key identifying the statistic. The key type depends on ``GroupedStats.key_type``, see the ``Snapshot.group_by()`` method. ``count`` attribute: Number of memory blocks (``int``). ``count_diff`` attribute: Difference of number of memory blocks (``int``). ``size`` attribute: Total size of memory blocks in bytes (``int``). ``size_diff`` attribute: Difference of total size of memory blocks in bytes (``int``). Prior Work ========== * `Python Memory Validator <http://www.softwareverify.com/python/memory/index.html>`_ (2005-2013): commercial Python memory validator developed by Software Verification. It uses the Python Reflection API. * `PySizer <http://pysizer.8325.org/>`_: Google Summer of Code 2005 project by Nick Smallbone. * `Heapy <http://guppy-pe.sourceforge.net/>`_ (2006-2013): part of the Guppy-PE project written by Sverker Nilsson. * Draft PEP: `Support Tracking Low-Level Memory Usage in CPython <http://svn.python.org/projects/python/branches/bcannon-sandboxing/PEP.txt>`_ (Brett Canon, 2006) * Muppy: project developed in 2008 by Robert Schuppenies. * `asizeof <http://code.activestate.com/recipes/546530/>`_: a pure Python module to estimate the size of objects by Jean Brouwers (2008). * `Heapmonitor <http://www.scons.org/wiki/LudwigHaehne/HeapMonitor>`_: It provides facilities to size individual objects and can track all objects of certain classes. It was developed in 2008 by Ludwig Haehne. * `Pympler <http://code.google.com/p/pympler/>`_ (2008-2011): project based on asizeof, muppy and HeapMonitor * `objgraph <http://mg.pov.lt/objgraph/>`_ (2008-2012) * `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013) * `Meliae <https://pypi.python.org/pypi/meliae>`_: Python Memory Usage Analyzer developed by John A Meinel since 2009 * `caulk <https://github.com/smartfile/caulk/>`_: written by Ben Timby in 2012 * `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_: written by Fabian Pedregosa (2011-2013) See also `Pympler Related Work <http://pythonhosted.org/Pympler/related.html>`_. Links ===== tracemalloc: * `#18874: Add a new tracemalloc module to trace Python memory allocations <http://bugs.python.org/issue18874>`_ * `pytracemalloc on PyPI <https://pypi.python.org/pypi/pytracemalloc>`_ Copyright ========= This document has been placed in the public domain.

5 18

PEP 455: TransformDict
by Antoine Pitrou Oct. 30, 2013

Oct. 30, 2013

Hello, Following the python-dev discussion, I've written a PEP to recap the proposal and the various arguments. It's inlined below, and it will probably appear soon at http://www.python.org/dev/peps/pep-0455/, too. Regards Antoine. PEP: 455 Title: Adding a key-transforming dictionary to collections Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Sep-2013 Python-Version: 3.4 Post-History: Abstract ======== This PEP proposes a new data structure for the ``collections`` module, called "TransformDict" in this PEP. This structure is a mutable mapping which transforms the key using a given function when doing a lookup, but retains the original key when reading. Rationale ========= Numerous specialized versions of this pattern exist. The most common is a case-insensitive case-preserving dict, i.e. a dict-like container which matches keys in a case-insensitive fashion but retains the original casing. It is a very common need in network programming, as many protocols feature some arrays of "key / value" properties in their messages, where the keys are textual strings whose casing isn't relevant. Another common request is an identity dict, where keys are matched according to their respective id()s instead of normal matching. Both are instances of a more general pattern, where a given transformation function is applied to keys when looking them up: that function being ``str.lower`` in the former example and the built-in ``id`` function in the latter. (it can be said that the pattern *projects* keys from the user-visible set onto the internal lookup set, hence this PEP's title) Semantics ========= TransformDict is a ``MutableMapping`` implementation: it faithfully implements the well-known API of mutable mappings, as ``dict`` itself and other dict-like classes in the standard library. Therefore, this PEP won't rehash the semantics of most TransformDict methods. The transformation function needn't be bijective, it can be strictly surjective as in the case-insensitive example:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 5 >>> d['somekey'] 5 >>> d['SOMEKEY'] 5 TransformDict retains the first key used when creating an entry:: >>> d = TransformDict(str.lower) >>> d['SomeKey'] = 1 >>> d['somekey'] = 2 >>> list(d.items()) [('SomeKey', 2)] The original keys needn't be hashable, as long as the transformation function returns a hashable one:: >>> d = TransformDict(id) >>> l = [None] >>> d[l] = 5 >>> l in d True Constructor ----------- As shown in the example aboves, creating a TransformDict requires passing the key transformation function as the first argument (much like creating a ``defaultdict`` requires passing the factory function as first argument). The constructor also takes other optional arguments which can be used to initialize the TransformDict with certain key-value pairs. Those optional arguments are the same as in the ``dict`` and ``defaultdict`` constructors:: >>> d = TransformDict(str.lower, [('Foo': 1)], Bar=2) >>> sorted(d.items()) [('Bar', 2), ('Foo', 1)] Alternative proposals and questions =================================== Retaining the last original key ------------------------------- Most python-dev respondents found retaining the first user-supplied key more intuitive than retaining the last. Also, it matches the dict object's own behaviour when using different but equal keys:: >>> d = {} >>> d[1] = 'hello' >>> d[1.0] = 'world' >>> d {1: 'world'} Furthermore, explicitly retaining the last key in a first-key-retaining scheme is still possible using the following approach:: d.pop(key, None) d[key] = value while the converse (retaining the first key in a last-key-retaining scheme) doesn't look possible without rewriting part of the container's code. Using an encoder / decoder pair ------------------------------- Using a function pair isn't necessary, since the original key is retained by the container. Moreover, an encoder / decoder pair would require the transformation to be bijective, which prevents important use cases like case-insensitive matching. Providing a transformation function for values ---------------------------------------------- Dictionary values are not used for lookup, their semantics are totally irrelevant to the container's operation. Therefore, there is no point in having both an "original" and a "transformed" value: the transformed value wouldn't be used for anything. Providing a specialized container, not generic ---------------------------------------------- It was asked why we would provide the generic TransformDict construct rather than a specialized case-insensitive dict variant. The answer is that it's nearly as cheap (code-wise and performance-wise) to provide the generic construct, and it can fill more use cases. Implementation ============== A patch for the collections module is tracked on the bug tracker at http://bugs.python.org/issue18986. Existing work ============= Case-insensitive dicts are a popular request: * http://twistedmatrix.com/documents/current/api/twisted.python.util.Insensit… * https://mail.python.org/pipermail/python-list/2013-May/647243.html * https://mail.python.org/pipermail/python-list/2005-April/296208.html * https://mail.python.org/pipermail/python-list/2004-June/241748.html * http://bugs.python.org/msg197376 * http://stackoverflow.com/a/2082169 * http://stackoverflow.com/a/3296782 * http://code.activestate.com/recipes/66315-case-insensitive-dictionary/ * https://gist.github.com/babakness/3901174 * http://www.wikier.org/blog/key-insensitive-dictionary-in-python * http://en.sharejs.com/python/14534 * http://www.voidspace.org.uk/python/archive.shtml#caseless Identity dicts have been requested too: * https://mail.python.org/pipermail/python-ideas/2010-May/007235.html * http://www.gossamer-threads.com/lists/python/python/209527 Python's own pickle module uses identity lookups for object memoization: http://hg.python.org/cpython/file/0e70bf1f32a3/Lib/pickle.py#l234 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

22 101

RELEASED: Python 2.6.9 final
by Barry Warsaw Oct. 29, 2013

Oct. 29, 2013

Hello Pythoneers and Pythonistas, Five years ago this month, we released Python 2.6. It was an important version in Python's evolution, as it was developed in tandem with Python 3.0 and had the express intent of starting the transition toward Python 3. Amazingly, here we are five years later, and I am happy (and a little sad) to announce the Python 2.6.9 final release. Python 2.6.9 is a security-only source-only release, and with this, I officially retire the Python 2.6 branch. All maintenance of Python 2.6, including for security issues, has now ended. So too has my latest stint as Python Release Manager. Over the 19 years I have been involved with Python, I've been honored, challenged, stressed, and immeasurably rewarded by managing several Python releases. I wish I could thank each of you individually for all the support you've generously given me in this role. You deserve most of the credit for all these great releases; I'm just a monkey pushing buttons. :) Here then are some release notes about 2.6.9. For ongoing maintenance, please see Python 2.7. Future issues with Python 2.6 may still be tracked in the Python bug tracker, and you may wish to visit specific issues to see if unofficial patches are available for 2.6. http://bugs.python.org The source can be downloaded from: http://www.python.org/download/releases/2.6.9/ You can also see what's changed since Python 2.6.8: http://www.python.org/download/releases/2.6.9/NEWS.txt Users on OS X 10.9 (Mavericks) please note that issue 18458, which can crash the interactive interpreter, is *not* fixed in 2.6.9. If this issue affects you, please review the tracker for possible options: http://bugs.python.org/issue18458 Enjoy, -Barry (on behalf of the Python development community)

2 1

tracemalloc.get_object_traceback() and "unclosed" ResourceWarning
by Victor Stinner Oct. 29, 2013

Oct. 29, 2013

Hi, Since Python 3.3, when a file or socket object is destroyed before being closed, a ResourceWarning is emitted. The warning is emitted where the object is destroyed: it's common to see "gc.collect()" location in the Python test suite for example. Tarek Ziadé asked on Twitter if there is a tool to track the origin of the object: "I wonder is there's a tool to trace back objects declarations that raise ResourceWarning in Py" https://twitter.com/tarek_ziade/status/394006754779877377 Nick Coghlan ‏replied that tracemalloc may help: "@tarek_ziade tracemalloc (http://www.python.org/dev/peps/pep-0454/ … - not accepted yet) will likely make it possible in 3.4, dunno about earlier versions." https://twitter.com/ncoghlan_dev/status/394010756242350080 I checked the code and it's not so simple. FileIO destructor emits the warning (fileio_dealloc_warn() in Modules/_io/fileio.c), but it's not possible to "hook" this function. I tried something with weakref, but I failed to set my hook. Does anyone see how to reuse tracemalloc (which is optional and disabled by default) in FileIO destructor? The file/object destructor may retrieve somehow the most recent frame of the object and call PyErr_WarnExplicitObject() instead of PyErr_WarnFormat() to pass the filename and lineno of the object. But how would Python decide to use tracemalloc or not? Add a new genric "get object traceback" callback which would be set by tracemalloc when the module is enabled? A frame is maybe not enough, it would be better to get the whole traceback! Do you know other features where it would be nice to know the origin of a Python object? Victor

2 2

Re: [Python-Dev] cpython: Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
by Georg Brandl Oct. 29, 2013

Oct. 29, 2013

Am 29.10.2013 01:46, schrieb victor.stinner: > http://hg.python.org/cpython/rev/e1d51c42e5a1 > changeset: 86716:e1d51c42e5a1 > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Oct 29 01:28:23 2013 +0100 > summary: > Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on > memory allocation failure > > files: > Objects/unicodeobject.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -3766,6 +3766,7 @@ > return NULL; > _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1); > if (_PyUnicode_UTF8(unicode) == NULL) { > + PyErr_NoMemory(); > Py_DECREF(bytes); > return NULL; > } Shouldn't this (and related commits from #18408) have been committed to the 3.3 branch? Georg

3 3

Re: [Python-Dev] cpython: Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
by Georg Brandl Oct. 29, 2013

Oct. 29, 2013

Am 29.10.2013 01:19, schrieb victor.stinner: > http://hg.python.org/cpython/rev/4ef4578db38a > changeset: 86715:4ef4578db38a > user: Victor Stinner <victor.stinner(a)gmail.com> > date: Tue Oct 29 01:19:37 2013 +0100 > summary: > Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle > exceptions when merging fast locals into f_locals of a frame. > PyEval_GetLocals() now raises an exception and return NULL on failure. You'll have to either make this private or document it. Georg

3 3

Block module import during Python finalization?
by Victor Stinner Oct. 28, 2013

Oct. 28, 2013

Hi, While working on a fix, I got an assertion error during Python finalization because Python tried to import the "io" module whereas the module was just unloaded. Python tried to import the io module to display a warning and display the Python line where the warning was emitted. See the following issue for detail: http://bugs.python.org/issue19421 I propose to block importing new modules (or import again unloaded modules) during Python finalization to avoid such tricky bugs. What do you think? Does anyone rely on the ability of importing Python modules at exit? :-) At least, the whole test suite pass with my import_finalization.patch which block import during Python finalization. Victor

4 3

Merge codeaccess.txt from wesnoth-contribcommunity project in Google Code Hosting
by Tae Wong Oct. 28, 2013

Oct. 28, 2013

Here's the codeaccess.txt file. https://wesnoth-contribcommunity.googlecode.com/svn/codeaccess.txt All committer names have (No): Python; check the sixth line. Merging the names of committers in the sixth line to the Misc/ACKS file might be good.

4 6

[RELEASED] Python 3.3.3 release candidate 1
by Georg Brandl Oct. 27, 2013

Oct. 27, 2013

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm quite happy to announce the Python 3.3.3 release candidate 1. Python 3.3.3 includes several security fixes and over 150 bug fixes compared to the Python 3.3.2 release. This release fully supports OS X 10.9 Mavericks. In particular, this release fixes an issue that could cause previous versions of Python to crash when typing in interactive mode on OS X 10.9. Python 3.3.3 also contains a new batteries-included feature for OS X users of IDLE and other Tkinter-based programs. The python.org Mac OS X 64-bit/32-bit x86-64/i386 Installer for OS X 10.6+ now includes its own builtin version of Tcl/Tk 8.5. It is no longer necessary to install a third-party version of Tcl/Tk 8.5 to workaround the problematic system versions. See http://www.python.org/download/mac/tcltk/ for more information. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html and for the detailed changelog of 3.3.3, see http://docs.python.org/3.3/whatsnew/changelog.html To download Python 3.3.3 rc1 visit: http://www.python.org/download/releases/3.3.3/ This is a preview release, please report any bugs to http://bugs.python.org/ The final version is scheduled to be released in two weeks' time, on or about the 10th of November. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJte8kACgkQN9GcIYhpnLDx8wCgqtabbC8DaoW+Vy03AYGWyLhw sWcAoIK5jQeXDAxHayG+VWH/rRF1+qHC =yOed -----END PGP SIGNATURE-----

1 0

pathlib (PEP 428) status
by Charles-François Natali Oct. 27, 2013

Oct. 27, 2013

Hi, What's the current status of pathlib? Is it targeted for 3.4? It would be a really nice addition, and AFAICT it has already been maturing a while on pypi, and discussed several times here. If I remember correctly, the only remaining issue was stat()'s result caching. cf

5 4

[RELEASE] Python 2.7.6 release candidate 1
by Benjamin Peterson Oct. 27, 2013

Oct. 27, 2013

I'm happy to announce the availability of Python 2.7.6 release candidate 1. Most importantly, this release candidate resolves crashes of the interactive interpreter on OS X 10.9. It also includes the usual collection of bugfixes over 2.7.5. These are described in excruciating detail in the Misc/NEWS file of the source tarball. You can view it online at http://hg.python.org/cpython/raw-file/9750acbf7c40/Misc/NEWS Downloads are at http://python.org/download/releases/2.7.6/ As always, please test the release and report bugs to http://bugs.python.org/ With any luck, the final release will follow in a week. Enjoy, Benjamin Peterson 2.7 Release Manager

1 0

OrderedDict.values() behavior for modified instance
by Nikolaus Rath Oct. 27, 2013

Oct. 27, 2013

Hello, The documentation says the following about modifying a dict while iterating through its view: | Iterating views while adding or deleting entries in the dictionary may | raise a RuntimeError or fail to iterate over all entries. (http://docs.python.org/3/library/stdtypes.html#dict-views) The OrderedDict documentation doesn't have anything to say on the subject. In practice, its implementation actually mostly correponds to naive expectations: >>> from collections import OrderedDict >>> d = OrderedDict() >>> for i in range(5): ... d[i] = i ... >>> i = iter(d.values()) >>> next(i) 0 >>> del d[0] >>> next(i) 1 >>> del d[2] >>> next(i) 3 >>> d.move_to_end(1) >>> next(i) 4 >>> next(i) 1 >>> etc. However, there is one case that results in a rather confusing error: >>> a = OrderedDict() >>> a[0] = 0 >>> a[1] = 1 >>> a[2] = 2 >>> i = iter(a.values()) >>> next(i) 0 >>> del a[0] >>> del a[1] >>> next(i) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.3/collections/abc.py", line 495, in __iter__ yield self._mapping[key] KeyError: 1 What happens here is that a[0] is returned from the linked list, but it still contains links to a[1]. When a[1] is deleted, the links of its predecessor and successor are updated, but a[0] still points to a[1]. The OrderedList then hands a non-existing key to the values() implementation in collections.abc. The result is not only mightily confusing, but it is also not covered by the documentation (KeyError isn't a RuntimeError). I would very much like to see this fixed, but I'm not sure if there's a good way to do that. I see the following options: (a) When deleting an element from an OrderedList, update the pointers in the corresponding linked list element to the end of the list. Then removing the currently "active" element during the iteration would automatically end the iteration. For that, we'd just have to add two lines to __delitem__: def __delitem__(self, key, dict_delitem=dict.__delitem__): dict_delitem(self, key) link = self.__map.pop(key) link_prev = link.prev link_next = link.next link_prev.next = link_next link_next.prev = link_prev link.prev = root # new link.next = root # new The new behavior is explicitly covered by the documentation (changing the dict may result in incomplete iteration), but it's a change to what happened before. (b) When iterating through an OrderedDict, check that the next element is actually still in the hash, i.e. change def __iter__(self): root = self.__root curr = root.next while curr is not root: yield curr.key curr = curr.next to def __iter__(self): root = self.__root curr = root.next while curr is not root and curr.key in self: yield curr.key curr = curr.next that would terminate the iteration only in the special case, but incur an extra dict lookup at every iteration. Alternatively, one could try very hard to not stop the iteration: while curr is not root: yield curr.key while curr is not root: curr = curr.next if curr.key in self: break (c) Catch the KeyError in values(), and re-raise the proper exception in class ValuesView: def __iter__(self): for key in self._mapping: try: yield self._mapping[key] except KeyError: raise RuntimeError("underlying Mapping instance modified during iteration") I consider this a bit ugly, because it may mask other problems in a custom Mapping implementation (that may raise a KeyError because of a bug in the Mapping implementation itself). (d) Extend the OrderedDict documentation to explicity document this case. This has the drawback that it would probably be nicer to just have the behavior be consistent and correspond to intuitive expectations. Would any of those fixes be acceptable? Or is there an even better option? Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«

2 3

Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests
by Stefan Krah Oct. 26, 2013

Oct. 26, 2013

nick.coghlan <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/a9bbc2d0c1dc > -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or > - sys.platform == 'win32' or > - sysconfig.get_config_var('WITH_DOC_STRINGS')) > +# Rather than trying to enumerate all the cases where docstrings may be > +# disabled, we just check for that directly > + > +def _check_docstrings(): > + """Just used to check if docstrings are enabled""" > + > +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None) > > requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS, I think that does not detect --without-doc-strings (i.e. the C docstrings are empty). Stefan Krah

3 5

Re: [Python-Dev] cpython (2.7): 2.7.6rc1
by Christian Heimes Oct. 26, 2013

Oct. 26, 2013

Am 26.10.2013 20:58, schrieb benjamin.peterson: > http://hg.python.org/cpython/rev/4913d0e9be30 > changeset: 86665:4913d0e9be30 > branch: 2.7 > tag: v2.7.6rc1 > user: Benjamin Peterson <benjamin(a)python.org> > date: Sat Oct 26 14:57:21 2013 -0400 > summary: > 2.7.6rc1 We need to solve http://bugs.python.org/issue19227 before you release 2.7.6. I have suggested two approaches http://bugs.python.org/issue19227#msg201401

1 0

Re: [Python-Dev] [Python-checkins] cpython: Fiddled Thread.join() to be a little simpler. Kinda ; -)
by A.M. Kuchling Oct. 26, 2013

Oct. 26, 2013

On Sat, Oct 26, 2013 at 05:34:05AM +0200, tim.peters wrote: > Fiddled Thread.join() to be a little simpler. Kinda ;-) > - # else it's a negative timeout - precise behavior isn't documented > - # then, but historically .join() returned in this case > + else: > + # the behavior of a negative timeout isn't documented, but > + # historically .join() has acted as if timeout=0 then I think the final 'then' on the new comment is an editing error. Unless perhaps you meant to write a further sentence of comment? --amk

3 2