Mailman 3 February 2016 - Python-Dev

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton Sept. 15, 2019

Sept. 15, 2019

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

Speeding up CPython 5-10%
by Yury Selivanov May 17, 2016

May 17, 2016

Hi, tl;dr The summary is that I have a patch that improves CPython performance up to 5-10% on macro benchmarks. Benchmarks results on Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available at [1]. There are no slowdowns that I could reproduce consistently. There are twodifferent optimizations that yield this speedup: LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop. LOAD_METHOD & CALL_METHOD ------------------------- We had a lot of conversations with Victor about his PEP 509, and he sent me a link to his amazing compilation of notes about CPython performance [2]. One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an idea first originated in PyPy. There is a patch that implements this optimization, it's tracked here: [3]. There are some low level details that I explained in the issue, but I'll go over the high level design in this email as well. Every time you access a method attribute on an object, a BoundMethod object is created. It is a fairly expensive operation, despite a freelist of BoundMethods (so that memory allocation is generally avoided). The idea is to detect what looks like a method call in the compiler, and emit a pair of specialized bytecodes for that. So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD. LOAD_METHOD looks at the object on top of the stack, and checks if the name resolves to a method or to a regular attribute. If it's a method, then we push the unbound method object and the object to the stack. If it's an attribute, we push the resolved attribute and NULL. When CALL_METHOD looks at the stack it knows how to call the unbound method properly (pushing the object as a first arg), or how to call a regular callable. This idea does make CPython faster around 2-4%. And it surely doesn't make it slower. I think it's a safe bet to at least implement this optimization in CPython 3.6. So far, the patch only optimizes positional-only method calls. It's possible to optimize all kind of calls, but this will necessitate 3 more opcodes (explained in the issue). We'll need to do some careful benchmarking to see if it's really needed. Per-opcode cache in ceval ------------------------- While reading PEP 509, I was thinking about how we can use dict->ma_version in ceval to speed up globals lookups. One of the key assumptions (and this is what makes JITs possible) is that real-life programs don't modify globals and rebind builtins (often), and that most code paths operate on objects of the same type. In CPython, all pure Python functions have code objects. When you call a function, ceval executes its code object in a frame. Frames contain contextual information, including pointers to the globals and builtins dict. The key observation here is that almost all code objects always have same pointers to the globals (the module they were defined in) and to the builtins. And it's not a good programming practice to mutate globals or rebind builtins. Let's look at this function: def spam(): print(ham) Here are its opcodes: 2 0 LOAD_GLOBAL 0 (print) 3 LOAD_GLOBAL 1 (ham) 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3. Let's look at the first one, that loads the 'print' function from builtins. The opcode knows the following bits of information: - its offset (0), - its argument (0 -> 'print'), - its type (LOAD_GLOBAL). And these bits of information will *never* change. So if this opcode could resolve the 'print' name (from globals or builtins, likely the latter) and save the pointer to it somewhere, along with globals->ma_version and builtins->ma_version, it could, on its second call, just load this cached info back, check that the globals and builtins dict haven't changed and push the cached ref to the stack. That would save it from doing two dict lookups. We can also optimize LOAD_METHOD. There are high chances, that 'obj' in 'obj.method()' will be of the same type every time we execute the code object. So if we'd have an opcodes cache, LOAD_METHOD could then cache a pointer to the resolved unbound method, a pointer to obj.__class__, and tp_version_tag of obj.__class__. Then it would only need to check if the cached object type is the same (and that it wasn't modified) and that obj.__dict__ doesn't override 'method'. Long story short, this caching really speeds up method calls on types implemented in C. list.append becomes very fast, because list doesn't have a __dict__, so the check is very cheap (with cache). A straightforward way to implement such a cache is simple, but consumes a lot of memory, that would be just wasted, since we only need such a cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative about the cache design. Here's what I came up with: 1. We add a few fields to the code object. 2. ceval will count how many times each code object is executed. 3. When the code object is executed over ~900 times, we mark it as "hot". We also create an 'unsigned char' array "MAPPING", with length set to match the length of the code object. So we have a 1-to-1 mapping between opcodes and MAPPING array. 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and LOAD_METHOD do "MAPPING[opcode_offset()]++". 5. After 1024 calls to the code object, ceval loop will iterate through the MAPPING, counting all opcodes that were executed more than 50 times. 6. We then create an array of cache structs "CACHE" (here's a link to the updated code.h file: [6]). We update MAPPING to be a mapping between opcode position and position in the CACHE. The code object is now "optimized". 7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use the CACHE array for fast path. 8. When there is a cache miss, i.e. the builtins/global/obj.__dict__ were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and it will never try to use the cache again. Here's a link to the issue tracker with the first version of the patch: [5]. I'm working on the patch in a github repo here: [4]. Summary ------- There are many things about this algorithm that we can improve/tweak. Perhaps we should profile code objects longer, or account for time they were executed. Maybe we shouldn't deoptimize opcodes on their first cache miss. Maybe we can come up with better data structures. We also need to profile the memory and see how much more this cache will require. One thing I'm certain about, is that we can get a 5-10% speedup of CPython with relatively low memory impact. And I think it's worth exploring that! If you're interested in these kind of optimizations, please help with code reviews, ideas, profiling and benchmarks. The latter is especially important, I'd never imagine how hard it is to come up with a good macro benchmark. I also want to thank my company MagicStack (magic.io) for sponsoring this work. Thanks, Yury [1] https://gist.github.com/1st1/aed69d63a2ff4de4c7be [2] http://faster-cpython.readthedocs.org/index.html [3] http://bugs.python.org/issue26110 [4] https://github.com/1st1/cpython/tree/opcache2 [5] http://bugs.python.org/issue26219 [6] https://github.com/python/cpython/compare/master...1st1:opcache2?expand=1#d…

12 26

Wordcode v2
by Demur Rumed May 15, 2016

May 15, 2016

Saw recent discussion: https://mail.python.org/pipermail/python-dev/2016-February/143013.html I remember trying WPython; it was fast. Unfortunately it feels it came at the wrong time when development was invested in getting py3k out the door. It also had a lot of other ideas like *_INT instructions which allowed having oparg to be a constant int rather than needing to LOAD_CONST one. Anyways I'll stop reminiscing abarnert has started an experiment with wordcode: https://github.com/abarnert/cpython/blob/c095a32f2a68ac708466b9c64906cc4d0f… I've personally benchmarked this fork with positive results. This experiment seeks to be conservative-- it doesn't seek to introduce new opcodes or combine BINARY_OP's all into a single op where the currently unused-in-wordcode arg then states the kind of binary op (à la COMPARE_OP). I've submitted a pull request which is working on fixing tests & updating peephole.c Bringing this up on the list to figure out if there's interest in a basic wordcode change. It feels like there's no downsides: faster code, smaller bytecode, simpler interpretation of bytecode (The Nth instruction starts at the 2Nth byte if you count EXTENDED_ARG as an instruction). The only downside is the transitional cost What'd be necessary for this to be pulled upstream?

7 8

PEP 515: Underscores in Numeric Literals (revision 3)
by Georg Brandl May 12, 2016

May 12, 2016

Hi all, after talking to Guido and Serhiy we present the next revision of this PEP. It is a compromise that we are all happy with, and a relatively restricted rule that makes additions to PEP 8 basically unnecessary. I think the discussion has shown that supporting underscores in the from-string constructors is valuable, therefore this is now added to the specification section. The remaining open question is about the reverse direction: do we want a string formatting modifier that adds underscores as thousands separators? cheers, Georg ----------------------------------------------------------------- PEP: 515 Title: Underscores in Numeric Literals Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl, Serhiy Storchaka Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2016 Python-Version: 3.6 Post-History: 10-Feb-2016, 11-Feb-2016 Abstract and Rationale ====================== This PEP proposes to extend Python's syntax and number-from-string constructors so that underscores can be used as visual separators for digit grouping purposes in integral, floating-point and complex number literals. This is a common feature of other modern languages, and can aid readability of long literals, or literals whose value should clearly separate into parts, such as bytes or words in hexadecimal notation. Examples:: # grouping decimal numbers by thousands amount = 10_000_000.0 # grouping hexadecimal addresses by words addr = 0xDEAD_BEEF # grouping bits into nibbles in a binary literal flags = 0b_0011_1111_0100_1110 # same, for string conversions flags = int('0b_1111_0000', 2) Specification ============= The current proposal is to allow one underscore between digits, and after base specifiers in numeric literals. The underscores have no semantic meaning, and literals are parsed as if the underscores were absent. Literal Grammar --------------- The production list for integer literals would therefore look like this:: integer: decinteger | bininteger | octinteger | hexinteger decinteger: nonzerodigit (["_"] digit)* | "0" (["_"] "0")* bininteger: "0" ("b" | "B") (["_"] bindigit)+ octinteger: "0" ("o" | "O") (["_"] octdigit)+ hexinteger: "0" ("x" | "X") (["_"] hexdigit)+ nonzerodigit: "1"..."9" digit: "0"..."9" bindigit: "0" | "1" octdigit: "0"..."7" hexdigit: digit | "a"..."f" | "A"..."F" For floating-point and complex literals:: floatnumber: pointfloat | exponentfloat pointfloat: [digitpart] fraction | digitpart "." exponentfloat: (digitpart | pointfloat) exponent digitpart: digit (["_"] digit)* fraction: "." digitpart exponent: ("e" | "E") ["+" | "-"] digitpart imagnumber: (floatnumber | digitpart) ("j" | "J") Constructors ------------ Following the same rules for placement, underscores will be allowed in the following constructors: - ``int()`` (with any base) - ``float()`` - ``complex()`` - ``Decimal()`` Prior Art ========= Those languages that do allow underscore grouping implement a large variety of rules for allowed placement of underscores. In cases where the language spec contradicts the actual behavior, the actual behavior is listed. ("single" or "multiple" refer to allowing runs of consecutive underscores.) * Ada: single, only between digits [8]_ * C# (open proposal for 7.0): multiple, only between digits [6]_ * C++14: single, between digits (different separator chosen) [1]_ * D: multiple, anywhere, including trailing [2]_ * Java: multiple, only between digits [7]_ * Julia: single, only between digits (but not in float exponent parts) [9]_ * Perl 5: multiple, basically anywhere, although docs say it's restricted to one underscore between digits [3]_ * Ruby: single, only between digits (although docs say "anywhere") [10]_ * Rust: multiple, anywhere, except for between exponent "e" and digits [4]_ * Swift: multiple, between digits and trailing (although textual description says only "between digits") [5]_ Alternative Syntax ================== Underscore Placement Rules -------------------------- Instead of the relatively strict rule specified above, the use of underscores could be limited. As we seen from other languages, common rules include: * Only one consecutive underscore allowed, and only between digits. * Multiple consecutive underscores allowed, but only between digits. * Multiple consecutive underscores allowed, in most positions except for the start of the literal, or special positions like after a decimal point. The syntax in this PEP has ultimately been selected because it covers the common use cases, and does not allow for syntax that would have to be discouraged in style guides anyway. A less common rule would be to allow underscores only every N digits (where N could be 3 for decimal literals, or 4 for hexadecimal ones). This is unnecessarily restrictive, especially considering the separator placement is different in different cultures. Different Separators -------------------- A proposed alternate syntax was to use whitespace for grouping. Although strings are a precedent for combining adjoining literals, the behavior can lead to unexpected effects which are not possible with underscores. Also, no other language is known to use this rule, except for languages that generally disregard any whitespace. C++14 introduces apostrophes for grouping (because underscores introduce ambiguity with user-defined literals), which is not considered because of the use in Python's string literals. [1]_ Open Proposals ============== It has been proposed [11]_ to extend the number-to-string formatting language to allow ``_`` as a thousans separator, where currently only ``,`` is supported. This could be used to easily generate code with more readable literals. Implementation ============== A preliminary patch that implements the specification given above has been posted to the issue tracker. [12]_ References ========== .. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3499.html .. [2] http://dlang.org/spec/lex.html#integerliteral .. [3] http://perldoc.perl.org/perldata.html#Scalar-value-constructors .. [4] http://doc.rust-lang.org/reference.html#number-literals .. [5] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… .. [6] https://github.com/dotnet/roslyn/issues/216 .. [7] https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores… .. [8] http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html#2.4 .. [9] http://docs.julialang.org/en/release-0.4/manual/integers-and-floating-point… .. [10] http://ruby-doc.org/core-2.3.0/doc/syntax/literals_rdoc.html#label-Numbers .. [11] https://mail.python.org/pipermail/python-dev/2016-February/143283.html .. [12] http://bugs.python.org/issue26331 Copyright ========= This document has been placed in the public domain.

9 21

Modify PyMem_Malloc to use pymalloc for performance
by Victor Stinner April 22, 2016

April 22, 2016

Hi, There is an old discussion about the performance of PyMem_Malloc() memory allocator. CPython is stressing a lot memory allocators. Last time I made statistics, it was for the PEP 454: "For example, the Python test suites calls malloc() , realloc() or free() 270,000 times per second in average." https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator I proposed a simple change: modify PyMem_Malloc() to use the pymalloc allocator which is faster for allocation smaller than 512 bytes, or fallback to malloc() (which is the current internal allocator of PyMem_Malloc()). This tiny change makes Python up to 6% faster on some specific (macro) benchmarks, and it doesn't seem to make Python slower on any benchmark: http://bugs.python.org/issue26249#msg259445 Do you see any drawback of using pymalloc for PyMem_Malloc()? Does anyone recall the rationale to have two families to memory allocators? FYI Python has 3 families since 3.4: PyMem, PyObject but also PyMem_Raw! https://www.python.org/dev/peps/pep-0445/ -- Since pymalloc is only used for small memory allocations, I understand that small objects will not more be allocated on the heap memory, but only in pymalloc arenas which are allocated by mmap. The advantage of arenas is that it's possible to "punch holes" in the memory when a whole arena is freed, whereas the heap memory has the famous "fragmentation" issue because the heap is a single contiguous memory block. The libc malloc() uses mmap() for allocations larger than a threshold which is now dynamic, and initialized to 128 kB or 256 kB by default (I don't recall exactly the default value). Is there a risk of *higher* memory fragmentation if we start to use pymalloc for PyMem_Malloc()? Does someone know how to test it? Victor

4 19

PEP 506 secrets module
by Steven D'Aprano April 15, 2016

April 15, 2016

Hi, As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement. https://www.python.org/dev/peps/pep-0506/ There is code and tests here: https://bitbucket.org/sdaprano/secrets or you can run hg clone https://sdaprano@bitbucket.org/sdaprano/secrets The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4. -- Steve

13 28

Python environment registration in the Windows Registry
by Steve Dower March 13, 2016

March 13, 2016

I was throwing around some ideas with colleagues about how we detect Python installations on Windows from within Visual Studio, and it came up that there are many Python distros that install into different locations but write the same registry entries. (I knew about this, of course, but this time I decided to do something.) Apart from not being detected properly by all IDEs/tools/installers, non-standard distros that register themselves in the official keys may also mess with the default sys.path values. For example, at one point (possibly still true) if you installed both Canopy and Anaconda, you would break the first one because they tried to load the other's stdlib. Other implementations have different structures or do not register themselves at all, which also makes it more complicated for tools to discover them. So here is a rough proposal to standardise the registry keys that can be set on Windows in a way that (a) lets other installers besides the official ones have equal footing, (b) provides consistent search and resolution semantics for tools, and (c) includes slightly more rich metadata (such as display names and URLs). Presented in PEP-like form here, but if feedback suggests just putting it in the docs I'm okay with that too. It is fully backwards compatible with official releases of Python (at least back to 2.5, possibly further) and does not require modifications to Python or the official installer - it is purely codifying a superset of what we already do. Any and all feedback welcomed, especially from the owners of other distros, Python implementations or tools on the list. Cheers, Steve ----- PEP: ??? Title: Python environment registration in the Windows Registry Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Status: Draft Type: ??? Content-Type: text/x-rst Created: 02-Feb-2016 Abstract ======== When installed on Windows, the official Python installer creates a registry key for discovery and detection by other applications. Unofficial installers, such as those used by distributions, typically create identical keys for the same purpose. However, these may conflict with the official installer or other distributions. This PEP defines a schema for the Python registry key to allow unofficial installers to separately register their installation, and to allow applications to detect and correctly display all Python environments on a user's machine. No implementation changes to Python are proposed with this PEP. The schema matches the registry values that have been used by the official installer since at least Python 2.5, and the resolution behaviour matches the behaviour of the official Python releases. Specification ============= We consider there to be a single collection of Python environments on a machine, where the collection may be different for each user of the machine. There are three potential registry locations where the collection may be stored based on the installation options of each environment. These are:: HKEY_CURRENT_USER\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag> On a given machine, an environment is uniquely identified by its Company-Tag pair. Keys should be searched in the order shown, and if the same Company-Tag pair appears in more than one of the above locations, only the first occurrence is offerred. Official Python releases use ``PythonCore`` for Company, and the value of ``sys.winver`` for Tag. Other registered environments may use any values for Company and Tag. Recommendations are made in the following sections. Backwards Compatibility ----------------------- Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in ``sys.winver``. As a result, it is possible to have valid side-by-side installations of both 32-bit and 64-bit interpreters. To ensure backwards compatibility, applications should treat environments listed under the following two registry keys as distinct, even if Tag matches:: HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag> HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag> Note that this does not apply to Python 3.5 and later, which uses different Tags. Environments registered under other Company names must use distinct Tags for side-by-side installations. 1. Environments in ``HKEY_CURRENT_USER`` are always preferred 2. Environments in ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` are preferred if the interpreter is known to be 32-bit Company ------- The Company part of the key is intended to group related environments and to ensure that Tags are namespaced appropriately. The key name should be alphanumeric without spaces and likely to be unique. For example, a trademarked name, a UUID, or a hostname would be appropriate:: HKEY_CURRENT_USER\Software\Python\ExampleCorp HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60 HKEY_CURRENT_USER\Software\Python\www.example.com If a string value named ``DisplayName`` exists, it should be used to identify the environment category to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp (Default) = (value not set) DisplayName = "Example Corp" SupportUrl = "http://www.example.com" Tag --- The Tag part of the key is intended to uniquely identify an environment within those provided by a single company. The key name should be alphanumeric without spaces and stable across installations. For example, the Python language version, a UUID or a partial/complete hash would be appropriate; an integer counter that increases for each new environment may not:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6 HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 If a string value named ``DisplayName`` exists, it should be used to identify the environment to users. Otherwise, the name of the key should be used. If a string value named ``SupportUrl`` exists, it may be displayed or otherwise used to direct users to a web site related to the environment. If a string value named ``Version`` exists, it should be used to identify the version of the environment. This is independent from the version of Python implemented by the environment. If a string value named ``SysVersion`` exists, it must be in ``x.y`` or ``x.y.z`` format matching the version returned by ``sys.version_info`` in the interpreter. Otherwise, if the Tag matches this format it is used. If not, the Python version is unknown. Note that each of these values is recommended, but optional. A complete example may look like this:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66 (Default) = (value not set) DisplayName = "Distro 3" SupportUrl = "http://www.example.com/distro-3" Version = "3.0.12345.0" SysVersion = "3.6.0" InstallPath ----------- Beneath the environment key, an ``InstallPath`` key must be created. This key is always named ``InstallPath``, and the default value must match ``sys.prefix``:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath (Default) = "C:\ExampleCorpPy36" If a string value named ``ExecutablePath`` exists, it must be a path to the ``python.exe`` (or equivalent) executable. Otherwise, the interpreter executable is assumed to be called ``python.exe`` and exist in the directory referenced by the default value. If a string value named ``WindowedExecutablePath`` exists, it must be a path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the windowed interpreter executable is assumed to be called ``pythonw.exe`` and exist in the directory referenced by the default value. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath (Default) = "C:\ExampleDistro30" ExecutablePath = "C:\ExampleDistro30\ex_python.exe" WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe" Help ---- Beneath the environment key, a ``Help`` key may be created. This key is always named ``Help`` if present and has no default value. Each subkey of ``Help`` specifies a documentation file, tool, or URL associated with the environment. The subkey may have any name, and the default value is a string appropriate for passing to ``os.startfile`` or equivalent. If a string value named ``DisplayName`` exists, it should be used to identify the help file to users. Otherwise, the key name should be used. A complete example may look like:: HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help Python\ (Default) = "C:\ExampleDistro30\python36.chm" DisplayName = "Python Documentation" Extras\ (Default) = "http://www.example.com/tutorial" DisplayName = "Example Distro Online Tutorial"

13 31

Bug in build system for cross-platform builds
by Russell Keith-Magee March 11, 2016

March 11, 2016

Hi all, I’ve been working on developing Python builds for mobile platforms, and I’m looking for some help resolving a bug in Python’s build system. The problem affects cross-platform builds - builds where you are compiling python for a CPU architecture other than the one on the machine that is doing the compilation. This requirement stems from supporting mobile platforms (iOS, Android etc) where you compile on your laptop, then ship the compiled binary to the device. In the Python 3.5 dev cycle, Issue 22359 [1] was addressed, fixing parallel builds. However, as a side effect, this patch broke (as far as I can tell) *all* cross platform builds. This was reported in issue 22625 [2]. Since that time, the problem has gotten slightly worse; the addition of changeset 95566 [3] and 95854 [4] has cemented the problem. I’ve been able to hack together a fix that enables me to get a set of binaries, but the patch is essentially reverting 22359, and making some (very dubious) assumptions about the order in which things are built. Autoconf et al aren’t my strong suit; I was hoping someone might be able to help me resolve this issue. Yours, Russ Magee %-) [1] http://bugs.python.org/issue22359 [2] http://bugs.python.org/issue22625 [3] https://hg.python.org/cpython/rev/565b96093ec8 [4] https://hg.python.org/cpython/rev/02e3bf65b2f8

2 5

[ANNOUNCE] fuzzpy
by Brian Cain Feb. 28, 2016

Feb. 28, 2016

################################################################## *---------------------------------------------------* * fuzzpy: CPython fuzz tester is now available * * * * Version 0.8 * * https://bitbucket.org/ebadf/fuzzpy/ * *---------------------------------------------------* I am pleased to announce the creation of a coverage-guided fuzz tester for CPython. It's a pretty small wrapper around LLVM's libFuzzer that enables some powerful testing logic. AFL (American Fuzzy Lop) is another popular fuzzer lately -- libFuzzer is very similar in concept to AFL. From what I've read on list archives, Victor Stinner had previously done some good fuzz testing on CPython using fusil. This project should expand on that concept. I'd love to get feedback, suggestions, patches and anything else the list can offer. Q: What is fuzzpy for? A: It's primarily for testing CPython itself, but could also be used for individual python projects too. Pure-python projects will be the simplest to integrate at this point. Also, interesting test cases output by fuzzpy may end up being useful in testing others such as pypy, pyston, etc. Q: What is a fuzz tester? A: It modifies inputs to a test case in order to find unique/rare failures. Q: What does "coverage-guided" mean? A: It means that libFuzzer is able to witness the specific code executed as a result of a given test case. It feeds this information back into an engine to modify the test cases to optimize for coverage. Q: How can I help? A1: donate cycles: build the project and crank away on one of the existing tests. Relative to other common fuzzing, it's awfully slow, so consider throwing as many cycles as you can afford to. A2: contribute tests: write a ~10-line python script that exercises a feature that you think could benefit from fuzz testing. A3: if there's interest, I can accept cryptocoin donations to purchase cycles on a cloud server. ################################################################## -- -Brian

1 0

Very old git mirror under github user "python-git"
by John Mark Vandenberg Feb. 28, 2016

Feb. 28, 2016

Does anyone know who controls this mirror, which is attracting pull requests? https://github.com/python-git/python/pulls Can it be pulled down to avoid confusion, since it is using Python's logo? https://github.com/python-git -- John Vandenberg

13 14

Re: [Python-Dev] Python should be easily compilable on Windows with MinGW
by Alexander Walters Feb. 27, 2016

Feb. 27, 2016

Theoretically yes. Practically, I think, but do not know for sure, it would have the same annoying issues as other SDK builds (of which, the most annoying is just different paths for the tools). Making that a supported build would require some of the same effort as supporting any other compiler though (a build bot configured to compile python this way). I also think, but am not sure, that what you linked is in fact the leaner meaner toolchain that Steve was referring to. On 2/27/2016 16:49, Chris Krycho wrote: > > Outsider/observer here; but is it not possible to build Python using > the VS *toolchain* (compiler, linker, etc.) outside of VS itself—i.e. > using MSBuild[1] and so on? That would remove the need for the full VS > install, and is *much* smaller (~800MB after installation, rather than > 9GB). A lean and mean VS will be a great improvement regardless, but > it seems like that would be a good intermediate solution if it’s possible. > > [1]: https://www.microsoft.com/en-us/download/details.aspx?id=49983 > > – Chris > > On February 27, 2016 at 4:36:54 PM, Alexander Walters > (tritium-list(a)sdamon.com <mailto:tritium-list@sdamon.com>) wrote: > >> The 9 gig initial download is not the only problem. Visual studio is >> very bandwidth hungry in day to day operations (between polling >> websites and vcs remotes, near constant updating, integration with >> the VS web service, etc.). You can of course shut all of that off, >> but it's a pain. It's my understanding from Steve's post that a >> leaner, meaner edition of VS is in the works, so waiting for that >> might just be an overall better solution. >> >> On 2/27/2016 16:27, Franklin? Lee wrote: >>> >>> For this particular case, is there someone generous enough (or, can >>> someone apply for a PSF grant) to ship Mathieu a DVD/two/flash drive? >>> >>> On Feb 26, 2016 12:18 PM, "Mathieu Dupuy" <deronnax(a)gmail.com >>> <mailto:deronnax@gmail.com>> wrote: >>> >>> Hi. >>> I am currently working on adding some functionality on a standard >>> library module (http://bugs.python.org/issue15873). The Python part >>> went fine, but now I have to do the C counterpart, and I have >>> ran into >>> in several problems, which, stacked up, are a huge obstacle to >>> easily >>> contribute further. Currently, despite I could work, I can't go >>> further >>> on my patch. >>> >>> I am currently working in very limited network, CPU and time >>> ressources* which are quite uncommon in the western world, but are >>> much less in the rest of the world. I have a 2GB/month mobile data >>> plan and a 100KB/s speed. For the C part of my patch, I should >>> download Visual Studio. The Express Edition 2015 is roughly 9GB. I >>> can't afford that. >>> >>> I downloaded Virtualbox and two Linux netinstall (Ubuntu 15.10 and >>> Fedora 23). Shortly, I couldn't get something working quickly and >>> simply (quickly = less than 2 hours, downloading time NOT included, >>> which is anyway way too already much). What went wrong and why >>> it went >>> wrong could be a whole new thread and is outside of the scope of >>> this >>> message. >>> Let me precise this : at my work I use many virtualbox instances >>> automatically fired and run in parallel to test new deployments and >>> run unittests. I like this tool, >>> but despite its simple look, it (most of the time) can not be used >>> simply by a profane. The concepts it requires you to understand are >>> not intuitive at first sight and there is *always* a thing that go >>> wrong (guest additions, mostly).(for example : Ubuntu and Virtualbox >>> shipped for a moment a broken version of mount.vboxsf, preventing >>> sharing folder to mount. Despite it's fixed, the broken releases >>> spread everywhere and you may encounter them a lot in various Ubuntu >>> and Virtualbox version. I downloaded the last versions of both and I >>> am yet infected. https://www.virtualbox.org/ticket/12879). I >>> could do >>> whole new thread on why you can't ask newcomers to use Virtualbox >>> (currently, at least). >>> >>> I ran into is a whole patch set to make CPython compile on MinGW >>> (https://bugs.python.org/issue3871#msg199695). But it is not denying >>> it's very experimental, and I know I would again spent useless hours >>> trying to get it work rather than joyfully improving Python, and >>> that's exactly what I do not want to happen. >>> >>> Getting ready to contribute to CPython pure python modules from an >>> standard, average mr-everyone Windows PC for a beginner-to-medium >>> contributor only require few megabytes of internet and few >>> minutes of his >>> time: getting a tarball of CPython sources (or cloning the >>> github CPython >>> mirror)**, a basic text editor and msys-git. The step further, >>> if doing >>> some -even basic- C code is required, implies downloading 9GB of >>> Visual >>> Studio and countless hours for it to be ready to use. >>> I think downloading the whole Visual Studio suite is a huge >>> stopper to >>> contribute further for an average medium-or-below-contributor. >>> >>> I think (and I must not be the only one since CPython is to be moved >>> to github), that barriers to contribute to CPython should be set to >>> the lowest. >>> Of course my situation is a bit special but I think it represents >>> daily struggle of a *lot* of non-western programmer (at least for >>> limited internet)(even here in Australia, landline limited internet >>> connections are very common). >>> It's not a big deal if the MinGW result build is twenty time >>> slower or >>> if some of the most advanced modules can't be build. But everyone >>> programmer should be able to easily make some C hacks and get >>> them to >>> work. >>> >>> Hoping you'll be receptive to my pleas, >>> Cheers >>> >>> >>> * I am currently picking fruits in the regional Australia. I >>> live in a van >>> and have internet through with smartphone through an EDGE >>> connection. I can >>> plug the laptop in the farm but not in the van. >>> ** No fresh programmer use mercurial unless he has a gun pointed >>> on his >>> head. >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev(a)python.org <mailto:Python-Dev@python.org> >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/leewangzhong%2Bpython%40… >>> >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev(a)python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe:https://mail.python.org/mailman/options/python-dev/tritium-list… >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev(a)python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/chris%40chriskrycho.com

2 1

Python should be easily compilable on Windows with MinGW
by Mathieu Dupuy Feb. 27, 2016

Feb. 27, 2016

Hi. I am currently working on adding some functionality on a standard library module (http://bugs.python.org/issue15873). The Python part went fine, but now I have to do the C counterpart, and I have ran into in several problems, which, stacked up, are a huge obstacle to easily contribute further. Currently, despite I could work, I can't go further on my patch. I am currently working in very limited network, CPU and time ressources* which are quite uncommon in the western world, but are much less in the rest of the world. I have a 2GB/month mobile data plan and a 100KB/s speed. For the C part of my patch, I should download Visual Studio. The Express Edition 2015 is roughly 9GB. I can't afford that. I downloaded Virtualbox and two Linux netinstall (Ubuntu 15.10 and Fedora 23). Shortly, I couldn't get something working quickly and simply (quickly = less than 2 hours, downloading time NOT included, which is anyway way too already much). What went wrong and why it went wrong could be a whole new thread and is outside of the scope of this message. Let me precise this : at my work I use many virtualbox instances automatically fired and run in parallel to test new deployments and run unittests. I like this tool, but despite its simple look, it (most of the time) can not be used simply by a profane. The concepts it requires you to understand are not intuitive at first sight and there is *always* a thing that go wrong (guest additions, mostly).(for example : Ubuntu and Virtualbox shipped for a moment a broken version of mount.vboxsf, preventing sharing folder to mount. Despite it's fixed, the broken releases spread everywhere and you may encounter them a lot in various Ubuntu and Virtualbox version. I downloaded the last versions of both and I am yet infected. https://www.virtualbox.org/ticket/12879). I could do whole new thread on why you can't ask newcomers to use Virtualbox (currently, at least). I ran into is a whole patch set to make CPython compile on MinGW (https://bugs.python.org/issue3871#msg199695). But it is not denying it's very experimental, and I know I would again spent useless hours trying to get it work rather than joyfully improving Python, and that's exactly what I do not want to happen. Getting ready to contribute to CPython pure python modules from an standard, average mr-everyone Windows PC for a beginner-to-medium contributor only require few megabytes of internet and few minutes of his time: getting a tarball of CPython sources (or cloning the github CPython mirror)**, a basic text editor and msys-git. The step further, if doing some -even basic- C code is required, implies downloading 9GB of Visual Studio and countless hours for it to be ready to use. I think downloading the whole Visual Studio suite is a huge stopper to contribute further for an average medium-or-below-contributor. I think (and I must not be the only one since CPython is to be moved to github), that barriers to contribute to CPython should be set to the lowest. Of course my situation is a bit special but I think it represents daily struggle of a *lot* of non-western programmer (at least for limited internet)(even here in Australia, landline limited internet connections are very common). It's not a big deal if the MinGW result build is twenty time slower or if some of the most advanced modules can't be build. But everyone programmer should be able to easily make some C hacks and get them to work. Hoping you'll be receptive to my pleas, Cheers * I am currently picking fruits in the regional Australia. I live in a van and have internet through with smartphone through an EDGE connection. I can plug the laptop in the farm but not in the van. ** No fresh programmer use mercurial unless he has a gun pointed on his head.

10 11

Re: [Python-Dev] Python should be easily compilable on Windows with MinGW
by Alexander Walters Feb. 26, 2016

Feb. 26, 2016

You mean honestly pointing out what would happen with a suggestion? It is a horrifically bad idea. I didn't say they were bad people. On 2/26/2016 13:14, Brian Curtin wrote: > The attitude in these responses is counter productive and not really > how it works on this list. > > On Fri, Feb 26, 2016 at 1:10 PM, Alexander Walters > <tritium-list(a)sdamon.com> wrote: >> Ok, fine. Bring a windows build bot online. And also take on the support >> burden of guiding people to which version of which compiler you use for each >> of the currently supported python versions. And go ahead and write the pep >> to change how wheel distributions work (which will effectively kill them, so >> yeah, good side benefit there.) >> >> Want to kill python on windows for anything that needs a c extension? go >> ahead, release one version of python with 2 ABIs. >> >> What do I know. >> >> >> On 2/26/2016 13:05, Dan Stromberg wrote: >>> But what do you really think? >>> >>> IMO, windows builds probably should do both visual studio and mingw. >>> That is, there probably should be two builds on windows, since there's >>> no clear consensus about which to use. >>> >>> I certainly prefer mingw over visual studio - and I have adequate >>> bandwidth for either. >>> >>> >>> On Fri, Feb 26, 2016 at 9:55 AM, Alexander Walters >>> <tritium-list(a)sdamon.com> wrote: >>>> No. >>>> >>>> Visual Studio is a solid compiler suit, mingw is a jenky mess, especially >>>> when you try and move to 64bit (where I don't think there is one true >>>> version of mingw). I'm sorry that Visual Studio makes it very hard for >>>> you >>>> to contribute, but changing THE compiler of the distribution from the >>>> platform compiler, especially when we FINALLY got a stable abi with it, >>>> is >>>> going to be a non starter. >>>> >>>> Compiling on MinGW for your own edification is fine, but that's not the >>>> build platform for windows python, nor should it be. Contributions are, >>>> and >>>> should continue to be, tested against Visual Studio. >>>> >>>> >>>> On 2/26/2016 05:12, Mathieu Dupuy wrote: >>>>> Hi. >>>>> I am currently working on adding some functionality on a standard >>>>> library module (http://bugs.python.org/issue15873). The Python part >>>>> went fine, but now I have to do the C counterpart, and I have ran into >>>>> in several problems, which, stacked up, are a huge obstacle to easily >>>>> contribute further. Currently, despite I could work, I can't go >>>>> further >>>>> on my patch. >>>>> >>>>> I am currently working in very limited network, CPU and time >>>>> ressources* which are quite uncommon in the western world, but are >>>>> much less in the rest of the world. I have a 2GB/month mobile data >>>>> plan and a 100KB/s speed. For the C part of my patch, I should >>>>> download Visual Studio. The Express Edition 2015 is roughly 9GB. I >>>>> can't afford that. >>>>> >>>>> I downloaded Virtualbox and two Linux netinstall (Ubuntu 15.10 and >>>>> Fedora 23). Shortly, I couldn't get something working quickly and >>>>> simply (quickly = less than 2 hours, downloading time NOT included, >>>>> which is anyway way too already much). What went wrong and why it went >>>>> wrong could be a whole new thread and is outside of the scope of this >>>>> message. >>>>> Let me precise this : at my work I use many virtualbox instances >>>>> automatically fired and run in parallel to test new deployments and >>>>> run unittests. I like this tool, >>>>> but despite its simple look, it (most of the time) can not be used >>>>> simply by a profane. The concepts it requires you to understand are >>>>> not intuitive at first sight and there is *always* a thing that go >>>>> wrong (guest additions, mostly).(for example : Ubuntu and Virtualbox >>>>> shipped for a moment a broken version of mount.vboxsf, preventing >>>>> sharing folder to mount. Despite it's fixed, the broken releases >>>>> spread everywhere and you may encounter them a lot in various Ubuntu >>>>> and Virtualbox version. I downloaded the last versions of both and I >>>>> am yet infected. https://www.virtualbox.org/ticket/12879). I could do >>>>> whole new thread on why you can't ask newcomers to use Virtualbox >>>>> (currently, at least). >>>>> >>>>> I ran into is a whole patch set to make CPython compile on MinGW >>>>> (https://bugs.python.org/issue3871#msg199695). But it is not denying >>>>> it's very experimental, and I know I would again spent useless hours >>>>> trying to get it work rather than joyfully improving Python, and >>>>> that's exactly what I do not want to happen. >>>>> >>>>> Getting ready to contribute to CPython pure python modules from an >>>>> standard, average mr-everyone Windows PC for a beginner-to-medium >>>>> contributor only require few megabytes of internet and few minutes of >>>>> his >>>>> time: getting a tarball of CPython sources (or cloning the github >>>>> CPython >>>>> mirror)**, a basic text editor and msys-git. The step further, if doing >>>>> some -even basic- C code is required, implies downloading 9GB of Visual >>>>> Studio and countless hours for it to be ready to use. >>>>> I think downloading the whole Visual Studio suite is a huge stopper to >>>>> contribute further for an average medium-or-below-contributor. >>>>> >>>>> I think (and I must not be the only one since CPython is to be moved >>>>> to github), that barriers to contribute to CPython should be set to >>>>> the lowest. >>>>> Of course my situation is a bit special but I think it represents >>>>> daily struggle of a *lot* of non-western programmer (at least for >>>>> limited internet)(even here in Australia, landline limited internet >>>>> connections are very common). >>>>> It's not a big deal if the MinGW result build is twenty time slower or >>>>> if some of the most advanced modules can't be build. But everyone >>>>> programmer should be able to easily make some C hacks and get them to >>>>> work. >>>>> >>>>> Hoping you'll be receptive to my pleas, >>>>> Cheers >>>>> >>>>> >>>>> * I am currently picking fruits in the regional Australia. I live in a >>>>> van >>>>> and have internet through with smartphone through an EDGE connection. I >>>>> can >>>>> plug the laptop in the farm but not in the van. >>>>> ** No fresh programmer use mercurial unless he has a gun pointed on his >>>>> head. >>>>> _______________________________________________ >>>>> Python-Dev mailing list >>>>> Python-Dev(a)python.org >>>>> https://mail.python.org/mailman/listinfo/python-dev >>>>> Unsubscribe: >>>>> >>>>> https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com >>>> >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev(a)python.org >>>> https://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: >>>> https://mail.python.org/mailman/options/python-dev/drsalists%40gmail.com >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev(a)python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brian%40python.org

2 1

Summary of Python tracker Issues
by Python tracker Feb. 26, 2016

Feb. 26, 2016

ACTIVITY SUMMARY (2016-02-19 - 2016-02-26) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5437 ( +5) closed 32764 (+46) total 38201 (+51) Open issues with patches: 2378 Issues opened (25) ================== #26393: random.shuffled http://bugs.python.org/issue26393 opened by palaviv #26394: Have argparse provide ability to require a fallback value be p http://bugs.python.org/issue26394 opened by quabla #26396: Create json.JSONType http://bugs.python.org/issue26396 opened by brett.cannon #26403: Catch FileNotFoundError in socketserver.DatagramRequestHandler http://bugs.python.org/issue26403 opened by desbma #26404: socketserver context manager http://bugs.python.org/issue26404 opened by palaviv #26407: csv.writer.writerows masks exceptions from __iter__ http://bugs.python.org/issue26407 opened by Ilja Everilä #26410: "incompatible pointer type" while compiling Python3.5.1 http://bugs.python.org/issue26410 opened by Devyn Johnson #26414: os.defpath too permissive http://bugs.python.org/issue26414 opened by jbeck #26415: Out of memory, trying to parse a 35MB dict http://bugs.python.org/issue26415 opened by A. Skrobov #26418: multiprocessing.pool.ThreadPool eats up memories http://bugs.python.org/issue26418 opened by renlifeng #26420: IDEL for Python 3.5.1 for x64 Windows exits when pasted a stri http://bugs.python.org/issue26420 opened by tats.u. #26421: string_richcompare invalid check Py_NotImplemented http://bugs.python.org/issue26421 opened by yuriy_levchenko #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows http://bugs.python.org/issue26423 opened by Dave Hibbitts #26425: 'TypeError: object of type 'NoneType' has no len()' in 'splitd http://bugs.python.org/issue26425 opened by Konrad #26432: Add partial.kwargs http://bugs.python.org/issue26432 opened by serhiy.storchaka #26433: urllib.urlencode() does not explain how to handle unicode http://bugs.python.org/issue26433 opened by Thomas Güttler #26434: multiprocessing cannot spawn grandchild from a Windows service http://bugs.python.org/issue26434 opened by schlamar #26436: Add the regex-dna benchmark http://bugs.python.org/issue26436 opened by serhiy.storchaka #26437: asyncio create_server() not always accepts the 'port' paramete http://bugs.python.org/issue26437 opened by xdegaye #26439: ctypes.util.find_library fails when ldconfig/glibc not availab http://bugs.python.org/issue26439 opened by Michael.Felt #26440: tarfile._FileInFile.seekable is broken in stream mode http://bugs.python.org/issue26440 opened by Bill Lee #26441: email.charset: to_splittable and from_splittable are not there http://bugs.python.org/issue26441 opened by martin.panter #26442: Doc refers to xmlrpc.client but means xmlrpc.server http://bugs.python.org/issue26442 opened by Valentin.Lorentz #26443: cross building extensions picks up host headers http://bugs.python.org/issue26443 opened by hundeboll #26444: Fix 2 typos on ElementTree docs http://bugs.python.org/issue26444 opened by Ismail s Most recent 15 issues with no replies (15) ========================================== #26444: Fix 2 typos on ElementTree docs http://bugs.python.org/issue26444 #26443: cross building extensions picks up host headers http://bugs.python.org/issue26443 #26442: Doc refers to xmlrpc.client but means xmlrpc.server http://bugs.python.org/issue26442 #26441: email.charset: to_splittable and from_splittable are not there http://bugs.python.org/issue26441 #26433: urllib.urlencode() does not explain how to handle unicode http://bugs.python.org/issue26433 #26432: Add partial.kwargs http://bugs.python.org/issue26432 #26418: multiprocessing.pool.ThreadPool eats up memories http://bugs.python.org/issue26418 #26396: Create json.JSONType http://bugs.python.org/issue26396 #26393: random.shuffled http://bugs.python.org/issue26393 #26391: Specialized sub-classes of Generic never call __init__ http://bugs.python.org/issue26391 #26383: benchmarks (perf.py): number of decimal places in csv output http://bugs.python.org/issue26383 #26373: asyncio: add support for async context manager on streams? http://bugs.python.org/issue26373 #26363: __builtins__ propagation is misleading described in exec and e http://bugs.python.org/issue26363 #26359: CPython build options for out-of-the box performance http://bugs.python.org/issue26359 #26358: mmap.mmap.__iter__ is broken (yields bytes instead of ints) http://bugs.python.org/issue26358 Most recent 15 issues waiting for review (15) ============================================= #26444: Fix 2 typos on ElementTree docs http://bugs.python.org/issue26444 #26443: cross building extensions picks up host headers http://bugs.python.org/issue26443 #26436: Add the regex-dna benchmark http://bugs.python.org/issue26436 #26432: Add partial.kwargs http://bugs.python.org/issue26432 #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows http://bugs.python.org/issue26423 #26414: os.defpath too permissive http://bugs.python.org/issue26414 #26404: socketserver context manager http://bugs.python.org/issue26404 #26403: Catch FileNotFoundError in socketserver.DatagramRequestHandler http://bugs.python.org/issue26403 #26393: random.shuffled http://bugs.python.org/issue26393 #26388: Disabling changing sys.argv[0] with runpy.run_module(...alter_ http://bugs.python.org/issue26388 #26387: Crash calling sqlite3_close with invalid pointer http://bugs.python.org/issue26387 #26386: tkinter - Treeview - .selection_add and selection_toggle http://bugs.python.org/issue26386 #26385: the call of tempfile.NamedTemporaryFile fails and leaves a fil http://bugs.python.org/issue26385 #26384: UnboundLocalError in socket._sendfile_use_sendfile http://bugs.python.org/issue26384 #26382: List object memory allocator http://bugs.python.org/issue26382 Top 10 most discussed issues (10) ================================= #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows http://bugs.python.org/issue26423 15 msgs #26221: awaiting asyncio.Future swallows StopIteration http://bugs.python.org/issue26221 11 msgs #26385: the call of tempfile.NamedTemporaryFile fails and leaves a fil http://bugs.python.org/issue26385 9 msgs #26415: Out of memory, trying to parse a 35MB dict http://bugs.python.org/issue26415 8 msgs #19475: Add timespec optional flag to datetime isoformat() to choose t http://bugs.python.org/issue19475 7 msgs #26376: Tkinter root window won't close if packed. http://bugs.python.org/issue26376 7 msgs #26039: More flexibility in zipfile interface http://bugs.python.org/issue26039 6 msgs #26281: Clear sys.path_importer_cache from importlib.invalidate_caches http://bugs.python.org/issue26281 6 msgs #26323: Add assert_called() and assert_called_once() methods for mock http://bugs.python.org/issue26323 6 msgs #21042: ctypes.util.find_library() should return full pathname instead http://bugs.python.org/issue21042 5 msgs Issues closed (44) ================== #1429: FD leak in SocketServer when request handler throws exception http://bugs.python.org/issue1429 closed by martin.panter #5824: SocketServer.DatagramRequestHandler Broken under Linux http://bugs.python.org/issue5824 closed by martin.panter #21996: gettarinfo method does not handle files without text string na http://bugs.python.org/issue21996 closed by martin.panter #22088: base64 module still ignores non-alphabet characters http://bugs.python.org/issue22088 closed by martin.panter #22468: Tarfile using fstat on GZip file object http://bugs.python.org/issue22468 closed by martin.panter #23430: socketserver.BaseServer.handle_error() should not catch exitin http://bugs.python.org/issue23430 closed by martin.panter #24229: pathlib.Path should have a copy() method http://bugs.python.org/issue24229 closed by serhiy.storchaka #25080: The example-code for making XML-RPC requests through proxy, fa http://bugs.python.org/issue25080 closed by berker.peksag #25136: Python doesn't find Xcode 7 SDK stub libraries http://bugs.python.org/issue25136 closed by ned.deily #25139: socketserver.ThreadingMixIn exception handler: Just a little r http://bugs.python.org/issue25139 closed by martin.panter #25801: ResourceWarning in test_zipfile64 http://bugs.python.org/issue25801 closed by SilentGhost #25913: base64.a85decode adobe flag incorrectly utilizes <~ as a marke http://bugs.python.org/issue25913 closed by serhiy.storchaka #26261: NamedTemporaryFile documentation is vague about the `name` att http://bugs.python.org/issue26261 closed by martin.panter #26302: cookies module allows commas in keys http://bugs.python.org/issue26302 closed by jason.coombs #26366: Use “.. versionadded” over “.. versionchanged” where a http://bugs.python.org/issue26366 closed by rhettinger #26367: importlib.__import__ does not fail for invalid relative import http://bugs.python.org/issue26367 closed by brett.cannon #26390: hashlib's pbkdf2_hmac documentation "rounds" does not match so http://bugs.python.org/issue26390 closed by martin.panter #26392: socketserver.BaseServer.close_server should stop serve_forever http://bugs.python.org/issue26392 closed by palaviv #26395: asyncio does not support yielding from recvfrom (socket/udp) http://bugs.python.org/issue26395 closed by Simon Bernier St-Pierre #26397: Tweak importlib Example of importlib.import_module() to use im http://bugs.python.org/issue26397 closed by brett.cannon #26398: cgi.escape() Can Lead To XSS and HTML Vulnerabilities http://bugs.python.org/issue26398 closed by gregory.p.smith #26399: CSV Injection Vulnerability http://bugs.python.org/issue26399 closed by maciej.szulik #26400: SyntaxError when running Python 2.7 interpreter with subproces http://bugs.python.org/issue26400 closed by giumas #26401: Error in documentation for "compile" built-in function http://bugs.python.org/issue26401 closed by berker.peksag #26402: Regression in Python 3.5 xmlrpc.client, raises RemoteDisconnec http://bugs.python.org/issue26402 closed by martin.panter #26405: tkinter askopenfilename doubleclick issue on windows http://bugs.python.org/issue26405 closed by serhiy.storchaka #26406: getaddrinfo is thread-safe on NetBSD and OpenBSD http://bugs.python.org/issue26406 closed by ned.deily #26408: pep-8 requires a few corrections http://bugs.python.org/issue26408 closed by python-dev #26409: Support latest Tcl/Tk on future versions of Mac installer http://bugs.python.org/issue26409 closed by ned.deily #26411: Suggestion concerning compile-time warnings http://bugs.python.org/issue26411 closed by brett.cannon #26412: Segmentation Fault: 11 http://bugs.python.org/issue26412 closed by christian.heimes #26413: python 3.5.1 uses wrong registry in system-wide installation http://bugs.python.org/issue26413 closed by eryksun #26416: Deprecate the regex_v8, telco, and spectral_norm benchmarks http://bugs.python.org/issue26416 closed by brett.cannon #26417: Default IDLE 2.7.11 configuration files are out-of-sync on OS http://bugs.python.org/issue26417 closed by ned.deily #26422: printing 1e23 and up is incorrect http://bugs.python.org/issue26422 closed by eric.smith #26424: QPyNullVariant http://bugs.python.org/issue26424 closed by haypo #26426: email examples: incorrect use of email.headerregistry.Address http://bugs.python.org/issue26426 closed by berker.peksag #26427: w* format in PyArg_ParseTupleAndKeywords for optional argument http://bugs.python.org/issue26427 closed by serhiy.storchaka #26428: The range for xrange() is too narrow on Windows 64-bit http://bugs.python.org/issue26428 closed by serhiy.storchaka #26429: os.path.dirname returns empty string instead of "." when file http://bugs.python.org/issue26429 closed by serhiy.storchaka #26430: quote marks problem on loaded file http://bugs.python.org/issue26430 closed by SilentGhost #26431: string template substitute tests http://bugs.python.org/issue26431 closed by serhiy.storchaka #26435: Fix versionadded/versionchanged documentation directives http://bugs.python.org/issue26435 closed by python-dev #26438: Complete your registration to Python tracker -- key4g5ti2VWPYC http://bugs.python.org/issue26438 closed by christian.heimes

1 0

PEP 493: HTTPS verification migration tools for Python 2.7
by Nick Coghlan Feb. 25, 2016

Feb. 25, 2016

Hi folks, Since the last discussion back in November (just after the RHEL 7.2 release), I've rewritten PEP 493 to be a standards track PEP targeting Python 2.7.12. Barry also kindly volunteered to serve as BDFL-Delegate, so we have a clear path to pronouncement if nobody notices any new problems or concerns that didn't come up in previous discussions :) The PEP now focuses on adding two new configuration mechanisms to the PEP 476 backport in Python 2.7: * turning off default certificate verification through a Python API * turning off default certificate verification through an environment variable Both of these are defined in such a way that if backported to a version where default verification is still off by default, they can be used to turn it *on*. The original file based configuration proposal to change the default behaviour of older versions is also still covered, but moved to a clearly optional section. (The gist of that section is now "If you backport this capability, aim to stay consistent with already existing backports of it") Regards, Nick. ========================================== PEP: 493 Title: HTTPS verification migration tools for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Robert Kuska <rkuska(a)redhat.com>, Marc-André Lemburg <mal(a)lemburg.com> BDFL-Delegate: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015 Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015, 24-Feb-2016 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates in client modules to align with certificate handling in web browsers, by validating that the certificates received belonged to the server the client was attempting to contact. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This has created a non-trivial barrier to adoption for affected Python 2.7 maintenance releases, so this PEP proposes additional Python 2.7 specific features that allow system administrators and other users to more easily decouple the decision to verify server certificates in HTTPS client modules from the decision to update to newer Python 2.7 maintenance releases. Rationale ========= PEP 476 changed Python's default behaviour to align with expectations established by web browsers in regards to the semantics of HTTPS URLs: starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library validate server certificates by default. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. To manage these kinds of situations, web browsers provide users with "click through" warnings that allow the user to add the server's certificate to the browser's certificate store. Network client tools like ``curl`` and ``wget`` offer options to switch off certificate checking entirely (by way of ``curl --insecure`` and ``wget --no-check-certificate``, respectively). At a different layer of the technology stack, Linux security modules like `SELinux` and `AppArmor`, while enabled by default by distribution vendors, offer relatively straightforward mechanisms for turning them off. At the moment, no such convenient mechanisms exist to disable Python's default certificate checking for a whole process. PEP 476 did attempt to address this question, by covering how to revert to the old settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case that led to the initial creation of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path < https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes an additional feature to be added to Python 2.7.12 to make it easier to revert a process to the past behaviour of skipping certificate validation in HTTPS client modules. It also provides additional recommendations to redistributors backporting these features to versions of Python prior to Python 2.7.9. Alternatives ------------ In the absence of clear upstream guidance and recommendations, commercial redistributors will still make their own design decisions in the interests of their customers. The main approaches available are: * Continuing to rebase on new Python 2.7.x releases, while providing no additional assistance beyond the mechanisms defined in PEP 476 in migrating from unchecked to checked hostnames in standard library HTTPS clients * Gating availability of the changes in default handling of HTTPS connections on upgrading from Python 2 to Python 3 * For Linux distribution vendors, gating availability of the changes in default handling of HTTPS connections on upgrading to a new operating system version * Implementing one or both of the backport suggestions described in this PEP, regardless of the formal status of the PEP Scope Limitations ================= These changes are being proposed purely as tools for helping to manage the transition to the new default certificate handling behaviour in the context of Python 2.7. They are not being proposed as new features for Python 3, as it is expected that the vast majority of client applications affected by this problem without the ability to update the application itself will be Python 2 applications. It would likely be desirable for a future version of Python 3 to allow the default certificate handling for secure protocols to be configurable on a per-protocol basis, but that question is beyond the scope of this PEP. Requirements for capability detection ===================================== As the proposals in this PEP aim to facilitate backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, they are designed to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl.<_relevant_attribute>" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The feature detection attributes defined by this PEP are: * ``ssl._https_verify_certificates``: runtime configuration API * ``ssl._https_verify_envvar``: environment based configuration * ``ssl._cert_verification_config``: file based configuration (PEP 476 opt-in) The marker attributes are prefixed with an underscore to indicate the implementation dependent and security sensitive nature of these capabilities. Feature: Configuration API ========================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``ssl._https_verify_certificates()`` to specify the default handling of HTTPS certificates in standard library client libraries. It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is the ``ssl._https_verify_certificates`` function itself. Specification ------------- The ``ssl._https_verify_certificates`` function will work as follows:: def _https_verify_certificates(enable=True): """Verify server HTTPS certificates by default?""" global _create_default_https_context if enable: _create_default_https_context = create_default_context else: _create_default_https_context = _create_unverified_context If called without arguments, or with ``enable`` set to a true value, then standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification. If called with ``enable`` set to a false value, then standard library client modules will subsequently skip verifying HTTPS certificates by default. Security Considerations ----------------------- The inclusion of this feature will allow security sensitive applications to include the following forward-compatible snippet in their code:: if hasattr(ssl, "_https_verify_certificates"): ssl._https_verify_certificates() Some developers may also choose to opt out of certificate checking using ``ssl._https_verify_certificates(enable=False)``. This doesn't introduce any major new security concerns, as monkeypatching the affected internal APIs was already possible. Feature: environment based configuration ======================================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that can be set to ``'0'`` to disable the default verification without modifying the application source code (which may not even be available in cases of bytecode-only application distribution) It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is: * the ``ssl._https_verify_envvar`` attribute, giving the name of environment variable affecting the default behaviour This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Specification ------------- Rather than always defaulting to the use of ``ssl.create_default_context``, the ``ssl`` module will be modified to: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the default behaviour used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Reference Implementation ======================== A patch for Python 2.7 implementing the above two features is attached to the `relevant tracker issue <http://bugs.python.org/issue23857>`__. Backporting this PEP to earlier Python versions =============================================== If this PEP is accepted, then commercial Python redistributors may choose to backport the per-process configuration mechanisms defined in this PEP to base versions older than Python 2.7.9, *without* also backporting PEP 476's change to the default behaviour of the overall Python installation. Such a backport would differ from the mechanism proposed in this PEP solely in the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would continue to default to skipping certificate validation. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Feature detection ----------------- There's no specific attribute indicating that this situation applies. Rather, it is indicated by the ``ssl._https_verify_certificates`` and ``ssl._https_verify_envvar`` attributes being present in a Python version that is nominally older than Python 2.7.12. Specification ------------- Implementing this backport involves backporting the changes in PEP 466, 476 and this PEP, with the following change to the handling of the ``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` if this environment variable is present and set to any value other than ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting != '0': return create_default_context return _create_unverified_context _create_default_https_context = _get_https_context_factory() def _disable_https_default_verification(): """Skip verification of HTTPS certificates by default""" global _create_default_https_context _create_default_https_context = _create_unverified_context Security Considerations ----------------------- This change would be a strict security upgrade for any Python version that currently defaults to skipping certificate validation in standard library HTTPS clients. The technical trade-offs to be taken into account relate largely to the magnitude of the PEP 466 backport also required rather than to anything security related. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Backporting PEP 476 to earlier Python versions ============================================== The backporting approach described above leaves the default HTTPS certificate verification behaviour of a Python 2.7 installation unmodified: verifying certificates still needs to be opted into on a per-connection or per-process basis. To allow the default behaviour of the entire installation to be modified without breaking backwards compatibility, Red Hat designed a configuration mechanism for the system Python 2.7 installation in Red Hat Enterprise Linux 7.2+ that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the operating system version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default As it only affects backports to earlier releases of Python 2.7, this change is not proposed for inclusion in upstream CPython, but rather is offered as a recommendation to other redistributors that choose to offer a similar feature to their users. This PEP doesn't take a position on whether or not this particular change is a good idea - rather, it suggests that *if* a redistributor chooses to go down the path of making the default behaviour configurable in a version of Python older than Python 2.7.9, then maintaining a consistent approach across redistributors would be beneficial for users. However, this approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will verify HTTPS certificates by default. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is:: _cert_verification_config = '<path to configuration file>' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- As the PEP authors are not aware of any vendors providing long-term support releases targeting Windows, Mac OS X or \*BSD systems, this approach is currently only specifically defined for Linux system Python installations. The recommended configuration file name on Linux systems is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. The suggested section name is taken from the "https" URL schema passed to affected client APIs. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for this backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off installation wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to retain the default pre-2.7.9 behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Interaction with Python virtual environments -------------------------------------------- This setting is scoped by the interpreter installation and affects all Python processes using that interpreter, regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Origins of this recommendation ------------------------------ This recommendation is based on the backporting approach adopted for Red Hat Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP and described in detail in `this KnowledgeBase article <https://access.redhat.com/articles/2039753>`__. Red Hat's patches implementing this backport for Python 2.7.5 can be found in the `CentOS git repository <https://git.centos.org/commit/rpms!python.git/refs!heads!c7>`__. Recommendation for combined feature backports ============================================= If a redistributor chooses to backport the environment variable based configuration setting from this PEP to a modified Python version that also implements the configuration file based PEP 476 backport, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user or application, regardless of the installation-wide default behaviour. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for an environmental override of the default behaviour if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Copyright ========= This document has been placed into the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 6

Thank you.
by Alexander Walters Feb. 22, 2016

Feb. 22, 2016

I don't know if it is appropriate for this list, or not. I don't exactly care. As much as I might disagree with some of you... Thank you. Your work on Python has made a notable difference in how happy my life is.

4 3

Disabling changing sys.argv[0] with runpy.run_module(...alter_sys=True)
by Mike Kaplinskiy Feb. 22, 2016

Feb. 22, 2016

Hey folks, I hope this is the right list for this sort of thing (python-ideas seemed more far-fetched). For some context: there is currently a issue with pex that causes sys.modules lookups to stop working for __main__. In turns this makes unittest.run() & pkg_resources.resource_* fail. The root cause is that pex uses runpy.run_module with alter_sys=False. The fix should be to just pass alter_sys=True, but that changes sys.argv[0] and various existing pex files depend on that being the pex file. You can read more at https://github.com/pantsbuild/pex/pull/211 . Conservatively, I'd like to propose adding an argument to disable this behavior. The current behavior breaks a somewhat reasonable invariant that you can restart your program via `os.execv([sys.executable] + sys.argv)`. Moreover it might be user-friendly to add a `argv=sys.argv[1:]` argument to set & restore the full arguments to the module, where `argv=None` disables argv[0] switching. What do you think? Mike.

4 5

Python 3.2.7 and 3.3.7
by Georg Brandl Feb. 21, 2016

Feb. 21, 2016

Hi all, I'm planning to release 3.2.7 and 3.3.7 at the end of February. There will be a release candidate on Feb 20, and the final on Feb 27, if there is no holdup. These are both security (source-only) releases. 3.2.7 will be the last release from the 3.2 series. If you know of any patches that should go in, make sure to commit them in time or notify me. Thanks, Georg

1 1

Regular expression bytecode
by Jonathan Goble Feb. 19, 2016

Feb. 19, 2016

I'm new to Python's mailing lists, so please forgive me if I'm sending this to the wrong list. :) I filed http://bugs.python.org/issue26336 a few days ago, but now I think this list might be a better place to get discussion going. Basically, I'd like to see the bytecode of a compiled regex object exposed as a public (probably read-only) attribute of the object. Currently, although compiled in pure Python through modules sre_compile and sre_parse, the list of opcodes is then passed into C and copied into an array in a C struct, without being publicly exposed in any way. The only way for a user to get an internal representation of the regex is the re.DEBUG flag, which only produces an intermediate representation rather than the actual bytecode and only goes to stdout, which makes it useless for someone who wants to examine it programmatically. I'm sure others can think of other potential use cases for this, but one in particular would be that someone could write a debugger that can allow a user to step through a regex one opcode at a time to see exactly where it is failing. It would also perhaps be nice to have a public constructor for the regex object type, which would enable users to modify the bytecode and directly create a new regex object from it, similar to what is currently possible through the types.FunctionType and types.CodeType constructors. In addition to exposing the code in a public attribute, a helper module written in Python similar to the dis module (which is for Python's own bytecode) would be very helpful, allowing the code to be easily disassembled and examined at a higher level. Is this a good idea, or am I barking up the wrong tree? I think it's a great idea, but I'm open to being told this is a horrible idea. :) I welcome any and all comments both here and on the bug tracker. Jonathan Goble

3 4

Summary of Python tracker Issues
by Python tracker Feb. 19, 2016

Feb. 19, 2016

ACTIVITY SUMMARY (2016-02-12 - 2016-02-19) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5432 (+15) closed 32718 (+27) total 38150 (+42) Open issues with patches: 2375 Issues opened (35) ================== #25939: _ssl.enum_certificates() fails with ERROR_ACCESS_DENIED if pyt http://bugs.python.org/issue25939 reopened by benjamin.peterson #26351: Occasionally check for Ctrl-C in long-running operations like http://bugs.python.org/issue26351 opened by steven.daprano #26352: getpass incorrectly displays password prompt on stderr on fall http://bugs.python.org/issue26352 opened by Matt Hooks #26353: IDLE: Saving Shell should not add \n http://bugs.python.org/issue26353 opened by terry.reedy #26355: Emit major version based canonical URLs for docs http://bugs.python.org/issue26355 opened by ncoghlan #26357: asyncio.wait loses coroutine return value http://bugs.python.org/issue26357 opened by André Caron #26358: mmap.mmap.__iter__ is broken (yields bytes instead of ints) http://bugs.python.org/issue26358 opened by ztane #26359: CPython build options for out-of-the box performance http://bugs.python.org/issue26359 opened by alecsandru.patrascu #26360: Deadlock in thread.join on Python 2.7/Mac OS X 10.9 http://bugs.python.org/issue26360 opened by mark.dickinson #26362: Approved API for creating a temporary file path http://bugs.python.org/issue26362 opened by bignose #26363: __builtins__ propagation is misleading described in exec and e http://bugs.python.org/issue26363 opened by xcombelle #26366: Use “.. versionadded” over “.. versionchanged” where a http://bugs.python.org/issue26366 opened by Tony R. #26367: importlib.__import__ does not fail for invalid relative import http://bugs.python.org/issue26367 opened by mjacob #26369: doc for unicode.decode and str.encode is unnecessarily confusi http://bugs.python.org/issue26369 opened by benspiller #26370: shelve filename inconsistent between platforms http://bugs.python.org/issue26370 opened by Dima.Tisnek #26372: Popen.communicate not ignoring BrokenPipeError http://bugs.python.org/issue26372 opened by memeplex #26373: asyncio: add support for async context manager on streams? http://bugs.python.org/issue26373 opened by haypo #26374: concurrent_futures Executor.map semantics better specified in http://bugs.python.org/issue26374 opened by F.D. Sacerdoti #26375: New versions of Python hangs on imaplib.IMAP4_SSL() http://bugs.python.org/issue26375 opened by mniklas #26376: Tkinter root window won't close if packed. http://bugs.python.org/issue26376 opened by Sam Yeager #26377: Tkinter dialogs will not close if root window not packed. http://bugs.python.org/issue26377 opened by Sam Yeager #26379: zlib decompress as_bytearray flag http://bugs.python.org/issue26379 opened by llllllllll #26380: Add an http method enum http://bugs.python.org/issue26380 opened by demian.brecht #26381: Add 'geo' URI scheme (RFC 5870) to urllib.parse.uses_params http://bugs.python.org/issue26381 opened by Serhiy Int #26382: List object memory allocator http://bugs.python.org/issue26382 opened by catalin.manciu #26383: benchmarks (perf.py): number of decimal places in csv output http://bugs.python.org/issue26383 opened by florin.papa #26384: UnboundLocalError in socket._sendfile_use_sendfile http://bugs.python.org/issue26384 opened by berker.peksag #26385: the call of tempfile.NamedTemporaryFile fails and leaves a fil http://bugs.python.org/issue26385 opened by Eugene Viktorov #26386: tkinter - Treeview - .selection_add and selection_toggle http://bugs.python.org/issue26386 opened by gbarnabic #26387: Race condition in sqlite module http://bugs.python.org/issue26387 opened by scorp #26388: Disabling changing sys.argv[0] with runpy.run_module(...alter_ http://bugs.python.org/issue26388 opened by Mike Kaplinskiy #26389: Expand traceback module API to accept just an exception as an http://bugs.python.org/issue26389 opened by brett.cannon #26390: hashlib's pbkdf2_hmac documentation "rounds" does not match so http://bugs.python.org/issue26390 opened by dbakker #26391: Specialized sub-classes of Generic never call __init__ http://bugs.python.org/issue26391 opened by Kai Wohlfahrt #26392: socketserver.BaseServer.close_server should stop serve_forever http://bugs.python.org/issue26392 opened by palaviv Most recent 15 issues with no replies (15) ========================================== #26392: socketserver.BaseServer.close_server should stop serve_forever http://bugs.python.org/issue26392 #26391: Specialized sub-classes of Generic never call __init__ http://bugs.python.org/issue26391 #26389: Expand traceback module API to accept just an exception as an http://bugs.python.org/issue26389 #26386: tkinter - Treeview - .selection_add and selection_toggle http://bugs.python.org/issue26386 #26384: UnboundLocalError in socket._sendfile_use_sendfile http://bugs.python.org/issue26384 #26383: benchmarks (perf.py): number of decimal places in csv output http://bugs.python.org/issue26383 #26377: Tkinter dialogs will not close if root window not packed. http://bugs.python.org/issue26377 #26375: New versions of Python hangs on imaplib.IMAP4_SSL() http://bugs.python.org/issue26375 #26373: asyncio: add support for async context manager on streams? http://bugs.python.org/issue26373 #26363: __builtins__ propagation is misleading described in exec and e http://bugs.python.org/issue26363 #26359: CPython build options for out-of-the box performance http://bugs.python.org/issue26359 #26358: mmap.mmap.__iter__ is broken (yields bytes instead of ints) http://bugs.python.org/issue26358 #26353: IDLE: Saving Shell should not add \n http://bugs.python.org/issue26353 #26338: remove duplicate bind addresses in create_server http://bugs.python.org/issue26338 #26335: Make mmap.write return the number of bytes written like other http://bugs.python.org/issue26335 Most recent 15 issues waiting for review (15) ============================================= #26392: socketserver.BaseServer.close_server should stop serve_forever http://bugs.python.org/issue26392 #26390: hashlib's pbkdf2_hmac documentation "rounds" does not match so http://bugs.python.org/issue26390 #26387: Race condition in sqlite module http://bugs.python.org/issue26387 #26385: the call of tempfile.NamedTemporaryFile fails and leaves a fil http://bugs.python.org/issue26385 #26384: UnboundLocalError in socket._sendfile_use_sendfile http://bugs.python.org/issue26384 #26382: List object memory allocator http://bugs.python.org/issue26382 #26380: Add an http method enum http://bugs.python.org/issue26380 #26379: zlib decompress as_bytearray flag http://bugs.python.org/issue26379 #26372: Popen.communicate not ignoring BrokenPipeError http://bugs.python.org/issue26372 #26367: importlib.__import__ does not fail for invalid relative import http://bugs.python.org/issue26367 #26366: Use “.. versionadded” over “.. versionchanged” where a http://bugs.python.org/issue26366 #26359: CPython build options for out-of-the box performance http://bugs.python.org/issue26359 #26352: getpass incorrectly displays password prompt on stderr on fall http://bugs.python.org/issue26352 #26347: BoundArguments.apply_defaults doesn't handle empty arguments http://bugs.python.org/issue26347 #26342: Faster bit ops for single-digit positive longs http://bugs.python.org/issue26342 Top 10 most discussed issues (10) ================================= #15873: datetime: add ability to parse RFC 3339 dates and times http://bugs.python.org/issue15873 29 msgs #26360: Deadlock in thread.join on Python 2.7/Mac OS X 10.9 http://bugs.python.org/issue26360 17 msgs #26323: Add assert_called() and assert_called_once() methods for mock http://bugs.python.org/issue26323 10 msgs #26366: Use “.. versionadded” over “.. versionchanged” where a http://bugs.python.org/issue26366 10 msgs #26331: Tokenizer: allow underscores for grouping in numeric literals http://bugs.python.org/issue26331 9 msgs #21145: Add the @cached_property decorator http://bugs.python.org/issue21145 8 msgs #26372: Popen.communicate not ignoring BrokenPipeError http://bugs.python.org/issue26372 8 msgs #26385: the call of tempfile.NamedTemporaryFile fails and leaves a fil http://bugs.python.org/issue26385 7 msgs #26380: Add an http method enum http://bugs.python.org/issue26380 6 msgs #14597: Cannot unload dll in ctypes until script exits http://bugs.python.org/issue14597 5 msgs Issues closed (28) ================== #15608: Improve socketserver doc http://bugs.python.org/issue15608 closed by martin.panter #16915: mode of socket.makefile is more limited than documentation sug http://bugs.python.org/issue16915 closed by berker.peksag #19841: ConfigParser PEP issues http://bugs.python.org/issue19841 closed by berker.peksag #20169: random module doc page has broken links http://bugs.python.org/issue20169 closed by python-dev #23992: multiprocessing: MapResult shouldn't fail fast upon exception http://bugs.python.org/issue23992 closed by neologix #25179: PEP 498 f-strings need to be documented http://bugs.python.org/issue25179 closed by martin.panter #25713: Setuptools included with 64-bit Windows installer is outdated http://bugs.python.org/issue25713 closed by ned.deily #25833: pyvenv: venvs cannot be moved because activate scripts hard-co http://bugs.python.org/issue25833 closed by vinay.sajip #25887: awaiting on coroutine more than once should be an error http://bugs.python.org/issue25887 closed by yselivanov #25924: investigate if getaddrinfo(3) on OSX is thread-safe http://bugs.python.org/issue25924 closed by ned.deily #26265: build errors on OS X 10.11 with --enable-universalsdk http://bugs.python.org/issue26265 closed by ned.deily #26309: socketserver.BaseServer._handle_request_noblock() doesn't shut http://bugs.python.org/issue26309 closed by martin.panter #26316: Probable typo in Arg Clinic's linear_format() http://bugs.python.org/issue26316 closed by martin.panter #26318: `io.open(fd, ...).name` returns numeric fd instead of None http://bugs.python.org/issue26318 closed by terry.reedy #26319: Check recData size before unpack in zipfile http://bugs.python.org/issue26319 closed by terry.reedy #26327: IDLE: File > Save in 2.7 Shell with non-ascii fails http://bugs.python.org/issue26327 closed by terry.reedy #26333: Multiprocessing imap hangs when generator input errors http://bugs.python.org/issue26333 closed by terry.reedy #26334: bytes.translate() doesn't take keyword arguments; docs suggest http://bugs.python.org/issue26334 closed by Nicholas Chammas #26348: activate.fish sets VENV prompt incorrectly http://bugs.python.org/issue26348 closed by vinay.sajip #26349: Ship python35.lib with the embedded distribution, please http://bugs.python.org/issue26349 closed by steve.dower #26354: re.I does not work as expected http://bugs.python.org/issue26354 closed by ezio.melotti #26356: Registration http://bugs.python.org/issue26356 closed by SilentGhost #26361: lambda in dict comprehension is broken http://bugs.python.org/issue26361 closed by steven.daprano #26364: pip uses colour in messages that does not work on white termin http://bugs.python.org/issue26364 closed by berker.peksag #26365: ntpath.py Error in Windows http://bugs.python.org/issue26365 closed by serhiy.storchaka #26368: grammatical error in asyncio stream documentation http://bugs.python.org/issue26368 closed by ned.deily #26371: asynchat.async_chat and asyncore.dispatcher_with_send are not http://bugs.python.org/issue26371 closed by gvanrossum #26378: Typo in regex documentation http://bugs.python.org/issue26378 closed by python-dev

1 0

RE 25939 - _ssl.enum_certificates broken on Windows
by Dave Hirschfeld Feb. 18, 2016

Feb. 18, 2016

I've run into issue 25939 (https://bugs.python.org/issue25939) when trying to deploy a python webapp with IIS on Windows. This issue is preventing us from deploying the app to production as the workaround AFAICT requires running the app under an admin account. Apologies if this is an inappropriate forum for a +1 but I just wanted to let the devs know that this is an issue which affects the use of Python (on Windows) in the enterprise. I noticed that the patch hasn't been merged yet so was interested in making sure it didn't fall by the wayside... As a mere user I don't expect the devs to prioritize my own problems which no doubt only affect a very small number of python users but I would be very grateful if the patch did make it into a minor release. Regards, Dave

4 4

When does `PyType_Type.tp_alloc get assigned to PyType_GenericAlloc ?
by Randy Eels Feb. 17, 2016

Feb. 17, 2016

Hi everyone, I've a question about the implementation of the `type` builtin (in Python 3.5). In Objects/typeobject.c, the `tp_alloc` slot of PyType_Type gets set to 0. However, I can see (using gdb) that it later gets assigned to `&PyType_GenericAlloc`. I'd argue that this makes sense because, in `type_new`, there is a line where that member function gets called without previously checking whether that member points to something: ``` /* Allocate the type object */ type = (PyTypeObject *)metatype->tp_alloc(metatype, nslots); ``` Yet, I can't seem to understand where and when does the `tp_alloc` slot of PyType_Type get re-assigned to PyType_GenericAlloc. Does that even happen? Or am I missing something bigger? And, just out of further curiosity, why doesn't the aforementioned slot get initialised to `PyType_GenericAlloc` in the first place? Thanks a lot.

3 3

Buffer overflow bug in GNU C's getaddrinfo()
by MRAB Feb. 17, 2016

Feb. 17, 2016

Is this something that we need to worry about? Extremely severe bug leaves dizzying number of software and devices vulnerable http://arstechnica.com/security/2016/02/extremely-severe-bug-leaves-dizzyin…

4 3

Hash randomization for which types?
by Christoph Groth Feb. 17, 2016

Feb. 17, 2016

Hello, Recent Python versions randomize the hashes of str, bytes and datetime objects. I suppose that the choice of these three types is the result of a compromise. Has this been discussed somewhere publicly? I'm not a web programmer, but don't web applications also use dictionaries that are indexed by, say, tuples of integers? Just curious... Thanks, Christoph

8 11

CPython build options for out-of-the box performance
by Patrascu, Alecsandru Feb. 14, 2016

Feb. 14, 2016

Hi all, This is Alecsandru from the Dynamic Scripting Languages Optimization Team at Intel Corporation. I want to open a discussion regarding the way CPython is built, mainly the options that are available to the programmers. Analyzing the CPython ecosystem we can see that there are a lot of users that just download the sources and hit the commands "./configure", "make" and "make install" once and then continue using it with their Python scripts. One of the problems with this workflow it that the users do not benefit from the entire optimization features that are existing in the build system, such as PGO and LTO. Therefore, I propose a workflow, like the following. Assume some work has to be done into the CPython interpreter, a developer can do the following steps: A. Implementation and debugging phase. 1. The command "./configure PYDIST=debug" is ran once. It will enable the Py_DEBUG, -O0 and -g flags 2. The command "make" is ran once or multiple times B. Testing the implementation from step A, in a pre-release environment 1. The command "./configure PYDIST=devel" is ran once. It will disable the Py_DEBUG flags and will enable the -O3 and -g flags, and it is just like the current implementation in CPython 2. The command "make" is ran once or multiple times C. For any other CPython usage, for example distributing the interpreter, installing it inside an operating system, or just the majority of users who are not CPython developers and only want to compile it once and use it as-is: 1. The command "./configure" is ran once. Alternatively, the command "./configure PYDIST=release" can be used. It will disable all debugging functionality, enable the -O3 flag and will enable PGO and LTO. 2. The command "make" is ran once If you think this benefits CPython, I can create an issue and post the patches that enable all of the above. Thank you, Alecsandru

1 1

PEP 515: Underscores in Numeric Literals
by Georg Brandl Feb. 12, 2016

Feb. 12, 2016

This came up in python-ideas, and has met mostly positive comments, although the exact syntax rules are up for discussion. cheers, Georg -------------------------------------------------------------------------------- PEP: 515 Title: Underscores in Numeric Literals Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2016 Python-Version: 3.6 Abstract and Rationale ====================== This PEP proposes to extend Python's syntax so that underscores can be used in integral and floating-point number literals. This is a common feature of other modern languages, and can aid readability of long literals, or literals whose value should clearly separate into parts, such as bytes or words in hexadecimal notation. Examples:: # grouping decimal numbers by thousands amount = 10_000_000.0 # grouping hexadecimal addresses by words addr = 0xDEAD_BEEF # grouping bits into bytes in a binary literal flags = 0b_0011_1111_0100_1110 Specification ============= The current proposal is to allow underscores anywhere in numeric literals, with these exceptions: * Leading underscores cannot be allowed, since they already introduce identifiers. * Trailing underscores are not allowed, because they look confusing and don't contribute much to readability. * The number base prefixes ``0x``, ``0o``, and ``0b`` cannot be split up, because they are fixed strings and not logically part of the number. * No underscore allowed after a sign in an exponent (``1e-_5``), because underscores can also not be used after the signs in front of the number (``-1e5``). * No underscore allowed after a decimal point, because this leads to ambiguity with attribute access (the lexer cannot know that there is no number literal in ``foo._5``). There appears to be no reason to restrict the use of underscores otherwise. The production list for integer literals would therefore look like this:: integer: decimalinteger | octinteger | hexinteger | bininteger decimalinteger: nonzerodigit [decimalrest] | "0" [("0" | "_")* "0"] nonzerodigit: "1"..."9" decimalrest: (digit | "_")* digit digit: "0"..."9" octinteger: "0" ("o" | "O") (octdigit | "_")* octdigit hexinteger: "0" ("x" | "X") (hexdigit | "_")* hexdigit bininteger: "0" ("b" | "B") (bindigit | "_")* bindigit octdigit: "0"..."7" hexdigit: digit | "a"..."f" | "A"..."F" bindigit: "0" | "1" For floating-point literals:: floatnumber: pointfloat | exponentfloat pointfloat: [intpart] fraction | intpart "." exponentfloat: (intpart | pointfloat) exponent intpart: digit (digit | "_")* fraction: "." intpart exponent: ("e" | "E") "_"* ["+" | "-"] digit [decimalrest] Alternative Syntax ================== Underscore Placement Rules -------------------------- Instead of the liberal rule specified above, the use of underscores could be limited. Common rules are (see the "other languages" section): * Only one consecutive underscore allowed, and only between digits. * Multiple consecutive underscore allowed, but only between digits. Different Separators -------------------- A proposed alternate syntax was to use whitespace for grouping. Although strings are a precedent for combining adjoining literals, the behavior can lead to unexpected effects which are not possible with underscores. Also, no other language is known to use this rule, except for languages that generally disregard any whitespace. C++14 introduces apostrophes for grouping, which is not considered due to the conflict with Python's string literals. [1]_ Behavior in Other Languages =========================== Those languages that do allow underscore grouping implement a large variety of rules for allowed placement of underscores. This is a listing placing the known rules into three major groups. In cases where the language spec contradicts the actual behavior, the actual behavior is listed. **Group 1: liberal (like this PEP)** * D [2]_ * Perl 5 (although docs say it's more restricted) [3]_ * Rust [4]_ * Swift (although textual description says "between digits") [5]_ **Group 2: only between digits, multiple consecutive underscores** * C# (open proposal for 7.0) [6]_ * Java [7]_ **Group 3: only between digits, only one underscore** * Ada [8]_ * Julia (but not in the exponent part of floats) [9]_ * Ruby (docs say "anywhere", in reality only between digits) [10]_ Implementation ============== A preliminary patch that implements the specification given above has been posted to the issue tracker. [11]_ References ========== .. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3499.html .. [2] http://dlang.org/spec/lex.html#integerliteral .. [3] http://perldoc.perl.org/perldata.html#Scalar-value-constructors .. [4] http://doc.rust-lang.org/reference.html#number-literals .. [5] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… .. [6] https://github.com/dotnet/roslyn/issues/216 .. [7] https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores… .. [8] http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html#2.4 .. [9] http://docs.julialang.org/en/release-0.4/manual/integers-and-floating-point… .. [10] http://ruby-doc.org/core-2.3.0/doc/syntax/literals_rdoc.html#label-Numbers .. [11] http://bugs.python.org/issue26331 Copyright ========= This document has been placed in the public domain.

22 59

Summary of Python tracker Issues
by Python tracker Feb. 12, 2016

Feb. 12, 2016

ACTIVITY SUMMARY (2016-02-05 - 2016-02-12) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5417 ( +4) closed 32691 (+50) total 38108 (+54) Open issues with patches: 2368 Issues opened (39) ================== #22107: tempfile module misinterprets access denied error on Windows http://bugs.python.org/issue22107 reopened by serhiy.storchaka #25911: Regression: os.walk now using os.scandir() breaks bytes filena http://bugs.python.org/issue25911 reopened by serhiy.storchaka #26299: wsgiref.util FileWrapper raises ValueError: I/O operation on c http://bugs.python.org/issue26299 opened by samwyse #26300: "unpacked" bytecode http://bugs.python.org/issue26300 opened by abarnert #26302: cookies module allows commas in keys http://bugs.python.org/issue26302 opened by jason.coombs #26303: Shared execution context between doctests in a module http://bugs.python.org/issue26303 opened by kernc #26305: Make Argument Clinic to generate PEP 7 conforming code http://bugs.python.org/issue26305 opened by serhiy.storchaka #26306: Can't create abstract tuple http://bugs.python.org/issue26306 opened by Jack Hargreaves #26307: no PGO for built-in modules with `make profile-opt` http://bugs.python.org/issue26307 opened by tzot #26308: Solaris 10 build issues http://bugs.python.org/issue26308 opened by gms #26309: socketserver.BaseServer._handle_request_noblock() don't shutdw http://bugs.python.org/issue26309 opened by palaviv #26313: ssl.py _load_windows_store_certs fails if windows cert store i http://bugs.python.org/issue26313 opened by Jonathan Kamens #26314: interned strings are stored in a dict, a set would use less me http://bugs.python.org/issue26314 opened by gregory.p.smith #26316: Probable typo in Arg Clinic’s linear_format() http://bugs.python.org/issue26316 opened by martin.panter #26317: Build Problem with GCC + Macintosh OS X 10.11 El Capitain http://bugs.python.org/issue26317 opened by Robert P Fischer #26318: `io.open(fd, ...).name` returns numeric fd instead of None http://bugs.python.org/issue26318 opened by mmarkk #26319: Check recData size before unpack in zipfile http://bugs.python.org/issue26319 opened by j w #26322: Missing docs for typing.Set http://bugs.python.org/issue26322 opened by gvanrossum #26323: Add a assert_called() method for mock objects http://bugs.python.org/issue26323 opened by Amit.Saha #26326: Named entity "vertical line" missed in 2.7 htmlentitydefs.py http://bugs.python.org/issue26326 opened by andreas.roehler #26327: File > Save in IDLE shell window not working http://bugs.python.org/issue26327 opened by xflr6 #26328: shutil._copyxattr() function shouldn't fail if setting securit http://bugs.python.org/issue26328 opened by bigon #26329: os.path.normpath("//") returns // http://bugs.python.org/issue26329 opened by Fred Rolland #26330: shutil.disk_usage() on Windows can't properly handle unicode http://bugs.python.org/issue26330 opened by giampaolo.rodola #26331: Tokenizer: allow underscores for grouping in numeric literals http://bugs.python.org/issue26331 opened by georg.brandl #26332: OSError: exception: access violation writing <...> (Windows 10 http://bugs.python.org/issue26332 opened by jk #26333: Multiprocessing imap hangs when generator input errors http://bugs.python.org/issue26333 opened by Aaron Halfaker #26334: bytes.translate() doesn't take keyword arguments; docs suggest http://bugs.python.org/issue26334 opened by Nicholas Chammas #26335: Make mmap.write return the number of bytes written like other http://bugs.python.org/issue26335 opened by jstasiak #26336: Expose regex bytecode as attribute of compiled pattern object http://bugs.python.org/issue26336 opened by Jonathan Goble #26337: Bypass imghdr module determines the type of image http://bugs.python.org/issue26337 opened by Ramin Farajpour Cami #26338: remove duplicate bind addresses in create_server http://bugs.python.org/issue26338 opened by sebastien.bourdeauducq #26340: modal dialog with transient method; parent window fails to ico http://bugs.python.org/issue26340 opened by vs #26342: Faster bit ops for single-digit positive longs http://bugs.python.org/issue26342 opened by yselivanov #26346: PySequenceMethods documentation missing sq_slice and sq_ass_sl http://bugs.python.org/issue26346 opened by atuining #26347: BoundArguments.apply_defaults doesn't handle empty arguments http://bugs.python.org/issue26347 opened by Frederick Wagner #26348: activate.fish sets VENV prompt incorrectly http://bugs.python.org/issue26348 opened by Dan McCombs #26349: Ship python35.lib with the embedded distribution, please http://bugs.python.org/issue26349 opened by Thomas Führinger #26350: Windoes: signal doc should state certains signals can't be reg http://bugs.python.org/issue26350 opened by giampaolo.rodola Most recent 15 issues with no replies (15) ========================================== #26349: Ship python35.lib with the embedded distribution, please http://bugs.python.org/issue26349 #26348: activate.fish sets VENV prompt incorrectly http://bugs.python.org/issue26348 #26340: modal dialog with transient method; parent window fails to ico http://bugs.python.org/issue26340 #26338: remove duplicate bind addresses in create_server http://bugs.python.org/issue26338 #26336: Expose regex bytecode as attribute of compiled pattern object http://bugs.python.org/issue26336 #26335: Make mmap.write return the number of bytes written like other http://bugs.python.org/issue26335 #26333: Multiprocessing imap hangs when generator input errors http://bugs.python.org/issue26333 #26332: OSError: exception: access violation writing <...> (Windows 10 http://bugs.python.org/issue26332 #26327: File > Save in IDLE shell window not working http://bugs.python.org/issue26327 #26322: Missing docs for typing.Set http://bugs.python.org/issue26322 #26319: Check recData size before unpack in zipfile http://bugs.python.org/issue26319 #26313: ssl.py _load_windows_store_certs fails if windows cert store i http://bugs.python.org/issue26313 #26308: Solaris 10 build issues http://bugs.python.org/issue26308 #26306: Can't create abstract tuple http://bugs.python.org/issue26306 #26299: wsgiref.util FileWrapper raises ValueError: I/O operation on c http://bugs.python.org/issue26299 Most recent 15 issues waiting for review (15) ============================================= #26348: activate.fish sets VENV prompt incorrectly http://bugs.python.org/issue26348 #26347: BoundArguments.apply_defaults doesn't handle empty arguments http://bugs.python.org/issue26347 #26342: Faster bit ops for single-digit positive longs http://bugs.python.org/issue26342 #26338: remove duplicate bind addresses in create_server http://bugs.python.org/issue26338 #26335: Make mmap.write return the number of bytes written like other http://bugs.python.org/issue26335 #26331: Tokenizer: allow underscores for grouping in numeric literals http://bugs.python.org/issue26331 #26323: Add a assert_called() method for mock objects http://bugs.python.org/issue26323 #26314: interned strings are stored in a dict, a set would use less me http://bugs.python.org/issue26314 #26309: socketserver.BaseServer._handle_request_noblock() don't shutdw http://bugs.python.org/issue26309 #26308: Solaris 10 build issues http://bugs.python.org/issue26308 #26305: Make Argument Clinic to generate PEP 7 conforming code http://bugs.python.org/issue26305 #26302: cookies module allows commas in keys http://bugs.python.org/issue26302 #26285: Garbage collection of unused input sections from CPython binar http://bugs.python.org/issue26285 #26282: Add support for partial keyword arguments in extension functio http://bugs.python.org/issue26282 #26280: ceval: Optimize list[int] (subscript) operation similarly to C http://bugs.python.org/issue26280 Top 10 most discussed issues (10) ================================= #21955: ceval.c: implement fast path for integers with a single digit http://bugs.python.org/issue21955 28 msgs #26302: cookies module allows commas in keys http://bugs.python.org/issue26302 14 msgs #26200: SETREF adds unnecessary work in some cases http://bugs.python.org/issue26200 11 msgs #26331: Tokenizer: allow underscores for grouping in numeric literals http://bugs.python.org/issue26331 11 msgs #24165: Free list for single-digits ints http://bugs.python.org/issue24165 10 msgs #26182: Deprecation warnings for the future async and await keywords i http://bugs.python.org/issue26182 6 msgs #24916: In sysconfig, don't rely on sys.version format http://bugs.python.org/issue24916 5 msgs #26330: shutil.disk_usage() on Windows can't properly handle unicode http://bugs.python.org/issue26330 5 msgs #25195: mock.ANY doesn't match mock.MagicMock() object http://bugs.python.org/issue25195 4 msgs #25911: Regression: os.walk now using os.scandir() breaks bytes filena http://bugs.python.org/issue25911 4 msgs Issues closed (47) ================== #15731: Mechanism for inheriting docstrings and signatures http://bugs.python.org/issue15731 closed by ncoghlan #16023: IDLE freezes on ^5 or ^6 (Un-)Tabify Region with OS X Cocoa Tk http://bugs.python.org/issue16023 closed by ned.deily #19543: Add -3 warnings for codec convenience method changes http://bugs.python.org/issue19543 closed by ncoghlan #22983: Cookie parsing should be more permissive http://bugs.python.org/issue22983 closed by martin.panter #25226: "suffix" attribute not documented in logging.TimedRotatingFile http://bugs.python.org/issue25226 closed by vinay.sajip #25295: functools.lru_cache raises KeyError http://bugs.python.org/issue25295 closed by Peter Brady #25639: open 'PhysicalDriveN' on windows fails (since python 3.5) with http://bugs.python.org/issue25639 closed by eryksun #25698: The copy_reg module becomes unexpectedly empty in test_cpickle http://bugs.python.org/issue25698 closed by serhiy.storchaka #25709: Problem with string concatenation and utf-8 cache. http://bugs.python.org/issue25709 closed by georg.brandl #25848: Tkinter tests failed on Windows buildbots http://bugs.python.org/issue25848 closed by zach.ware #25949: Lazy creation of __dict__ in OrderedDict http://bugs.python.org/issue25949 closed by serhiy.storchaka #25983: Add tests for multi-argument type() http://bugs.python.org/issue25983 closed by serhiy.storchaka #25985: Use sys.version_info instead of sys.version http://bugs.python.org/issue25985 closed by serhiy.storchaka #25992: test_gdb fails on OSX http://bugs.python.org/issue25992 closed by ned.deily #25994: File descriptor leaks in os.scandir() http://bugs.python.org/issue25994 closed by serhiy.storchaka #25995: os.walk() consumes a lot of file descriptors http://bugs.python.org/issue25995 closed by serhiy.storchaka #26045: Improve error message for http.client when posting unicode str http://bugs.python.org/issue26045 closed by martin.panter #26086: Bug in os module http://bugs.python.org/issue26086 closed by steven.daprano #26117: Close directory descriptor in scandir iterator on error http://bugs.python.org/issue26117 closed by serhiy.storchaka #26136: DeprecationWarning for PEP 479 (generator_stop) http://bugs.python.org/issue26136 closed by martin.panter #26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow http://bugs.python.org/issue26198 closed by serhiy.storchaka #26204: compiler: ignore constants used as statements (don't emit LOAD http://bugs.python.org/issue26204 closed by haypo #26223: decimal.to_eng_string() does not implement engineering notatio http://bugs.python.org/issue26223 closed by rhettinger #26243: zlib.compress level as keyword argument http://bugs.python.org/issue26243 closed by martin.panter #26248: Improve scandir DirEntry docs, especially re symlinks and cach http://bugs.python.org/issue26248 closed by gvanrossum #26279: time.strptime does not properly convert out-of-bounds values http://bugs.python.org/issue26279 closed by ned.deily #26287: Core dump in f-string with formatting errors http://bugs.python.org/issue26287 closed by eric.smith #26288: Optimize PyLong_AsDouble for single-digit longs http://bugs.python.org/issue26288 closed by yselivanov #26289: Optimize floor division for ints http://bugs.python.org/issue26289 closed by yselivanov #26294: Queue().unfinished_tasks not in docs - deliberate? http://bugs.python.org/issue26294 closed by rhettinger #26297: Move constant folding to AST level http://bugs.python.org/issue26297 closed by serhiy.storchaka #26298: Split ceval.c into small files http://bugs.python.org/issue26298 closed by haypo #26301: ceval.c: reintroduce fast-path for list[index] in BINARY_SUBSC http://bugs.python.org/issue26301 closed by haypo #26304: Fix “allows to <verb>” in documentation http://bugs.python.org/issue26304 closed by martin.panter #26310: Fix typo “variariables” in socketserver.py http://bugs.python.org/issue26310 closed by martin.panter #26311: Typo in documentation for xml.parsers.expat http://bugs.python.org/issue26311 closed by martin.panter #26312: Raise SystemError on programmical errors in PyArg_Parse*() http://bugs.python.org/issue26312 closed by serhiy.storchaka #26315: Optimize mod division for ints http://bugs.python.org/issue26315 closed by yselivanov #26320: Web documentation for 2.7 has unreadable highlights in Table o http://bugs.python.org/issue26320 closed by python-dev #26321: datetime.strptime fails to parse AM/PM correctly http://bugs.python.org/issue26321 closed by zach.ware #26324: sum() incorrect on negative zeros http://bugs.python.org/issue26324 closed by Jim.Jewett #26325: Add helper to check that no ResourceWarning is emitted http://bugs.python.org/issue26325 closed by serhiy.storchaka #26339: Python rk0.3b1 KeyError: 'config_argparse_rel_path' http://bugs.python.org/issue26339 closed by berker.peksag #26341: Implement free-list for single-digit longs http://bugs.python.org/issue26341 closed by yselivanov #26343: os.O_CLOEXEC not available on OS X http://bugs.python.org/issue26343 closed by ned.deily #26344: `sys.meta_path` Skipped for Packages with Non-Standard Suffixe http://bugs.python.org/issue26344 closed by brett.cannon #26345: Extra newline appended to UTF-8 strings on Windows http://bugs.python.org/issue26345 closed by eryksun

1 0

[ANN] MicroPython 1.6
by Paul Sokolovsky Feb. 12, 2016

Feb. 12, 2016

Hello, MicroPython is a lean and efficient Python implementation for microcontrollers, embedded, and mobile systems (which also runs just as fine on desktops, servers, and clouds). https://github.com/micropython/micropython https://github.com/micropython/micropython/releases/tag/v1.6 There're following major changes since 1.5: 1. LwIP module support for embedded TCP/IP networking. 2. IPv6 support in the Unix port. 3. Beta support for persistent bytecode (similar to CPython's .pyc) 4. 64-bit NaN boxing (improved floating-point performance if enabled). 5. Support for new official PyBoards PYBv1.1 and PYBLITEv1.0. 6. Long int constant folding during bytecode compilation (glad that CPython will catch up in that area thanks to FAT Python project). 7. There's a ongoing crowdfunding campaign to fund complete and well-maintained MicroPython port to ubiquitous ESP8266 WiFi SoC, and improve networking and IoT support in MicroPython in general: https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-b… -- Best regards, Paul mailto:pmiscml@gmail.com

1 0

Py_SETREF again
by Serhiy Storchaka Feb. 12, 2016

Feb. 12, 2016

Sorry to bringing this up again. I was hoping we were done with that. When discussing the name of the Py_SETREF macro I was supposed to add a pair of macros: for Py_DECREF and Py_XDECREF. But I got a lot of opinions to be limited to only one macro. On 28.02.14 15:58, Kristján Valur Jónsson wrote: > Also, for the equivalence to hold there is no separate Py_XSETREF, the X > behaviour is implied, which I favour. Enough of this X-proliferation > already! On 16.12.15 16:53, Random832 wrote: > I think "SET" names imply that it's safe if the original > reference is NULL. This isn't an objection to the names, but if > it is given one of those names I think it should use Py_XDECREF. It was my initial intension. But then I had got a number of voices for single macros. On 16.12.15 23:16, Victor Stinner wrote: > I would prefer a single macro to avoid bugs, I don't think that such > macro has a critical impact on performances. It's more designed for > safety, no? On 17.12.15 08:22, Nick Coghlan wrote: >> 1. Py_SETREF > > +1 if it always uses Py_XDECREF on the previous value (as I'd expect > this to work even if the previous value was NULL) There was no (besides my) clearly expressed vote for two macros. As a result I have replaced both Py_DECREF and Py_XDECREF with the macro that always uses Py_XDECREF. Now Raymond, who was not involved in the previous discussions, expressed the view that we should to rename Py_SETREF to Py_XSETREF and add new Py_SETREF that uses Py_DECREF for using in the code that used Py_DECREF previously. [1] We should discuss the need for this, and may be re-discuss the names for the macros. [1] http://bugs.python.org/issue26200

3 3

Windows: Remove support of bytes filenames in the os module?
by Victor Stinner Feb. 11, 2016

Feb. 11, 2016

Hi, Since 3.3, functions of the os module started to emit DeprecationWarning when called with bytes filenames. The rationale is quite simple: Windows native type for filenames is Unicode, and the Windows has a weird behaviour when you use bytes. For example, os.listdir(b'.') gives you paths which cannot be used with open() on filenames which are not encodable the ANSI code page. Unencodable characters are replaced with "?". The following issue was opened to document this weird behaviour (but the doc was never completed): "Document that bytes OS API can returns unusable results on Windows" http://bugs.python.org/issue16700 When the new os.scandir() API was designed, I asked to *not* support bytes filenames since they are "broken by design". https://www.python.org/dev/peps/pep-0471/ Recently, an user complained that os.walk() doesn't work with bytes on Windows anymore: "Regression: os.walk now using os.scandir() breaks bytes filenames on windows" http://bugs.python.org/issue25911 Serhiy Storchaka just pushed a change to reintroduce support bytes support on Windows in os.walk(), but I would prefer to do the *opposite*: drop supports for bytes filenames on Windows. Are we brave enough to force users to use the "right" type for filenames? -- On Python 2, it wasn't possible to use Unicode for filenames, many functions fail badly with Unicode, especially when you mix bytes and Unicode. On Python 3, Unicode is the "natural" types, most Python functions prefer Unicode, and the PEP 383 (surrogateescape) allows to safetely use Unicode on UNIX even with undecodable filenames (invalid bytes are stored as Unicode surrogate characters). Victor

14 39

PEP 515: Underscores in Numeric Literals
by Georg Brandl Feb. 11, 2016

Feb. 11, 2016

Hey all, based on the feedback so far, I revised the PEP. There is now a much simpler rule for allowed underscores, with no exceptions. This made the grammar simpler as well. --------------------------------------------------------------------------- PEP: 515 Title: Underscores in Numeric Literals Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-Feb-2016 Python-Version: 3.6 Abstract and Rationale ====================== This PEP proposes to extend Python's syntax so that underscores can be used in integral, floating-point and complex number literals. This is a common feature of other modern languages, and can aid readability of long literals, or literals whose value should clearly separate into parts, such as bytes or words in hexadecimal notation. Examples:: # grouping decimal numbers by thousands amount = 10_000_000.0 # grouping hexadecimal addresses by words addr = 0xDEAD_BEEF # grouping bits into bytes in a binary literal flags = 0b_0011_1111_0100_1110 # making the literal suffix stand out more imag = 1.247812376e-15_j Specification ============= The current proposal is to allow one or more consecutive underscores following digits and base specifiers in numeric literals. The production list for integer literals would therefore look like this:: integer: decimalinteger | octinteger | hexinteger | bininteger decimalinteger: nonzerodigit (digit | "_")* | "0" ("0" | "_")* nonzerodigit: "1"..."9" digit: "0"..."9" octinteger: "0" ("o" | "O") "_"* octdigit (octdigit | "_")* hexinteger: "0" ("x" | "X") "_"* hexdigit (hexdigit | "_")* bininteger: "0" ("b" | "B") "_"* bindigit (bindigit | "_")* octdigit: "0"..."7" hexdigit: digit | "a"..."f" | "A"..."F" bindigit: "0" | "1" For floating-point and complex literals:: floatnumber: pointfloat | exponentfloat pointfloat: [intpart] fraction | intpart "." exponentfloat: (intpart | pointfloat) exponent intpart: digit (digit | "_")* fraction: "." intpart exponent: ("e" | "E") ["+" | "-"] intpart imagnumber: (floatnumber | intpart) ("j" | "J") Alternative Syntax ================== Underscore Placement Rules -------------------------- Instead of the liberal rule specified above, the use of underscores could be limited. Common rules are (see the "other languages" section): * Only one consecutive underscore allowed, and only between digits. * Multiple consecutive underscore allowed, but only between digits. A less common rule would be to allow underscores only every N digits (where N could be 3 for decimal literals, or 4 for hexadecimal ones). This is unnecessarily restrictive, especially considering the separator placement is different in different cultures. Different Separators -------------------- A proposed alternate syntax was to use whitespace for grouping. Although strings are a precedent for combining adjoining literals, the behavior can lead to unexpected effects which are not possible with underscores. Also, no other language is known to use this rule, except for languages that generally disregard any whitespace. C++14 introduces apostrophes for grouping, which is not considered due to the conflict with Python's string literals. [1]_ Behavior in Other Languages =========================== Those languages that do allow underscore grouping implement a large variety of rules for allowed placement of underscores. This is a listing placing the known rules into three major groups. In cases where the language spec contradicts the actual behavior, the actual behavior is listed. **Group 1: liberal** This group is the least homogeneous: the rules vary slightly between languages. All of them allow trailing underscores. Some allow underscores after non-digits like the ``e`` or the sign in exponents. * D [2]_ * Perl 5 (underscores basically allowed anywhere, although docs say it's more restricted) [3]_ * Rust (allows between exponent sign and digits) [4]_ * Swift (although textual description says "between digits") [5]_ **Group 2: only between digits, multiple consecutive underscores** * C# (open proposal for 7.0) [6]_ * Java [7]_ **Group 3: only between digits, only one underscore** * Ada [8]_ * Julia (but not in the exponent part of floats) [9]_ * Ruby (docs say "anywhere", in reality only between digits) [10]_ Implementation ============== A preliminary patch that implements the specification given above has been posted to the issue tracker. [11]_ Open Questions ============== This PEP currently only proposes changing the literal syntax. The following extensions are open for discussion: * Allowing underscores in string arguments to the ``Decimal`` constructor. It could be argued that these are akin to literals, since there is no Decimal literal available (yet). * Allowing underscores in string arguments to ``int()`` with base argument 0, ``float()`` and ``complex()``. References ========== .. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3499.html .. [2] http://dlang.org/spec/lex.html#integerliteral .. [3] http://perldoc.perl.org/perldata.html#Scalar-value-constructors .. [4] http://doc.rust-lang.org/reference.html#number-literals .. [5] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swif… .. [6] https://github.com/dotnet/roslyn/issues/216 .. [7] https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores… .. [8] http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html#2.4 .. [9] http://docs.julialang.org/en/release-0.4/manual/integers-and-floating-point… .. [10] http://ruby-doc.org/core-2.3.0/doc/syntax/literals_rdoc.html#label-Numbers .. [11] http://bugs.python.org/issue26331 Copyright ========= This document has been placed in the public domain.

11 13

Re: [Python-Dev] fullOfEels, assistant program for writing Python extension modules in C
by Hugh Fisher Feb. 11, 2016

Feb. 11, 2016

On Fri, Feb 12, 2016 at 3:30 AM, Nathaniel Smith <njs(a)pobox.com> wrote: > You're almost certainly aware of this, but just to double check since you > don't mention it in the email: cython is also a great tool for handling > similar situations. Not quite the same since in addition to generating all > the boilerplate for you it then lets you use almost-python to actually write > the C implementations as well, and I understand that with your tool you > write the actual implementations in C. But probably also worth considering > in cases where you'd consider this tool, so wanted to make sure it was on > your radar. Yes, cython is a fine tool and I wouldn't try to dissuade anyone from using it if it works for them. FullOfEels is for when the implementation should be hidden altogether. Most often this is because of cross-platform differences or coding horrors, but could also be handy for teaching when it's easier to just give students plain Python modules to look at. Thanks for replying. -- cheers, Hugh Fisher

1 0

fullOfEels, assistant program for writing Python extension modules in C
by Hugh Fisher Feb. 11, 2016

Feb. 11, 2016

I've written a Python program named fullOfEels to speed up the first stages of writing Python extension modules in C. It is not a replacement for SWIG, SIP, or ctypes. It's for the case where you want to work in the opposite direction, specifying a Python API and then writing an implementation in C. (A small niche maybe, but I hope it isn't just me who sometimes works this way.) The input is a Python module specifying what it should do but not how, with all the functions, classes, and methods being just pass. The output is a pair of .h and .c files with all the boilerplate C code required: module initialization, class type structs, C method functions and method tables. Downloadable from https://bitbucket.org/hugh_fisher/fullofeels All feedback and suggestions welcome. -- cheers, Hugh Fisher

2 1

why we have both re.match and re.string?
by Luca Sangiacomo Feb. 10, 2016

Feb. 10, 2016

Hi, I hope the question is not too silly, but why I would like to understand the advantages of having both re.match() and re.search(). Wouldn't be more clear to have just one function with one additional parameters like this: re.search(regexp, text, from_beginning=True|False) ? In this way we prevent, as written in the documentation, people writing ".*" in front of the regexp used with re.match() Thanks.

3 2

Experiences with Creating PEP 484 Stub Files
by Phil Thompson Feb. 10, 2016

Feb. 10, 2016

I've been adding support to the SIP wrapper generator for automatically generating PEP 484 compatible stub files so that future versions of PyQt can be shipped with them. By way of feedback I thought I'd share my experience, confusions and suggestions. There are a number of things I'd like to express but cannot find a way to do so... - objects that implement the buffer protocol - type objects - slice objects - capsules - sequences of fixed size (ie. specified in the same way as Tuple) - distinguishing between instance and class attributes. The documentation is incomplete - there is no mention of Set or Tuple for example. I found the documentation confusing regarding Optional. Intuitively it seems to be the way to specify arguments with default values. However it is explained in terms of (for example) Union[str, None] and I (intuitively but incorrectly) read that as meaning "a str or None" as opposed to "a str or nothing". bytes can be used as shorthand for bytes, bytearray and memoryview - but what about objects that really only support bytes? Shouldn't the shorthand be called something like AnyBytes? Is there any recommended way to test the validity and completeness of stub files? What's the recommended way to parse them? Phil

5 10

Re: [Python-Dev] Windows: Remove support of bytes filenames in the os module?
by Victor Stinner Feb. 10, 2016

Feb. 10, 2016

Le mercredi 10 février 2016, Steve Dower <python(a)stevedower.id.au> a écrit : > > I really don't like the idea of not being able to use bytes in cross > platform code. Unless it's become feasible to use Unicode for lossless > filenames on Linux - last I heard it wasn't. > The point of my email is that even on Python 3, users kept bad habits because of Python 2. *Yes*, you can use Unicode filenames on all platforms on Python 3 since 2009 thanks to the following PEP: https://www.python.org/dev/peps/pep-0383/ In my first email, I mentioned a bug report of an user still using bytes filenames on Windows with Python 3. It is on the Blender project which *only* supports Python 3. Or maybe I missed something huge which really force Blender to use bytes??? But if a few functions still require bytes, I would suggest to use instead os.fsencode() for them. It's more much convenient to handle filenames as Unicode on Python 3. Victor

1 0

Issue #26204: compiler now emits a SyntaxWarning on constant statement
by Victor Stinner Feb. 9, 2016

Feb. 9, 2016

Hi, I changed the Python compiler to ignore any kind "constant expressions", whereas it only ignored strings and integers before: http://bugs.python.org/issue26204 The compiler now also emits a SyntaxWarning on such case. IMHO the warning can help to detect bugs for developers who just learnt Python. The warning is *not* emited for strings, since triple quoted strings are a common syntax for multiline comments. The warning is *not* emited neither for ellispis (...) since "f(): ..." is a legit syntax for abstract function. Are you ok with the new warning? New behaviour: haypo@smithers$ ./python Python 3.6.0a0 (default:759a975e1230, Feb 8 2016, 18:21:23) >>> def f(): ... False ... <stdin>:2: SyntaxWarning: ignore constant statement >>> import dis; dis.dis(f) 2 0 LOAD_CONST 0 (None) 3 RETURN_VALUE Old behaviour: haypo@smithers$ python3 Python 3.4.3 (default, Jun 29 2015, 12:16:01) >>> def f(): ... False ... >>> import dis; dis.dis(f) 2 0 LOAD_CONST 1 (False) 3 POP_TOP 4 LOAD_CONST 0 (None) 7 RETURN_VALUE Before strings and numbers were already ignored. Example: haypo@smithers$ python3 Python 3.4.3 (default, Jun 29 2015, 12:16:01) >>> def f(): ... 123 ... >>> import dis; dis.dis(f) 2 0 LOAD_CONST 0 (None) 3 RETURN_VALUE Victor

18 38

Improving docs for len() of set
by Ben Hoyt Feb. 8, 2016

Feb. 8, 2016

Hi folks, Just a suggestion for a documentation tweak. Currently the docs for len() on a set say this: .. describe:: len(s) Return the cardinality of set *s*. I'm a relatively seasoned programmer, but I don't really have a maths background, and I didn't know what "cardinality" meant. I could kind of grok it by context, but could we change this to something like the following? .. describe:: len(s) Return the number of elements in set *s* (cardinality of *s*). Happy to open a bugs.python.org issue on this, but wanted to get general consensus first. -Ben

5 6

speed.python.org
by Zachary Ware Feb. 8, 2016

Feb. 8, 2016

I'm happy to announce that speed.python.org is finally functional! There's not much there yet, as each benchmark builder has only sent one result so far (and one of those involved a bit of cheating on my part), but it's there. There are likely to be rough edges that still need smoothing out. When you find them, please report them at https://github.com/zware/codespeed/issues or on the speed(a)python.org mailing list. Many thanks to Intel for funding the work to get it set up and to Brett Cannon and Benjamin Peterson for their reviews. Happy benchmarking, -- Zach

6 10

Syntax Highlightning for C API of CPython in VIM
by Stephane Wirtel Feb. 8, 2016

Feb. 8, 2016

Hi everyone, With my talk "Exploring our Python Interpreter", I think this VIM plugin can be useful for the community. It's a syntax highlighter for the C API of CPython 3.5 and 3.6. I used Clang for the parsing and automatically generated the keywords for VIM. PyObject and the others typedefs of CPython will have the defined color of your favourite editor and it's the same for the enums, the typedefs, the functions and the macros. Where can you use this VIM plugin ? If you want to write a CPython extension or if you want to hack in the CPython code. Check this screenshot: http://i.imgur.com/0k13KOU.png Here is the repository: https://github.com/matrixise/cpython-vim-syntax Please, if you see some issues, tell me via an issue on Github. Thank you so much, Stephane -- Stéphane Wirtel - http://wirtel.be - @matrixise

1 0

PEP 0492 __aenter__ & __aexit__
by Daniel Miller Feb. 8, 2016

Feb. 8, 2016

Hi Python-Dev Group, I am trying to implement __aenter__ and __aexit__ for the RethinkDB <https://rethinkdb.com> Python driver. Looking at the PEP I don't see any definitions as to what the expected parameters that __exit__ are supposed to take and couldn't find any other similar implementations. Is there a piece of documentation I should be looking at that I'm missing? https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and… Many thanks, Daniel

3 5

HackIllinois 2016 + Python
by Kevin Hong Feb. 8, 2016

Feb. 8, 2016

Hi all! My name is Kevin and I am a staff member of HackIllinois, a 36-hour hackathon at the University of Illinois Urbana-Champaign where students from across the nation come to build some of the most innovative hardware and software projects. For highlights from last year’s event, check out go.hackillinois.org/video. >From February 19-21st 2016, HackIllinois returns and we are introducing a new initiative called OpenSource@HackIllinois to promote Open Source development during the event. This program is designed to provide students with the opportunity to meet and collaborate with experienced developers, like you all, who serve as a guide and mentor into the open source world. Over the course of the event, you and your group of hackers will build features for an open source project of your choosing. Please see http://www.hackillinois.org/opensource for more details! If you or any other open source developers you work with are interested in learning more about OpenSource@HackIllinois, feel free to email me at kevin.hong(a)hackillinois.org. I look forward to speaking with you soon! Best Regards, Kevin Hong

2 1

Google Summer of Code
by Victor Stinner Feb. 7, 2016

Feb. 7, 2016

Hi, I would like to propose the FAT Python project subject to the Google Summer of Code: https://developers.google.com/open-source/gsoc/ I have a long list of optimization ideas for fatoptimizer: http://fatoptimizer.readthedocs.org/en/latest/todo.html The fatoptimizer project is written in pure Python and has a simple design. I implemented quite simple optimizations which are learnt at school. IMHO such project fits well for a student. Does the PSF already plan to apply to the GSoC? Are there other projects? Victor

3 2

More optimisation ideas
by Steve Dower Feb. 6, 2016

Feb. 6, 2016

Since we're all talking about making Python faster, I thought I'd drop some previous ideas I've had here in case (1) someone wants to actually do them, and (2) they really are new ideas that haven't failed in the past. Mostly I was thinking about startup time. Here are the list of modules imported on clean startup on my Windows, US-English machine (from -v and cleaned up a bit): import _frozen_importlib import _imp import sys import '_warnings' import '_thread' import '_weakref' import '_frozen_importlib_external' import '_io' import 'marshal' import 'nt' import '_thread' import '_weakref' import 'winreg' import 'zipimport' import '_codecs' import 'codecs' import 'encodings.aliases' import 'encodings' import 'encodings.mbcs' import '_signal' import 'encodings.utf_8' import 'encodings.latin_1' import '_weakrefset' import 'abc' import 'io' import 'encodings.cp437' import 'errno' import '_stat' import 'stat' import 'genericpath' import 'ntpath' import '_collections_abc' import 'os' import '_sitebuiltins' import 'sysconfig' import '_locale' import '_bootlocale' import 'encodings.cp1252' import 'site' Obviously the easiest first thing is to remove or delay unnecessary imports. But a while ago I used a native profiler to trace through this and the most impactful modules were the encodings: import 'encodings.mbcs' import 'encodings.utf_8' import 'encodings.latin_1' import 'encodings.cp437' import 'encodings.cp1252' While I don't doubt that we need all of these for *some* reason, aliases, cp437 and cp1252 are relatively expensive modules to import. Mostly due to having large static dictionaries or data structures generated on startup. Given this is static and mostly read-only information[1], I see no reason why we couldn't either generate completely static versions of them, or better yet compile the resulting data structures into the core binary. ([1]: If being able to write to some of the encoding data is used by some people, I vote for breaking that for 3.6 and making it read-only.) This is probably the code snippet that bothered me the most: ### Encoding table encoding_table=codecs.charmap_build(decoding_table) It shows up in many of the encodings modules, and while it is not a bad function in itself, we are obviously generating a known data structure on every startup. Storing these in static data is a tradeoff between disk space and startup performance, and one I think it likely to be worthwhile. Anyway, just an idea if someone wants to try it and see what improvements we can get. I'd love to do it myself, but when it actually comes to finding time I keep coming up short. Cheers, Steve P.S. If you just want to discuss optimisation techniques or benchmarking in general, without specific application to CPython 3.6, there's a whole internet out there. Please don't make me the cause of a pointless centithread. :)

23 50

Summary of Python tracker Issues
by Python tracker Feb. 5, 2016

Feb. 5, 2016

ACTIVITY SUMMARY (2016-01-29 - 2016-02-05) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5413 (+32) closed 32641 (+26) total 38054 (+58) Open issues with patches: 2367 Issues opened (50) ================== #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 reopened by martin.panter #26239: distutils link-objects is not normalized http://bugs.python.org/issue26239 opened by jdfergason #26240: Docstring of the subprocess module should be cleaned up http://bugs.python.org/issue26240 opened by Antony.Lee #26243: zlib.compress level as keyword argument http://bugs.python.org/issue26243 opened by palaviv #26246: Code output toggle button uses removed jQuery method http://bugs.python.org/issue26246 opened by ccwang002 #26247: Document Chrome/Chromium for python2.7 http://bugs.python.org/issue26247 opened by Ismail s #26248: Improve scandir DirEntry docs, especially re symlinks and cach http://bugs.python.org/issue26248 opened by benhoyt #26249: Change PyMem_Malloc to use PyObject_Malloc allocator? http://bugs.python.org/issue26249 opened by haypo #26250: no document for sqlite3.Cursor.connection http://bugs.python.org/issue26250 opened by qinghao #26251: Use "Low-fragmentation Heap" memory allocator on Windows http://bugs.python.org/issue26251 opened by haypo #26252: Add an example to importlib docs on setting up an importer http://bugs.python.org/issue26252 opened by brett.cannon #26253: tarfile in stream mode always set zlib compression level to 9 http://bugs.python.org/issue26253 opened by Patrik Dufresne #26254: ssl should raise an exception when trying to load an unusable http://bugs.python.org/issue26254 opened by abacabadabacaba #26256: Fast decimalisation and conversion to other bases http://bugs.python.org/issue26256 opened by jneb #26257: Eliminate buffer_tests.py http://bugs.python.org/issue26257 opened by martin.panter #26258: readline module for python 3.x on windows http://bugs.python.org/issue26258 opened by Ali Razmjoo #26259: Memleak when repeated calls to asyncio.queue.Queue.get is perf http://bugs.python.org/issue26259 opened by Jonas Brunsgaard #26261: NamedTemporaryFile documentation is vague about the `name` att http://bugs.python.org/issue26261 opened by ztane #26262: Cannot compile with /fp:strict with MSVC http://bugs.python.org/issue26262 opened by zach.ware #26263: Serialize array.array to JSON by default http://bugs.python.org/issue26263 opened by Omer.Katz #26264: keyword module missing async and await keywords http://bugs.python.org/issue26264 opened by tuxtimo #26265: build errors on OS X 10.11 with --enable-universalsdk http://bugs.python.org/issue26265 opened by davidjamesbeck #26266: add classattribute to enum to handle non-Enum attributes http://bugs.python.org/issue26266 opened by ethan.furman #26267: UUID docs should say how to get "standard form" http://bugs.python.org/issue26267 opened by abarnert #26268: Update python.org installers to use OpenSSL 1.0.2f http://bugs.python.org/issue26268 opened by zach.ware #26269: zipfile should call lstat instead of stat if available http://bugs.python.org/issue26269 opened by Patrik Dufresne #26270: Support for read()/write()/select() on asyncio http://bugs.python.org/issue26270 opened by Paulo Costa #26271: freeze.py makefile uses the wrong flags variables http://bugs.python.org/issue26271 opened by Daniel Shaulov #26273: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket modul http://bugs.python.org/issue26273 opened by Omar Sandoval #26275: perf.py: calibrate benchmarks using time, not using a fixed nu http://bugs.python.org/issue26275 opened by haypo #26276: Inconsistent behaviour of PEP 3101 formatting between versions http://bugs.python.org/issue26276 opened by Mark Shannon #26277: Allow zipapp to target modules http://bugs.python.org/issue26277 opened by flying sheep #26278: BaseTransport.close() does not trigger connection_lost() http://bugs.python.org/issue26278 opened by Sümer.Cip #26279: time.strptime does not properly convert out-of-bounds values http://bugs.python.org/issue26279 opened by iaslan #26280: ceval: Optimize [] operation similarly to CPython 2.7 http://bugs.python.org/issue26280 opened by yselivanov #26281: Clear sys.path_importer_cache from importlib.invalidate_caches http://bugs.python.org/issue26281 opened by brett.cannon #26282: Add support for partial keyword arguments in extension functio http://bugs.python.org/issue26282 opened by serhiy.storchaka #26283: zipfile can not handle the path build by os.path.join() http://bugs.python.org/issue26283 opened by 張伯誠 #26284: FIx telco benchmark http://bugs.python.org/issue26284 opened by skrah #26285: Garbage collection of unused input sections from CPython binar http://bugs.python.org/issue26285 opened by alecsandru.patrascu #26286: dis module: coroutine opcode documentation clarity http://bugs.python.org/issue26286 opened by Jim.Jewett #26287: Core dump in f-string with formatting errors due to refcount b http://bugs.python.org/issue26287 opened by encukou #26288: Optimize PyLong_AsDouble for single-digit longs http://bugs.python.org/issue26288 opened by yselivanov #26289: Optimize floor division for ints http://bugs.python.org/issue26289 opened by yselivanov #26290: fileinput and 'for line in sys.stdin' do strange mockery of in http://bugs.python.org/issue26290 opened by Don Hatch #26292: Raw I/O writelines() broken http://bugs.python.org/issue26292 opened by haypo #26293: Embedded zipfile fields dependent on absolute position http://bugs.python.org/issue26293 opened by spoo #26294: Queue().unfinished_tasks not in docs - deliberate? http://bugs.python.org/issue26294 opened by frankmillman #26295: Random failures when running test suite in parallel (-m test - http://bugs.python.org/issue26295 opened by haypo #26296: colorys rgb_to_hls algorithm error http://bugs.python.org/issue26296 opened by Mats Luspa Most recent 15 issues with no replies (15) ========================================== #26294: Queue().unfinished_tasks not in docs - deliberate? http://bugs.python.org/issue26294 #26289: Optimize floor division for ints http://bugs.python.org/issue26289 #26288: Optimize PyLong_AsDouble for single-digit longs http://bugs.python.org/issue26288 #26286: dis module: coroutine opcode documentation clarity http://bugs.python.org/issue26286 #26284: FIx telco benchmark http://bugs.python.org/issue26284 #26283: zipfile can not handle the path build by os.path.join() http://bugs.python.org/issue26283 #26281: Clear sys.path_importer_cache from importlib.invalidate_caches http://bugs.python.org/issue26281 #26279: time.strptime does not properly convert out-of-bounds values http://bugs.python.org/issue26279 #26278: BaseTransport.close() does not trigger connection_lost() http://bugs.python.org/issue26278 #26277: Allow zipapp to target modules http://bugs.python.org/issue26277 #26273: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket modul http://bugs.python.org/issue26273 #26271: freeze.py makefile uses the wrong flags variables http://bugs.python.org/issue26271 #26269: zipfile should call lstat instead of stat if available http://bugs.python.org/issue26269 #26268: Update python.org installers to use OpenSSL 1.0.2f http://bugs.python.org/issue26268 #26266: add classattribute to enum to handle non-Enum attributes http://bugs.python.org/issue26266 Most recent 15 issues waiting for review (15) ============================================= #26289: Optimize floor division for ints http://bugs.python.org/issue26289 #26288: Optimize PyLong_AsDouble for single-digit longs http://bugs.python.org/issue26288 #26285: Garbage collection of unused input sections from CPython binar http://bugs.python.org/issue26285 #26280: ceval: Optimize [] operation similarly to CPython 2.7 http://bugs.python.org/issue26280 #26275: perf.py: calibrate benchmarks using time, not using a fixed nu http://bugs.python.org/issue26275 #26273: Expose TCP_CONGESTION and TCP_USER_TIMEOUT to the socket modul http://bugs.python.org/issue26273 #26271: freeze.py makefile uses the wrong flags variables http://bugs.python.org/issue26271 #26257: Eliminate buffer_tests.py http://bugs.python.org/issue26257 #26249: Change PyMem_Malloc to use PyObject_Malloc allocator? http://bugs.python.org/issue26249 #26248: Improve scandir DirEntry docs, especially re symlinks and cach http://bugs.python.org/issue26248 #26246: Code output toggle button uses removed jQuery method http://bugs.python.org/issue26246 #26243: zlib.compress level as keyword argument http://bugs.python.org/issue26243 #26228: pty.spawn hangs on FreeBSD 9.3, 10.x http://bugs.python.org/issue26228 #26224: Add "version added" for documentation of asyncio.timeout for d http://bugs.python.org/issue26224 #26219: implement per-opcode cache in ceval http://bugs.python.org/issue26219 Top 10 most discussed issues (10) ================================= #21955: ceval.c: implement fast path for integers with a single digit http://bugs.python.org/issue21955 56 msgs #26275: perf.py: calibrate benchmarks using time, not using a fixed nu http://bugs.python.org/issue26275 19 msgs #26249: Change PyMem_Malloc to use PyObject_Malloc allocator? http://bugs.python.org/issue26249 18 msgs #26280: ceval: Optimize [] operation similarly to CPython 2.7 http://bugs.python.org/issue26280 13 msgs #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 10 msgs #26194: Undefined behavior for deque.insert() when len(d) == maxlen http://bugs.python.org/issue26194 10 msgs #26256: Fast decimalisation and conversion to other bases http://bugs.python.org/issue26256 9 msgs #25924: investigate if getaddrinfo(3) on OSX is thread-safe http://bugs.python.org/issue25924 8 msgs #26229: Make number serialization ES6/V8 compatible http://bugs.python.org/issue26229 8 msgs #26285: Garbage collection of unused input sections from CPython binar http://bugs.python.org/issue26285 8 msgs Issues closed (26) ================== #12923: test_urllib fails in refleak mode http://bugs.python.org/issue12923 closed by martin.panter #19587: Remove empty tests in test_bytes.FixedStringTest http://bugs.python.org/issue19587 closed by martin.panter #19883: Integer overflow in zipimport.c http://bugs.python.org/issue19883 closed by serhiy.storchaka #21328: Resize doesn't change reported length on create_string_buffer( http://bugs.python.org/issue21328 closed by Dustin.Oprea #22923: No prompt for "display all X possibilities" on completion-enab http://bugs.python.org/issue22923 closed by yoha #23076: list(pathlib.Path().glob("")) fails with IndexError http://bugs.python.org/issue23076 closed by berker.peksag #23601: use small object allocator for dict key storage http://bugs.python.org/issue23601 closed by rhettinger #25774: [benchmarks] Adjust to allow uploading benchmark data to codes http://bugs.python.org/issue25774 closed by zach.ware #25798: Update python.org installers to use OpenSSL 1.0.2e http://bugs.python.org/issue25798 closed by zach.ware #25934: ICC compiler: ICC treats denormal floating point numbers as 0. http://bugs.python.org/issue25934 closed by zach.ware #25945: Type confusion in partial_setstate and partial_call leads to m http://bugs.python.org/issue25945 closed by serhiy.storchaka #26125: Incorrect error message in the module asyncio.selector_events. http://bugs.python.org/issue26125 closed by berker.peksag #26173: test_ssl.bad_cert_test() exception handling http://bugs.python.org/issue26173 closed by martin.panter #26218: Set PrependPath default to true http://bugs.python.org/issue26218 closed by steve.dower #26222: Missing code in linux_distribution python 2.7.11 http://bugs.python.org/issue26222 closed by berker.peksag #26233: select.epoll.poll() should avoid calling malloc() each time http://bugs.python.org/issue26233 closed by haypo #26238: httplib use wrong hostname in https request with SNI support http://bugs.python.org/issue26238 closed by lvhancy #26241: repr() and str() are identical for floats in 3.5 http://bugs.python.org/issue26241 closed by mark.dickinson #26242: reST formatting error in Doc/library/importlib.rst http://bugs.python.org/issue26242 closed by berker.peksag #26244: zlib.compressobj level default value documentation http://bugs.python.org/issue26244 closed by martin.panter #26245: AttributeError (GL_READ_WRITE) when importing OpenGL.GL http://bugs.python.org/issue26245 closed by berker.peksag #26255: symtable.Symbol.is_referenced() returns false for valid use http://bugs.python.org/issue26255 closed by benjamin.peterson #26260: utf8 decoding inconsistency between P2 and P3 http://bugs.python.org/issue26260 closed by haypo #26272: `zipfile.ZipFile` fails reading a file object in specific vers http://bugs.python.org/issue26272 closed by Pengyu Chen #26274: Add CPU affinity to perf.py http://bugs.python.org/issue26274 closed by haypo #26291: Floating-point arithmetic http://bugs.python.org/issue26291 closed by ebarry

1 0

Opcode cache in ceval loop
by Yury Selivanov Feb. 5, 2016

Feb. 5, 2016

Hi, This is the second email thread I start regarding implementing an opcode cache in ceval loop. Since my first post on this topic: - I've implemented another optimization (LOAD_ATTR); - I've added detailed statistics mode so that I can "see" how the cache performs and tune it; - some macro benchmarks are now 10-20% faster; 2to3 (a real application) is 7-8% faster; - and I have some good insights on the memory footprint. ** The purpose of this email is to get a general approval from python-dev, so that I can start polishing the patches and getting them reviewed/committed. ** Summary of optimizations ------------------------ When a code object is executed more than ~1000 times, it's considered "hot". It gets its opcodes analyzed to initialize caches for LOAD_METHOD (a new opcode I propose to add in [1]), LOAD_ATTR, and LOAD_GLOBAL. It's important to only optimize code objects that were executed "enough" times, to avoid optimizing code objects for modules, classes, and functions that were imported but never used. The cache struct is defined in code.h [2], and is 32 bytes long. When a code object becomes hot, it gets an cache offset table allocated for it (+1 byte for each opcode) + an array of cache structs. To measure the max/average memory impact, I tuned my code to optimize *every* code object on *first* run. Then I ran the entire Python test suite. Python test suite + standard library both contain around 72395 code objects, which required 20Mb of memory for caches. The test process consumed around 400Mb of memory. Thus, the absolute worst case scenario, the overhead is about 5%. Then I ran the test suite without any modifications to the patch. This means that only code objects that are called frequently enough are optimized. In this more, only 2072 code objects were optimized, using less than 1Mb of memory for the cache. LOAD_ATTR --------- Damien George mentioned that they optimize a lot of dict lookups in MicroPython by memorizing last key/value offset in the dict object, thus eliminating lots of hash lookups. I've implemented this optimization in my patch. The results are quite good. A simple micro-benchmark [3] shows ~30% speed improvement. Here are some debug stats generated by 2to3 benchmark: -- Opcode cache LOAD_ATTR hits = 14778415 (83%) -- Opcode cache LOAD_ATTR misses = 750 (0%) -- Opcode cache LOAD_ATTR opts = 282 -- Opcode cache LOAD_ATTR deopts = 60 -- Opcode cache LOAD_ATTR total = 17777912 Each "hit" makes LOAD_ATTR about 30% faster. LOAD_GLOBAL ----------- This turned out to be a very stable optimization. Here is the debug output of the 2to3 test: -- Opcode cache LOAD_GLOBAL hits = 3940647 (100%) -- Opcode cache LOAD_GLOBAL misses = 0 (0%) -- Opcode cache LOAD_GLOBAL opts = 252 All benchmarks (and real code) have stats like that. Globals and builtins are very rarely modified, so the cache works really well. With LOAD_GLOBAL opcode cache, global lookup is very cheap, there is no hash lookup for it at all. It makes optimizations like "def foo(len=len)" obsolete. LOAD_METHOD ----------- This is a new opcode I propose to add in [1]. The idea is to substitute LOAD_ATTR with it, and avoid instantiation of BoundMethod objects. With the cache, we can store a reference to the method descriptor (I use type->tp_version_tag for cache invalidation, the same thing _PyType_Lookup is built around). The cache makes LOAD_METHOD really efficient. A simple micro-benchmark like [4], shows that with the cache and LOAD_METHOD, "s.startswith('abc')" becomes as efficient as "s[:3] == 'abc'". LOAD_METHOD/CALL_FUNCTION without cache is about 20% faster than LOAD_ATTR/CALL_FUNCTION. With the cache, it's about 30% faster. Here's the debug output of the 2to3 benchmark: -- Opcode cache LOAD_METHOD hits = 5164848 (64%) -- Opcode cache LOAD_METHOD misses = 12 (0%) -- Opcode cache LOAD_METHOD opts = 94 -- Opcode cache LOAD_METHOD deopts = 12 -- Opcode cache LOAD_METHOD dct-chk= 1614801 -- Opcode cache LOAD_METHOD total = 7945954 What's next? ------------ First, I'd like to merge the new LOAD_METHOD opcode, see issue 26110 [1]. It's a very straightforward optimization, the patch is small and easy to review. Second, I'd like to merge the new opcode cache, see issue 26219 [5]. All unittests pass. Memory usage increase is very moderate (<1mb for the entire test suite), and the performance increase is significant. The only potential blocker for this is PEP 509 approval (which I'd be happy to assist with). What do you think? Thanks, Yury [1] http://bugs.python.org/issue26110 [2] https://github.com/1st1/cpython/blob/opcache5/Include/code.h#L10 [3] https://gist.github.com/1st1/37d928f1e84813bf1c44 [4] https://gist.github.com/1st1/10588e6e11c4d7c19445 [5] http://bugs.python.org/issue26219

11 36

Git for Mercurial Users
by Nikolaus Rath Feb. 3, 2016

Feb. 3, 2016

Hello, With the upcoming move to Git, I thought people might be interested in some thoughts that I wrote down when learning Git for the first time as a long-time Mercurial user: http://www.rath.org/mercurial-for-git-users-and-vice-versa.html Comments are welcome (but probably more appropriate off-list). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

2 1

Re: [Python-Dev] [Python-checkins] cpython: merge
by Martin Panter Feb. 2, 2016

Feb. 2, 2016

On 2 February 2016 at 05:21, raymond.hettinger <python-checkins(a)python.org> wrote: > https://hg.python.org/cpython/rev/0731f097157b > changeset: 100142:0731f097157b > parent: 100140:c7f1acdd8be1 > user: Raymond Hettinger <python(a)rcn.com> > date: Mon Feb 01 21:21:19 2016 -0800 > summary: > merge > > files: > Doc/library/collections.rst | 4 ++-- > Lib/test/test_deque.py | 23 ++++++++++++----------- > Modules/_collectionsmodule.c | 7 ++----- > 3 files changed, 16 insertions(+), 18 deletions(-) This wasn’t actually a merge (there is only one parent). Hopefully I fixed it up with <https://hg.python.org/cpython/rev/03708c680eca>. But it looks like the original NEWS entry didn’t get merged in your earlier merge <https://hg.python.org/cpython/rev/58266f5101cc>, so there was nothing for me to merge the NEWS changes into in the default branch.

1 0

Re: [Python-Dev] Opcode cache in ceval loop
by Damien George Feb. 1, 2016

Feb. 1, 2016

Hi Yury, That's great news about the speed improvements with the dict offset cache! > The cache struct is defined in code.h [2], and is 32 bytes long. When a > code object becomes hot, it gets an cache offset table allocated for it > (+1 byte for each opcode) + an array of cache structs. Ok, so each opcode has a 1-byte cache that sits separately to the actual bytecode. But a lot of opcodes don't use it so that leads to some wasted memory, correct? But then how do you index the cache, do you keep a count of the current opcode number? If I remember correctly, CPython has some opcodes taking 1 byte, and some taking 3 bytes, so the offset into the bytecode cannot be easily mapped to a bytecode number. Cheers, Damien.

2 2

just a friendly developer
by ty armour Jan. 31, 2016

Jan. 31, 2016

I am looking for tutorials on basically how to write the python language. As well I want to know how to write wrappers for things like alsa and directx and coreaudio and jackd and pulseaudio. I would be looking to write wrappers for coreaudio in mac and coreaudio for windows as well as alsa. i think it would benefit you to have this information. I am looking to aid in development of computers and languages and tutorials on how to write the actual python language would be super useful. but yeah im looking to write applications like guitarix rakarrack blender ardour lmms. I am also going to build my own computers so these tutorials would help a lot. anyhow thanks for your time

3 2