Mailman 3 September 2017 - Python-Dev

PEP 448 review
by Guido van Rossum 30 Mar '23

30 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton 16 Sep '19

16 Sep '19

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

Python startup time
by Victor Stinner 10 Oct '18

10 Oct '18

Hi, On Twitter, Raymond Hettinger wrote: "The decision making process on Python-dev is an anti-pattern, governed by anecdotal data and ambiguity over what problem is solved." https://twitter.com/raymondh/status/887069454693158912 About "anecdotal data", I would like to discuss the Python startup time. == Python 3.7 compared to 2.7 == First of all, on speed.python.org, we have: * Python 2.7: 6.4 ms with site, 3.0 ms without site (-S) * master (3.7): 14.5 ms with site, 8.4 ms without site (-S) Python 3.7 startup time is 2.3x slower with site (default mode), or 2.8x slower without site (-S command line option). (I will skip Python 3.4, 3.5 and 3.6 which are much worse than Python 3.7...) So if an user complained about Python 2.7 startup time: be prepared for a 2x - 3x more angry user when "forced" to upgrade to Python 3! == Mercurial vs Git, Python vs C, startup time == Startup time matters a lot for Mercurial since Mercurial is compared to Git. Git and Mercurial have similar features, but Git is written in C whereas Mercurial is written in Python. Quick benchmark on the speed.python.org server: * hg version: 44.6 ms +- 0.2 ms * git --version: 974 us +- 7 us Mercurial startup time is already 45.8x slower than Git whereas tested Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial developers, with a startup time 2x - 3x slower... I tested Mecurial 3.7.3 and Git 2.7.4 on Ubuntu 16.04.1 using "python3 -m perf command -- ...". == CPython core developers don't care? no, they do care == Christian Heimes, Naoki INADA, Serhiy Storchaka, Yury Selivanov, me (Victor Stinner) and other core developers made multiple changes last years to reduce the number of imports at startup, optimize impotlib, etc. IHMO all these core developers are well aware of the competition of programming languages, and honesty Python startup time isn't "good". So let's compare it to other programming languages similar to Python. == PHP, Ruby, Perl == I measured the startup time of other programming languages which are similar to Python, still on the speed.python.org server using "python3 -m perf command -- ...": * perl -e ' ': 1.18 ms +- 0.01 ms * php -r ' ': 8.57 ms +- 0.05 ms * ruby -e ' ': 32.8 ms +- 0.1 ms Wow, Perl is quite good! PHP seems as good as Python 2 (but Python 3 is worse). Ruby startup time seems less optimized than other languages. Tested versions: * perl 5, version 22, subversion 1 (v5.22.1) * PHP 7.0.18-0ubuntu0.16.04.1 (cli) ( NTS ) * ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu] == Quick Google search == I also searched for "python startup time" and "python slow startup time" on Google and found many articles. Some examples: "Reducing the Python startup time" http://www.draketo.de/book/export/html/498 => "The python startup time always nagged me (17-30ms) and I just searched again for a way to reduce it, when I found this: The Python-Launcher caches GTK imports and forks new processes to reduce the startup time of python GUI programs." https://nelsonslog.wordpress.com/2013/04/08/python-startup-time/ => "Wow, Python startup time is worse than I thought." "How to speed up python starting up and/or reduce file search while loading libraries?" https://stackoverflow.com/questions/15474160/how-to-speed-up-python-startin… => "The first time I log to the system and start one command it takes 6 seconds just to show a few line of help. If I immediately issue the same command again it takes 0.1s. After a couple of minutes it gets back to 6s. (proof of short-lived cache)" "How does one optimise the startup of a Python script/program?" https://www.quora.com/How-does-one-optimise-the-startup-of-a-Python-script-… => "I wrote a Python program that would be used very often (imagine 'cd' or 'ls') for very short runtimes, how would I make it start up as fast as possible?" "Python Interpreter Startup time" https://bytes.com/topic/python/answers/34469-pyhton-interpreter-startup-time "Python is very slow to start on Windows 7" https://stackoverflow.com/questions/29997274/python-is-very-slow-to-start-o… => "Python takes 17 times longer to load on my Windows 7 machine than Ubuntu 14.04 running on a VM" => "returns in 0.614s on Windows and 0.036s on Linux" "How to make a fast command line tool in Python" (old article Python 2.5.2) https://files.bemusement.org/talks/OSDC2008-FastPython/ => "(...) some techniques Bazaar uses to start quickly, such as lazy imports." -- So please continue efforts for make Python startup even faster to beat all other programming languages, and finally convince Mercurial to upgrade ;-) Victor

42 96

PEP 559 - built-in noop()
by Barry Warsaw 24 Nov '17

24 Nov '17

I couldn’t resist one more PEP from the Core sprint. I won’t reveal where or how this one came to me. -Barry PEP: 559 Title: Built-in noop() Author: Barry Warsaw <barry(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-08 Python-Version: 3.7 Post-History: 2017-09-09 Abstract ======== This PEP proposes adding a new built-in function called ``noop()`` which does nothing but return ``None``. Rationale ========= It is trivial to implement a no-op function in Python. It's so easy in fact that many people do it many times over and over again. It would be useful in many cases to have a common built-in function that does nothing. One use case would be for PEP 553, where you could set the breakpoint environment variable to the following in order to effectively disable it:: $ setenv PYTHONBREAKPOINT=noop Implementation ============== The Python equivalent of the ``noop()`` function is exactly:: def noop(*args, **kws): return None The C built-in implementation is available as a pull request. Rejected alternatives ===================== ``noop()`` returns something ---------------------------- YAGNI. This is rejected because it complicates the semantics. For example, if you always return both ``*args`` and ``**kws``, what do you return when none of those are given? Returning a tuple of ``((), {})`` is kind of ugly, but provides consistency. But you might also want to just return ``None`` since that's also conceptually what the function was passed. Or, what if you pass in exactly one positional argument, e.g. ``noop(7)``. Do you return ``7`` or ``((7,), {})``? And so on. The author claims that you won't ever need the return value of ``noop()`` so it will always return ``None``. Coghlin's Dialogs (edited for formatting): My counterargument to this would be ``map(noop, iterable)``, ``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and ``min`` all accept callables that accept a single argument, as do many of the itertools operations). Making ``noop()`` a useful default function in those cases just needs the definition to be:: def noop(*args, **kwds): return args[0] if args else None The counterargument to the counterargument is that using ``None`` as the default in all these cases is going to be faster, since it lets the algorithm skip the callback entirely, rather than calling it and having it do nothing useful. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

14 22

PEP 557: Data Classes
by Eric V. Smith 13 Oct '17

13 Oct '17

I've written a PEP for what might be thought of as "mutable namedtuples with defaults, but not inheriting tuple's behavior" (a mouthful, but it sounded simpler when I first thought of it). It's heavily influenced by the attrs project. It uses PEP 526 type annotations to define fields. From the overview section: @dataclass class InventoryItem: name: str unit_price: float quantity_on_hand: int = 0 def total_cost(self) -> float: return self.unit_price * self.quantity_on_hand Will automatically add these methods: def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0) -> None: self.name = name self.unit_price = unit_price self.quantity_on_hand = quantity_on_hand def __repr__(self): return f'InventoryItem(name={self.name!r},unit_price={self.unit_price!r},quantity_on_hand={self.quantity_on_hand!r})' def __eq__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) == (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented def __ne__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) != (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented def __lt__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) < (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented def __le__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) <= (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented def __gt__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) > (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented def __ge__(self, other): if other.__class__ is self.__class__: return (self.name, self.unit_price, self.quantity_on_hand) >= (other.name, other.unit_price, other.quantity_on_hand) return NotImplemented Data Classes saves you from writing and maintaining these functions. The PEP is largely complete, but could use some filling out in places. Comments welcome! Eric. P.S. I wrote this PEP when I was in my happy place.

25 88

Intention to accept PEP 552 soon (deterministic pyc files)
by Guido van Rossum 09 Oct '17

09 Oct '17

I've read the current version of PEP 552 over and I think everything looks good for acceptance. I believe there are no outstanding objections (or they have been adequately addressed in responses). Therefore I intend to accept PEP 552 this Friday, unless grave objections are raised on this mailing list (python-dev). Congratulations Benjamin. Gotta love those tristate options! -- --Guido van Rossum (python.org/~guido)

11 29

PEP 554 v3 (new interpreters module)
by Eric Snow 09 Oct '17

09 Oct '17

I've updated PEP 554 in response to feedback. (thanks all!) There are a few unresolved points (some of them added to the Open Questions section), but the current PEP has changed enough that I wanted to get it out there first. Notably changed: * the API relative to object passing has changed somewhat drastically (hopefully simpler and easier to understand), replacing "FIFO" with "channel" * added an examples section * added an open questions section * added a rejected ideas section * added more items to the deferred functionality section * the rationale section has moved down below the examples Please let me know what you think. I'm especially interested in feedback about the channels. Thanks! -eric ++++++++++++++++++++++++++++++++++++++++++++++++ PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: Abstract ======== CPython has supported subinterpreters, with increasing levels of support, since version 1.5. The feature has been available via the C-API. [c-api]_ Subinterpreters operate in `relative isolation from one another <Interpreter Isolation_>`_, which provides the basis for an `alternative concurrency model <Concurrency_>`_. This proposal introduces the stdlib ``interpreters`` module. The module will be `provisional <Provisional Status_>`_. It exposes the basic functionality of subinterpreters already provided by the C-API. Proposal ======== The ``interpreters`` module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level ``_interpreters`` module. The proposed API is inspired by the ``threading`` module. See the `Examples`_ section for concrete usage and use cases. API for interpreters -------------------- The module provides the following functions: ``list_all()``:: Return a list of all existing interpreters. ``get_current()``:: Return the currently running interpreter. ``create()``:: Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain idle until something is run in it. The interpreter may be used in any thread and will run in whichever thread calls ``interp.run()``. The module also provides the following class: ``Interpreter(id)``:: id: The interpreter's ID (read-only). is_running(): Return whether or not the interpreter is currently executing code. Calling this on the current interpreter will always return True. destroy(): Finalize and destroy the interpreter. This may not be called on an already running interpreter. Doing so results in a RuntimeError. run(source_str, /, **shared): Run the provided Python source code in the interpreter. Any keyword arguments are added to the interpreter's execution namespace. If any of the values are not supported for sharing between interpreters then RuntimeError gets raised. Currently only channels (see "create_channel()" below) are supported. This may not be called on an already running interpreter. Doing so results in a RuntimeError. A "run()" call is quite similar to any other function call. Once it completes, the code that called "run()" continues executing (in the original interpreter). Likewise, if there is any uncaught exception, it propagates into the code where "run()" was called. The big difference is that "run()" executes the code in an entirely different interpreter, with entirely separate state. The state of the current interpreter in the current OS thread is swapped out with the state of the target interpreter (the one that will execute the code). When the target finishes executing, the original interpreter gets swapped back in and its execution resumes. So calling "run()" will effectively cause the current Python thread to pause. Sometimes you won't want that pause, in which case you should make the "run()" call in another thread. To do so, add a function that calls "run()" and then run that function in a normal "threading.Thread". Note that interpreter's state is never reset, neither before "run()" executes the code nor after. Thus the interpreter state is preserved between calls to "run()". This includes "sys.modules", the "builtins" module, and the internal state of C extension modules. Also note that "run()" executes in the namespace of the "__main__" module, just like scripts, the REPL, "-m", and "-c". Just as the interpreter's state is not ever reset, the "__main__" module is never reset. You can imagine concatenating the code from each "run()" call into one long script. This is the same as how the REPL operates. Supported code: source text. API for sharing data -------------------- The mechanism for passing objects between interpreters is through channels. A channel is a simplex FIFO similar to a pipe. The main difference is that channels can be associated with zero or more interpreters on either end. Unlike queues, which are also many-to-many, channels have no buffer. ``create_channel()``:: Create a new channel and return (recv, send), the RecvChannel and SendChannel corresponding to the ends of the channel. The channel is not closed and destroyed (i.e. garbage-collected) until the number of associated interpreters returns to 0. An interpreter gets associated with a channel by calling its "send()" or "recv()" method. That association gets dropped by calling "close()" on the channel. Both ends of the channel are supported "shared" objects (i.e. may be safely shared by different interpreters. Thus they may be passed as keyword arguments to "Interpreter.run()". ``list_all_channels()``:: Return a list of all open (RecvChannel, SendChannel) pairs. ``RecvChannel(id)``:: The receiving end of a channel. An interpreter may use this to receive objects from another interpreter. At first only bytes will be supported. id: The channel's unique ID. interpreters: The list of associated interpreters (those that have called the "recv()" method). __next__(): Return the next object from the channel. If none have been sent then wait until the next send. recv(): Return the next object from the channel. If none have been sent then wait until the next send. If the channel has been closed then EOFError is raised. recv_nowait(default=None): Return the next object from the channel. If none have been sent then return the default. If the channel has been closed then EOFError is raised. close(): No longer associate the current interpreter with the channel (on the receiving end). This is a noop if the interpreter isn't already associated. Once an interpreter is no longer associated with the channel, subsequent (or current) send() and recv() calls from that interpreter will raise EOFError. Once number of associated interpreters on both ends drops to 0, the channel is actually marked as closed. The Python runtime will garbage collect all closed channels. Note that "close()" is automatically called when it is no longer used in the current interpreter. This operation is idempotent. Return True if the current interpreter was still associated with the receiving end of the channel and False otherwise. ``SendChannel(id)``:: The sending end of a channel. An interpreter may use this to send objects to another interpreter. At first only bytes will be supported. id: The channel's unique ID. interpreters: The list of associated interpreters (those that have called the "send()" method). send(obj): Send the object to the receiving end of the channel. Wait until the object is received. If the channel does not support the object then TypeError is raised. Currently only bytes are supported. If the channel has been closed then EOFError is raised. send_nowait(obj): Send the object to the receiving end of the channel. If the object is received then return True. Otherwise return False. If the channel does not support the object then TypeError is raised. If the channel has been closed then EOFError is raised. close(): No longer associate the current interpreter with the channel (on the sending end). This is a noop if the interpreter isn't already associated. Once an interpreter is no longer associated with the channel, subsequent (or current) send() and recv() calls from that interpreter will raise EOFError. Once number of associated interpreters on both ends drops to 0, the channel is actually marked as closed. The Python runtime will garbage collect all closed channels. Note that "close()" is automatically called when it is no longer used in the current interpreter. This operation is idempotent. Return True if the current interpreter was still associated with the sending end of the channel and False otherwise. Examples ======== Run isolated code ----------------- :: interp = interpreters.create() print('before') interp.run('print("during")') print('after') Run in a thread --------------- :: interp = interpreters.create() def run(): interp.run('print("during")') t = threading.Thread(target=run) print('before') t.start() print('after') Pre-populate an interpreter --------------------------- :: interp = interpreters.create() interp.run("""if True: import some_lib import an_expensive_module some_lib.set_up() """) wait_for_request() interp.run("""if True: some_lib.handle_request() """) Handling an exception --------------------- :: interp = interpreters.create() try: interp.run("""if True: raise KeyError """) except KeyError: print("got the error from the subinterpreter") Synchronize using a channel --------------------------- :: interp = interpreters.create() r, s = interpreters.create_channel() def run(): interp.run("""if True: reader.recv() print("during") reader.close() """, reader=r) t = threading.Thread(target=run) print('before') t.start() print('after') s.send(b'') s.close() Sharing a file descriptor ------------------------- :: interp = interpreters.create() r1, s1 = interpreters.create_channel() r2, s2 = interpreters.create_channel() def run(): interp.run("""if True: fd = int.from_bytes( reader.recv(), 'big') for line in os.fdopen(fd): print(line) writer.send(b'') """, reader=r1, writer=s2) t = threading.Thread(target=run) t.start() with open('spamspamspam') as infile: fd = infile.fileno().to_bytes(1, 'big') s.send(fd) r.recv() Passing objects via pickle -------------------------- :: interp = interpreters.create() r, s = interpreters.create_channel() interp.run("""if True: import pickle """, reader=r) def run(): interp.run("""if True: data = reader.recv() while data: obj = pickle.loads(data) do_something(obj) data = reader.recv() reader.close() """, reader=r) t = threading.Thread(target=run) t.start() for obj in input: data = pickle.dumps(obj) s.send(data) s.send(b'') Rationale ========= Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation. This proposal is focused on enabling the fundamental capability of multiple isolated interpreters in the same Python process. This is a new area for Python so there is relative uncertainly about the best tools to provide as companions to subinterpreters. Thus we minimize the functionality we add in the proposal as much as possible. Concerns -------- * "subinterpreters are not worth the trouble" Some have argued that subinterpreters do not add sufficient benefit to justify making them an official part of Python. Adding features to the language (or stdlib) has a cost in increasing the size of the language. So it must pay for itself. In this case, subinterpreters provide a novel concurrency model focused on isolated threads of execution. Furthermore, they present an opportunity for changes in CPython that will allow simulateous use of multiple CPU cores (currently prevented by the GIL). Alternatives to subinterpreters include threading, async, and multiprocessing. Threading is limited by the GIL and async isn't the right solution for every problem (nor for every person). Multiprocessing is likewise valuable in some but not all situations. Direct IPC (rather than via the multiprocessing module) provides similar benefits but with the same caveat. Notably, subinterpreters are not intended as a replacement for any of the above. Certainly they overlap in some areas, but the benefits of subinterpreters include isolation and (potentially) performance. In particular, subinterpreters provide a direct route to an alternate concurrency model (e.g. CSP) which has found success elsewhere and will appeal to some Python users. That is the core value that the ``interpreters`` module will provide. * "stdlib support for subinterpreters adds extra burden on C extension authors" In the `Interpreter Isolation`_ section below we identify ways in which isolation in CPython's subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext]_ Until that is resolved, C extension authors will face extra difficulty to support subinterpreters. Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state. Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often their users will be affected, what the additional maintenance burden will be for projects, and what the overall benefit of subinterpreters is to offset those costs. The position of this PEP is that the actual extra maintenance burden will be small and well below the threshold at which subinterpreters are worth it. About Subinterpreters ===================== Shared data ----------- Subinterpreters are inherently isolated (with caveats explained below), in contrast to threads. This enables `a different concurrency model <Concurrency_>`_ than is currently readily available in Python. `Communicating Sequential Processes`_ (CSP) is the prime example. A key component of this approach to concurrency is message passing. So providing a message/object passing mechanism alongside ``Interpreter`` is a fundamental requirement. This proposal includes a basic mechanism upon which more complex machinery may be built. That basic mechanism draws inspiration from pipes, queues, and CSP's channels. [fifo]_ The key challenge here is that sharing objects between interpreters faces complexity due in part to CPython's current memory model. Furthermore, in this class of concurrency, the ideal is that objects only exist in one interpreter at a time. However, this is not practical for Python so we initially constrain supported objects to ``bytes``. There are a number of strategies we may pursue in the future to expand supported objects and object sharing strategies. Note that the complexity of object sharing increases as subinterpreters become more isolated, e.g. after GIL removal. So the mechanism for message passing needs to be carefully considered. Keeping the API minimal and initially restricting the supported types helps us avoid further exposing any underlying complexity to Python users. To make this work, the mutable shared state will be managed by the Python runtime, not by any of the interpreters. Initially we will support only one type of objects for shared state: the channels provided by ``create_channel()``. Channels, in turn, will carefully manage passing objects between interpreters. Interpreter Isolation --------------------- CPython's interpreters are intended to be strictly isolated from each other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. [caveats]_ However, there are ways in which interpreters share some state. First of all, some process-global state remains shared: * file descriptors * builtin types (e.g. dict, bytes) * singletons (e.g. None) * underlying static module data (e.g. functions) for builtin/extension/frozen modules There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take subinterpreters into account. This includes things like extension modules that rely on C globals. [cryptography]_ In these cases bugs should be opened (some are already): * readline module hook functions (http://bugs.python.org/issue4202) * memory leaks on re-init (http://bugs.python.org/issue21387) Finally, some potential isolation is missing due to the current design of CPython. Improvements are currently going on to address gaps in this area: * interpreters share the GIL * interpreters share memory management (e.g. allocators, gc) * GC is not run per-interpreter [global-gc]_ * at-exit handlers are not run per-interpreter [global-atexit]_ * extensions using the ``PyGILState_*`` API are incompatible [gilstate]_ Concurrency ----------- Concurrency is a challenging area of software development. Decades of research and practice have led to a wide variety of concurrency models, each with different goals. Most center on correctness and usability. One class of concurrency models focuses on isolated threads of execution that interoperate through some message passing scheme. A notable example is `Communicating Sequential Processes`_ (CSP), upon which Go's concurrency is based. The isolation inherent to subinterpreters makes them well-suited to this approach. Existing Usage -------------- Subinterpreters are not a widely used feature. In fact, the only documented case of wide-spread usage is `mod_wsgi <https://github.com/GrahamDumpleton/mod_wsgi>`_. On the one hand, this case provides confidence that existing subinterpreter support is relatively stable. On the other hand, there isn't much of a sample size from which to judge the utility of the feature. Provisional Status ================== The new ``interpreters`` module will be added with "provisional" status (see PEP 411). This allows Python users to experiment with the feature and provide feedback while still allowing us to adjust to that feedback. The module will be provisional in Python 3.7 and we will make a decision before the 3.8 release whether to keep it provisional, graduate it, or remove it. Alternate Python Implementations ================================ TBD Open Questions ============== Leaking exceptions across interpreters -------------------------------------- As currently proposed, uncaught exceptions from ``run()`` propagate to the frame that called it. However, this means that exception objects are leaking across the inter-interpreter boundary. Likewise, the frames in the traceback potentially leak. While that might not be a problem currently, it would be a problem once interpreters get better isolation relative to memory management (which is necessary to stop sharing the GIL between interpreters). So the semantics of how the exceptions propagate needs to be resolved. Initial support for buffers in channels --------------------------------------- An alternative to support for bytes in channels in support for read-only buffers (the PEP 3119 kind). Then ``recv()`` would return a memoryview to expose the buffer in a zero-copy way. This is similar to what ``multiprocessing.Connection`` supports. [mp-conn] Switching to such an approach would help resolve questions of how passing bytes through channels will work once we isolate memory management in interpreters. Deferred Functionality ====================== In the interest of keeping this proposal minimal, the following functionality has been left out for future consideration. Note that this is not a judgement against any of said capability, but rather a deferment. That said, each is arguably valid. Interpreter.call() ------------------ It would be convenient to run existing functions in subinterpreters directly. ``Interpreter.run()`` could be adjusted to support this or a ``call()`` method could be added:: Interpreter.call(f, *args, **kwargs) This suffers from the same problem as sharing objects between interpreters via queues. The minimal solution (running a source string) is sufficient for us to get the feature out where it can be explored. timeout arg to pop() and push() ------------------------------- Typically functions that have a ``block`` argument also have a ``timeout`` argument. We can add it later if needed. get_main() ---------- CPython has a concept of a "main" interpreter. This is the initial interpreter created during CPython's runtime initialization. It may be useful to identify the main interpreter. For instance, the main interpreter should not be destroyed. However, for the basic functionality of a high-level API a ``get_main()`` function is not necessary. Furthermore, there is no requirement that a Python implementation have a concept of a main interpreter. So until there's a clear need we'll leave ``get_main()`` out. Interpreter.run_in_thread() --------------------------- This method would make a ``run()`` call for you in a thread. Doing this using only ``threading.Thread`` and ``run()`` is relatively trivial so we've left it out. Synchronization Primitives -------------------------- The ``threading`` module provides a number of synchronization primitives for coordinating concurrent operations. This is especially necessary due to the shared-state nature of threading. In contrast, subinterpreters do not share state. Data sharing is restricted to channels, which do away with the need for explicit synchronization. If any sort of opt-in shared state support is added to subinterpreters in the future, that same effort can introduce synchronization primitives to meet that need. CSP Library ----------- A ``csp`` module would not be a large step away from the functionality provided by this PEP. However, adding such a module is outside the minimalist goals of this proposal. Syntactic Support ----------------- The ``Go`` language provides a concurrency model based on CSP, so it's similar to the concurrency model that subinterpreters support. ``Go`` provides syntactic support, as well several builtin concurrency primitives, to make concurrency a first-class feature. Conceivably, similar syntactic (and builtin) support could be added to Python using subinterpreters. However, that is *way* outside the scope of this PEP! Multiprocessing --------------- The ``multiprocessing`` module could support subinterpreters in the same way it supports threads and processes. In fact, the module's maintainer, Davin Potts, has indicated this is a reasonable feature request. However, it is outside the narrow scope of this PEP. C-extension opt-in/opt-out -------------------------- By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily add a mechanism by which C-extension modules could opt out of support for subinterpreters. Then the import machinery, when operating in a subinterpreter, would need to check the module for support. It would raise an ImportError if unsupported. Alternately we could support opting in to subinterpreter support. However, that would probably exclude many more modules (unnecessarily) than the opt-out approach. The scope of adding the ModuleDef slot and fixing up the import machinery is non-trivial, but could be worth it. It all depends on how many extension modules break under subinterpreters. Given the relatively few cases we know of through mod_wsgi, we can leave this for later. Poisoning channels ------------------ CSP has the concept of poisoning a channel. Once a channel has been poisoned, and ``send()`` or ``recv()`` call on it will raise a special exception, effectively ending execution in the interpreter that tried to use the poisoned channel. This could be accomplished by adding a ``poison()`` method to both ends of the channel. The ``close()`` method could work if it had a ``force`` option to force the channel closed. Regardless, these semantics are relatively specialized and can wait. Sending channels over channels ------------------------------ Some advanced usage of subinterpreters could take advantage of the ability to send channels over channels, in addition to bytes. Given that channels will already be multi-interpreter safe, supporting then in ``RecvChannel.recv()`` wouldn't be a big change. However, this can wait until the basic functionality has been ironed out. Reseting __main__ ----------------- As proposed, every call to ``Interpreter.run()`` will execute in the namespace of the interpreter's existing ``__main__`` module. This means that data persists there between ``run()`` calls. Sometimes this isn't desireable and you want to execute in a fresh ``__main__``. Also, you don't necessarily want to leak objects there that you aren't using any more. Solutions include: * a ``create()`` arg to indicate resetting ``__main__`` after each ``run`` call * an ``Interpreter.reset_main`` flag to support opting in or out after the fact * an ``Interpreter.reset_main()`` method to opt in when desired This isn't a critical feature initially. It can wait until later if desirable. Support passing ints in channels -------------------------------- Passing ints around should be fine and ultimately is probably desirable. However, we can get by with serializing them as bytes for now. The goal is a minimal API for the sake of basic functionality at first. File descriptors and sockets in channels ---------------------------------------- Given that file descriptors and sockets are process-global resources, support for passing them through channels is a reasonable idea. They would be a good candidate for the first effort at expanding the types that channels support. They aren't strictly necessary for the initial API. Rejected Ideas ============== Explicit channel association ---------------------------- Interpreters are implicitly associated with channels upon ``recv()`` and ``send()`` calls. They are de-associated with ``close()`` calls. The alternative would be explicit methods. It would be either ``add_channel()`` and ``remove_channel()`` methods on ``Interpreter`` objects or something similar on channel objects. In practice, this level of management shouldn't be necessary for users. So adding more explicit support would only add clutter to the API. Use pipes instead of channels ----------------------------- A pipe would be a simplex FIFO between exactly two interpreters. For most use cases this would be sufficient. It could potentially simplify the implementation as well. However, it isn't a big step to supporting a many-to-many simplex FIFO via channels. Also, with pipes the API ends up being slightly more complicated, requiring naming the pipes. Use queues instead of channels ------------------------------ The main difference between queues and channels is that queues support buffering. This would complicate the blocking semantics of ``recv()`` and ``send()``. Also, queues can be built on top of channels. "enumerate" ----------- The ``list_all()`` function provides the list of all interpreters. In the threading module, which partly inspired the proposed API, the function is called ``enumerate()``. The name is different here to avoid confusing Python users that are not already familiar with the threading API. For them "enumerate" is rather unclear, whereas "list_all" is clear. References ========== .. [c-api] https://docs.python.org/3/c-api/init.html#sub-interpreter-support .. _Communicating Sequential Processes: .. [CSP] https://en.wikipedia.org/wiki/Communicating_sequential_processes https://github.com/futurecore/python-csp .. [fifo] https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/queue.html#module-queue http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html https://golang.org/doc/effective_go.html#sharing http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-fe… .. [caveats] https://docs.python.org/3/c-api/init.html#bugs-and-caveats .. [petr-c-ext] https://mail.python.org/pipermail/import-sig/2016-June/001062.html https://mail.python.org/pipermail/python-ideas/2016-April/039748.html .. [cryptography] https://github.com/pyca/cryptography/issues/2299 .. [global-gc] http://bugs.python.org/issue24554 .. [gilstate] https://bugs.python.org/issue10915 http://bugs.python.org/issue15751 .. [global-atexit] https://bugs.python.org/issue6531 .. [mp-conn] https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Conn… Copyright ========= This document has been placed in the public domain.

10 38

Summary of Python tracker Issues
by Python tracker 29 Sep '17

29 Sep '17

ACTIVITY SUMMARY (2017-09-22 - 2017-09-29) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6208 (+12) closed 37181 (+74) total 43389 (+86) Open issues with patches: 2384 Issues opened (54) ================== #25287: test_crypt fails on OpenBSD https://bugs.python.org/issue25287 reopened by serhiy.storchaka #31553: Extend json.tool to handle jsonlines (with a flag) https://bugs.python.org/issue31553 opened by Eric Moyer #31554: Warn when __loader__ != __spec__.loader https://bugs.python.org/issue31554 opened by brett.cannon #31555: Windows pyd slower when not loaded via load_dynamic https://bugs.python.org/issue31555 opened by Safihre #31556: asyncio.wait_for can cancel futures faster with timeout==0 https://bugs.python.org/issue31556 opened by hellysmile #31557: tarfile: incorrectly treats regular file as directory https://bugs.python.org/issue31557 opened by Joe Tsai #31558: gc.freeze() - an API to mark objects as uncollectable https://bugs.python.org/issue31558 opened by lukasz.langa #31562: snakebite.net is not available https://bugs.python.org/issue31562 opened by serhiy.storchaka #31565: open() doesn't use locale.getpreferredencoding() https://bugs.python.org/issue31565 opened by serhiy.storchaka #31567: Inconsistent documentation around decorators https://bugs.python.org/issue31567 opened by dmiyakawa #31572: Avoid suppressing all exceptions in PyObject_HasAttr() https://bugs.python.org/issue31572 opened by serhiy.storchaka #31573: PyStructSequence_New() doesn't validate its input type (crashe https://bugs.python.org/issue31573 opened by Oren Milman #31574: Add dtrace hook for importlib https://bugs.python.org/issue31574 opened by christian.heimes #31577: crash in os.utime() in case of a bad ns argument https://bugs.python.org/issue31577 opened by Oren Milman #31581: Reduce the number of imports for functools https://bugs.python.org/issue31581 opened by inada.naoki #31582: Add _pth breadcrumb to sys.path documentation https://bugs.python.org/issue31582 opened by Traveler Hauptman #31583: 2to3 call for file in current directory yields error https://bugs.python.org/issue31583 opened by denis-osipov #31584: Documentation Language mixed up https://bugs.python.org/issue31584 opened by asl #31589: Links for French documentation pdf is broken https://bugs.python.org/issue31589 opened by fabrice #31590: CSV module incorrectly treats escaped newlines as new records https://bugs.python.org/issue31590 opened by mallyvai #31591: Closing socket raises AttributeError: 'collections.deque' obje https://bugs.python.org/issue31591 opened by reidfaiv #31592: assertion failure in Python/ast.c in case of a bad unicodedata https://bugs.python.org/issue31592 opened by Oren Milman #31594: Make bytes and bytearray maketrans accept dictionaries as firs https://bugs.python.org/issue31594 opened by oleksandr.suvorov #31596: expose pthread_getcpuclockid in time module https://bugs.python.org/issue31596 opened by pdox #31601: Availability of utimensat and futimens not checked correctly o https://bugs.python.org/issue31601 opened by jmr #31602: assertion failure in zipimporter.get_source() in case of a bad https://bugs.python.org/issue31602 opened by Oren Milman #31603: Please add argument to override stdin/out/err in the input bui https://bugs.python.org/issue31603 opened by wt #31604: unittest.TestLoader().loadTestsFromTestCase(...) fails when ad https://bugs.python.org/issue31604 opened by Erasmus Cedernaes #31606: [Windows] Tests failing on installed Python 3.7a1 on Windows https://bugs.python.org/issue31606 opened by haypo #31607: Add listsize in pdb.py https://bugs.python.org/issue31607 opened by matrixise #31608: crash in methods of a subclass of _collections.deque with a ba https://bugs.python.org/issue31608 opened by Oren Milman #31609: PCbuild\clean.bat fails if the path contains whitespaces https://bugs.python.org/issue31609 opened by serhiy.storchaka #31610: Use select.poll instead of select.select in SocketServer.BaseS https://bugs.python.org/issue31610 opened by Беатрис Бонева #31611: Tests failures using -u largefile when the disk is full https://bugs.python.org/issue31611 opened by haypo #31612: Building 3.6 fails on Windows https://bugs.python.org/issue31612 opened by serhiy.storchaka #31613: tkinter.simpledialog default buttons (not buttonbox) are not l https://bugs.python.org/issue31613 opened by jcrmatos #31618: Change sys.settrace opcode tracing to occur after frame line n https://bugs.python.org/issue31618 opened by gwk #31619: Strange error when convert hexadecimal with underscores to int https://bugs.python.org/issue31619 opened by serhiy.storchaka #31620: asyncio.Queue leaks memory if the queue is empty and consumers https://bugs.python.org/issue31620 opened by zackelan #31622: Make threading.get_ident() return an opaque type https://bugs.python.org/issue31622 opened by pdox #31623: Allow to build MSI installer from the official tarball https://bugs.python.org/issue31623 opened by Ivan.Pozdeev #31625: stop using ranlib https://bugs.python.org/issue31625 opened by benjamin.peterson #31626: Crash in _PyUnicode_DecodeUnicodeEscape on OpenBSD https://bugs.python.org/issue31626 opened by serhiy.storchaka #31627: test_mailbox fails if the hostname is empty https://bugs.python.org/issue31627 opened by serhiy.storchaka #31628: test_emails failure on FreeBSD https://bugs.python.org/issue31628 opened by serhiy.storchaka #31629: test_multiprocessing_fork failure on FreeBSD https://bugs.python.org/issue31629 opened by serhiy.storchaka #31630: math.tan has poor accuracy near pi/2 on OpenBSD https://bugs.python.org/issue31630 opened by serhiy.storchaka #31631: test_c_locale_coercion fails on OpenBSD https://bugs.python.org/issue31631 opened by serhiy.storchaka #31632: Asyncio: SSL transport does not support set_protocol() https://bugs.python.org/issue31632 opened by jlacoline #31634: Consider installing wheel in ensurepip by default https://bugs.python.org/issue31634 opened by Segev Finer #31635: test_strptime failure on OpenBSD https://bugs.python.org/issue31635 opened by serhiy.storchaka #31636: test_locale failure on OpenBSD https://bugs.python.org/issue31636 opened by serhiy.storchaka #31638: zipapp module should support compression https://bugs.python.org/issue31638 opened by zmwangx #1649329: Extract file-finding and language-handling code from gettext.f https://bugs.python.org/issue1649329 reopened by merwok Most recent 15 issues with no replies (15) ========================================== #31638: zipapp module should support compression https://bugs.python.org/issue31638 #31636: test_locale failure on OpenBSD https://bugs.python.org/issue31636 #31635: test_strptime failure on OpenBSD https://bugs.python.org/issue31635 #31634: Consider installing wheel in ensurepip by default https://bugs.python.org/issue31634 #31632: Asyncio: SSL transport does not support set_protocol() https://bugs.python.org/issue31632 #31631: test_c_locale_coercion fails on OpenBSD https://bugs.python.org/issue31631 #31630: math.tan has poor accuracy near pi/2 on OpenBSD https://bugs.python.org/issue31630 #31629: test_multiprocessing_fork failure on FreeBSD https://bugs.python.org/issue31629 #31628: test_emails failure on FreeBSD https://bugs.python.org/issue31628 #31609: PCbuild\clean.bat fails if the path contains whitespaces https://bugs.python.org/issue31609 #31602: assertion failure in zipimporter.get_source() in case of a bad https://bugs.python.org/issue31602 #31601: Availability of utimensat and futimens not checked correctly o https://bugs.python.org/issue31601 #31596: expose pthread_getcpuclockid in time module https://bugs.python.org/issue31596 #31590: CSV module incorrectly treats escaped newlines as new records https://bugs.python.org/issue31590 #31583: 2to3 call for file in current directory yields error https://bugs.python.org/issue31583 Most recent 15 issues waiting for review (15) ============================================= #31638: zipapp module should support compression https://bugs.python.org/issue31638 #31634: Consider installing wheel in ensurepip by default https://bugs.python.org/issue31634 #31632: Asyncio: SSL transport does not support set_protocol() https://bugs.python.org/issue31632 #31627: test_mailbox fails if the hostname is empty https://bugs.python.org/issue31627 #31625: stop using ranlib https://bugs.python.org/issue31625 #31623: Allow to build MSI installer from the official tarball https://bugs.python.org/issue31623 #31620: asyncio.Queue leaks memory if the queue is empty and consumers https://bugs.python.org/issue31620 #31619: Strange error when convert hexadecimal with underscores to int https://bugs.python.org/issue31619 #31608: crash in methods of a subclass of _collections.deque with a ba https://bugs.python.org/issue31608 #31607: Add listsize in pdb.py https://bugs.python.org/issue31607 #31603: Please add argument to override stdin/out/err in the input bui https://bugs.python.org/issue31603 #31602: assertion failure in zipimporter.get_source() in case of a bad https://bugs.python.org/issue31602 #31592: assertion failure in Python/ast.c in case of a bad unicodedata https://bugs.python.org/issue31592 #31583: 2to3 call for file in current directory yields error https://bugs.python.org/issue31583 #31581: Reduce the number of imports for functools https://bugs.python.org/issue31581 Top 10 most discussed issues (10) ================================= #31415: Add -X option to show import time https://bugs.python.org/issue31415 11 msgs #31584: Documentation Language mixed up https://bugs.python.org/issue31584 11 msgs #30844: selectors: Add urgent data to read event https://bugs.python.org/issue30844 10 msgs #31158: test_pty: test_basic() fails randomly on Travis CI https://bugs.python.org/issue31158 10 msgs #31285: a SystemError and an assertion failure in warnings.warn_explic https://bugs.python.org/issue31285 10 msgs #31170: Update to expat 2.2.4 (expat: utf8_toUtf8 cannot properly hand https://bugs.python.org/issue31170 8 msgs #31612: Building 3.6 fails on Windows https://bugs.python.org/issue31612 8 msgs #29729: RFE: uuid.UUID(bytes=...): support bytes-like types, not only https://bugs.python.org/issue29729 7 msgs #31558: gc.freeze() - an API to mark objects as uncollectable https://bugs.python.org/issue31558 7 msgs #31530: Python 2.7 readahead feature of file objects is not thread saf https://bugs.python.org/issue31530 6 msgs Issues closed (72) ================== #5885: uuid.uuid1() is too slow https://bugs.python.org/issue5885 closed by haypo #11063: Rework uuid module: lazy initialization and add a new C extens https://bugs.python.org/issue11063 closed by pitrou #18257: Two copies of python-config https://bugs.python.org/issue18257 closed by r.david.murray #18558: Iterable glossary entry needs clarification https://bugs.python.org/issue18558 closed by rhettinger #20519: Replace uuid ctypes usage with an extension module. https://bugs.python.org/issue20519 closed by haypo #22140: "python-config --includes" returns a wrong path (double prefix https://bugs.python.org/issue22140 closed by benjamin.peterson #23702: docs.python.org/3/howto/descriptor.html still refers to "unbou https://bugs.python.org/issue23702 closed by rhettinger #25351: pyvenv activate script failure with specific bash option https://bugs.python.org/issue25351 closed by haypo #25532: infinite loop when running inspect.unwrap over unittest.mock.c https://bugs.python.org/issue25532 closed by ncoghlan #25732: functools.total_ordering does not correctly implement not equa https://bugs.python.org/issue25732 closed by serhiy.storchaka #27385: itertools.groupby has misleading doc string https://bugs.python.org/issue27385 closed by rhettinger #28129: assertion failures in ctypes https://bugs.python.org/issue28129 closed by haypo #28754: Argument Clinic for bisect.bisect_left https://bugs.python.org/issue28754 closed by rhettinger #29400: Add instruction level tracing via sys.settrace https://bugs.python.org/issue29400 closed by gwk #29624: Python 3.5.3 x86 web installer cannot install launcher https://bugs.python.org/issue29624 closed by steve.dower #30085: Discourage operator.__dunder__ functions https://bugs.python.org/issue30085 closed by terry.reedy #30152: Reduce the number of imports for argparse https://bugs.python.org/issue30152 closed by serhiy.storchaka #30346: Odd behavior when unpacking `itertools.groupby` https://bugs.python.org/issue30346 closed by serhiy.storchaka #30347: itertools.groupby() can fail a C assert() https://bugs.python.org/issue30347 closed by serhiy.storchaka #30947: Update embeded copy of libexpat from 2.2.1 to 2.2.3 https://bugs.python.org/issue30947 closed by haypo #30995: Support logging.getLogger(style='{') https://bugs.python.org/issue30995 closed by vinay.sajip #31153: Update docstrings of itertools functions https://bugs.python.org/issue31153 closed by rhettinger #31159: Doc: Language switch can't switch on specific cases https://bugs.python.org/issue31159 closed by mdk #31311: a SystemError and a crash in PyCData_setstate() when __dict__ https://bugs.python.org/issue31311 closed by serhiy.storchaka #31351: ensurepip discards pip's return code which leads to broken ven https://bugs.python.org/issue31351 closed by ncoghlan #31355: Remove Travis CI macOS job: rely on buildbots https://bugs.python.org/issue31355 closed by haypo #31389: Give pdb.set_trace() an optional `header` keyword argument https://bugs.python.org/issue31389 closed by barry #31423: Error while building PDF documentation https://bugs.python.org/issue31423 closed by zach.ware #31443: Possibly out of date C extension documentation https://bugs.python.org/issue31443 closed by skrah #31459: IDLE: Rename Class Browser as Module Browser https://bugs.python.org/issue31459 closed by terry.reedy #31490: assertion failure in ctypes in case an _anonymous_ attr appear https://bugs.python.org/issue31490 closed by serhiy.storchaka #31492: assertion failures in case a module has a bad __name__ attribu https://bugs.python.org/issue31492 closed by serhiy.storchaka #31494: Valgrind suppression file https://bugs.python.org/issue31494 closed by skrah #31502: IDLE: Config dialog again deletes custom themes and keysets. https://bugs.python.org/issue31502 closed by terry.reedy #31505: assertion failure in json, in case _json.make_encoder() receiv https://bugs.python.org/issue31505 closed by serhiy.storchaka #31508: Running test_ttk_guionly logs "test_widgets.py:1562: UserWarni https://bugs.python.org/issue31508 closed by serhiy.storchaka #31549: test_strptime and test_time fail on non-English Windows https://bugs.python.org/issue31549 closed by serhiy.storchaka #31559: IDLE: test_browser is failed when run twice https://bugs.python.org/issue31559 closed by terry.reedy #31560: hashlib.blake2: error in example for signed cookies https://bugs.python.org/issue31560 closed by benjamin.peterson #31561: difflib pathological behavior with mixed line endings https://bugs.python.org/issue31561 closed by rhettinger #31563: Change repr of dict_keys, dict_values, and dict_items to use c https://bugs.python.org/issue31563 closed by serhiy.storchaka #31564: NewType can subclass NewType https://bugs.python.org/issue31564 closed by Mariatta #31566: assertion failure in _warnings.warn() in case of a bad __name_ https://bugs.python.org/issue31566 closed by serhiy.storchaka #31568: Configure thinks it finds python3.5 but doesn't https://bugs.python.org/issue31568 closed by haypo #31569: inconsistent case of PCbuild/ directory https://bugs.python.org/issue31569 closed by steve.dower #31570: minor bug in documentataion relating to sending html email https://bugs.python.org/issue31570 closed by Mariatta #31571: Redundand information on Doc/reference/lexical_analysis.rst https://bugs.python.org/issue31571 closed by Mariatta #31575: Functional Programming HOWTO sub-optimal example for reduce https://bugs.python.org/issue31575 closed by rhettinger #31576: problem in math https://bugs.python.org/issue31576 closed by haypo #31578: Unexpected Floating Point Division Result https://bugs.python.org/issue31578 closed by r.david.murray #31579: Reference leak in enumerate https://bugs.python.org/issue31579 closed by serhiy.storchaka #31580: Defer compiling regular expressions https://bugs.python.org/issue31580 closed by barry #31585: Refactor the enumerate.__next__ implementation https://bugs.python.org/issue31585 closed by rhettinger #31586: SystemError in _collections._count_element() in case of a bad https://bugs.python.org/issue31586 closed by rhettinger #31587: Enum doesn't allow name https://bugs.python.org/issue31587 closed by Robert Gomułka #31588: SystemError in class creation in case of a metaclass with a ba https://bugs.python.org/issue31588 closed by ncoghlan #31593: [2.7][3.6] test_socketserver ForkingMixIn tests leaks child pr https://bugs.python.org/issue31593 closed by haypo #31595: Preferring MSVC 2017 in Python 3.6.x new minor releases https://bugs.python.org/issue31595 closed by steve.dower #31597: ipaddress.hosts() doesn't return anything on a /32 "network" https://bugs.python.org/issue31597 closed by serhiy.storchaka #31598: pyping 0.0.5 package crashed on ZeroDivisionError: float divis https://bugs.python.org/issue31598 closed by r.david.murray #31599: bug in doctest when comparing - 0.0 == 0.0 https://bugs.python.org/issue31599 closed by r.david.murray #31600: please allow association of google login with current account https://bugs.python.org/issue31600 closed by serhiy.storchaka #31605: meta issue: bugs.python.org search shows only issues with rece https://bugs.python.org/issue31605 closed by ezio.melotti #31614: can't list groupby generator without breaking the sub groups g https://bugs.python.org/issue31614 closed by rhettinger #31615: Python interpreter is failing due to BufferOverflow Exception https://bugs.python.org/issue31615 closed by serhiy.storchaka #31616: Windows installer: Python binaries are user-writable https://bugs.python.org/issue31616 closed by steve.dower #31617: os.path.join() return wrong value in windows https://bugs.python.org/issue31617 closed by eryksun #31621: Pluralization typo in Language Reference section 7.12 https://bugs.python.org/issue31621 closed by Mariatta #31624: remove support for BSD/OS https://bugs.python.org/issue31624 closed by benjamin.peterson #31633: test_crypt fails on OpenBSD https://bugs.python.org/issue31633 closed by serhiy.storchaka #31637: integer overflow in the size of a ctypes.Array https://bugs.python.org/issue31637 closed by Oren Milman #1612262: IDLE: Include nested functions and classes in module browser https://bugs.python.org/issue1612262 closed by terry.reedy

1 0

bugs.python.org updated
by Ezio Melotti 28 Sep '17

28 Sep '17

Hi, yesterday I updated the version of Roundup used to run our bugs.python.org instance and 4 other instances (the meta, jython, setuptools, and roundup trackers). If everything went right you shouldn't have noticed anything different :) This update included ~300 changesets from upstream and required an additional ~30 to update our instances and our fork of Roundup. A number of features that we added to our fork over the years have been ported upstream and they have now been removed from our fork, which is now -- except for the github integration -- almost aligned with upstream. If you notice any issue with the bug tracker (/with/, not /in/ -- that's expected!), please report it to the meta tracker ( http://psf.upfronthosting.co.za/roundup/meta/) and/or to me. If there are no reports and everything works fine I still have a few more updates coming up ;) A big thanks to John Rouillard (from the Roundup team) and R. David Murray for the help! Best Regards, Ezio Melotti P.S. Roundup started moving towards Python 3.

3 2