Mailman 3 April 2020 - Python-Dev

Mixed Python/C debugging
by Skip Montanaro April 26, 2023

April 26, 2023

Having tried comp.lang.python with no response, I turn here... After at least ten years away from Python's run-time interpreter & byte code compiler, I'm getting set to familiarize myself with that again. This will, I think, entail debugging a mixed Python/C environment. I'm an Emacs user and am aware that GDB since 7.0 has support for debugging at the Python code level. Is Emacs+GDB my best bet? Are there any Python IDEs which support C-level breakpoints and debugging? Thanks, Skip

5 6

PEP 448 review
by Guido van Rossum March 29, 2023

March 29, 2023

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com May 18, 2021

May 18, 2021

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

Boundaries between numbers and identifiers
by Serhiy Storchaka April 15, 2021

April 15, 2021

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

8 10

pathlib.Path: inconsistent symlink_to() and link_to()
by Barney Gale Jan. 22, 2021

Jan. 22, 2021

Hi, Pathlib's symlink_to() and link_to() methods have different argument orders, so: a.symlink_to(b) # Creates a symlink from A to B a.link_to(b) # Creates a hard link from B to A I don't think link_to() was intended to be implemented this way, as the docs say "Create a hard link pointing to a path named target.". It's also inconsistent with everything else in pathlib, most obviously symlink_to(). Bug report here: https://bugs.python.org/issue39291 This /really/ irks me. Apparently it's too late to fix link_to(), so I'd like to suggest we add a new hardlink_to() method that matches the symlink_to() argument order. link_to() then becomes deprecated/undocumented. Any thoughts? Barney

4 5

test_ssl.py hangs with SSL 1.1 built with no threads
by mig28suarez＠hotmail.com May 17, 2020

May 17, 2020

Hello! This is my first time posting to the group. I've been running builds of Python 3.7.x on CentOS Linux release 7.7 (64bit/Intel Core 2 Duo) and I ran into hangs with test_ssl.py when using latest SSL 1.1.1d sources. I've done a full compilation from source for Python 3.7.7 and SSL 1.1 in my workspaces. From what I can tell the problem is when SSL 1.1 is built with no threading there is no locking enabled by python. This one line change will make the hangs in test_ssl.py go away: Index: Modules/_ssl.c =================================================================== --- Modules/_ssl.c (revision 70) +++ Modules/_ssl.c (working copy) @@ -5875,7 +5875,7 @@ if (!_setup_ssl_threads()) { return NULL; } -#elif OPENSSL_VERSION_1_1 && defined(OPENSSL_THREADS) +#elif OPENSSL_VERSION_1_1 /* OpenSSL 1.1.0 builtin thread support is enabled */ _ssl_locks_count++; #endif There appears to be an assumption in _ssl.c and test_ssl.y that SSL 1.1 will be threaded but this may not be true (as in my case). Appreciate any feedback. Thanks!

2 1

Re: Improvement to SimpleNamespace
by Raymond Hettinger May 7, 2020

May 7, 2020

[GvR] > We should not try to import JavaScript's object model into Python. Yes, I get that. Just want to point-out that working with heavily nested dictionaries (typical for JSON) is no fun with square brackets and quotation marks. Raymond

8 10

PEP 617: New PEG parser for CPython
by Guido van Rossum May 7, 2020

May 7, 2020

Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros Nikolaou and myself have been working on a new parser for CPython. We are now far enough along that we present a PEP we've written: https://www.python.org/dev/peps/pep-0617/ Hopefully the PEP speaks for itself. We are hoping for a speedy resolution so we can land the code we've written before 3.9 beta 1. If people insist I can post a copy of the entire PEP here on the list, but since a lot of it is just background information on the old LL(1) and the new PEG parsing algorithms, I figure I'd spare everyone the need of reading through that. Below is a copy of the most relevant section from the PEP. I'd also like to point out the section on performance (which you can find through the above link) -- basically performance is on a par with that of the old parser. ============== Migration plan ============== This section describes the migration plan when porting to the new PEG-based parser if this PEP is accepted. The migration will be executed in a series of steps that allow initially to fallback to the previous parser if needed: 1. Before Python 3.9 beta 1, include the new PEG-based parser machinery in CPython with a command-line flag and environment variable that allows switching between the new and the old parsers together with explicit APIs that allow invoking the new and the old parsers independently. At this step, all Python APIs like ``ast.parse`` and ``compile`` will use the parser set by the flags or the environment variable and the default parser will be the current parser. 2. After Python 3.9 Beta 1 the default parser will be the new parser. 3. Between Python 3.9 and Python 3.10, the old parser and related code (like the "parser" module) will be kept until a new Python release happens (Python 3.10). In the meanwhile and until the old parser is removed, **no new Python Grammar addition will be added that requires the peg parser**. This means that the grammar will be kept LL(1) until the old parser is removed. 4. In Python 3.10, remove the old parser, the command-line flag, the environment variable and the "parser" module and related code. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

22 58

Moving tzdata into the python organization
by Paul Ganssle May 3, 2020

May 3, 2020

Hi all, PEP 615 specifies that we will add a first party tzdata package <https://www.python.org/dev/peps/pep-0615/#the-tzdata-python-package>. I created this package at pganssle/tzdata <https://github.com/pganssle/tzdata>, but I believe the final home for it should be python/tzdata. Are there any objections to me moving it into the python org as is? Thanks! Paul

2 2

Adding a "call_once" decorator to functools
by tom＠tomforb.es May 1, 2020

May 1, 2020

Hello, After a great discussion in python-ideas[1][2] it was suggested that I cross-post this proposal to python-dev to gather more comments from those who don't follow python-ideas. The proposal is to add a "call_once" decorator to the functools module that, as the name suggests, calls a wrapped function once, caching the result and returning it with subsequent invocations. The rationale behind this proposal is that: 1. Developers are using "lru_cache" to achieve this right now, which is less efficient than it could be 2. Special casing "lru_cache" to account for zero arity methods isn't trivial and we shouldn't endorse lru_cache as a way of achieving "call_once" semantics 3. Implementing a thread-safe (or even non-thread safe) "call_once" method is non-trivial 4. It complements the lru_cache and cached_property methods currently present in functools. The specifics of the method would be: 1. The wrapped method is guaranteed to only be called once when called for the first time by concurrent threads 2. Only functions with no arguments can be wrapped, otherwise an exception is thrown 3. There is a C implementation to keep speed parity with lru_cache I've included a naive implementation below (that doesn't meet any of the specifics listed above) to illustrate the general idea of the proposal: ``` def call_once(func): sentinel = object() # in case the wrapped method returns None obj = sentinel @functools.wraps(func) def inner(): nonlocal obj, sentinel if obj is sentinel: obj = func() return obj return inner ``` I'd welcome any feedback on this proposal, and if the response is favourable I'd love to attempt to implement it. 1. https://mail.python.org/archives/list/python-ideas@python.org/thread/5OR3LJ… 2. https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…

15 23

Virtual machine bleeds into generator implementation?
by Skip Montanaro April 30, 2020

April 30, 2020

This is more an observation and question than anything else, but perhaps it will stimulate some ideas from the experts. Consider this trivial generator function: def gen(a): yield a When the YIELD_VALUE instruction is executed, it executes (in the non-async case): retval = POP(); f->f_stacktop = stack_pointer; goto exiting; This is fine as far as it goes. However, execution eventually leads to Objects/genobject.c where we hit this code (I think after falling off the YIELD_VALUE instruction, but perhaps virtual machine execution reaches RETURN_VALUE): /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result && f->f_stacktop == NULL) { There are several other references to f->f_stacktop in genobject.c. I've not yet investigated all of them. As I'm working on a register-based virtual machine implementation, I don't fiddle with the stack at all, so it's a bit problematic that the generator implementation is so intimate with the stack. As this is an area of the core which is completely new to me, I wonder if someone can suggest alternate ways of achieving the same effect without relying on the state of the stack. It seems to me that from within PyEval_EvalFrameDefault the implementations of relevant instructions could reference the generator object via f->f_gen and call some (new?) private API on generators which toggles the relevant bit of state in the generator. I think it's worse that this though, as it seems that in gen_send_ex() it actually pushes a value onto the stack. That can't be solved by simply adding a state attribute to the generator object struct. Skip

3 4

PEP 554 comments
by Antoine Pitrou April 30, 2020

April 30, 2020

Hello, First, I would like to say that I have no fondamental problem with this PEP. While I agree with Nathaniel that the rationale given about the CSP concurrency model seems a bit weak, the author is obviously expressing his opinion there and I won't object to that. However, I think the PEP is desirable for other reasons. Mostly, I hope that by making the subinterpreters functionality available to pure Python programmers (while it was formally an advanced and arcane part of the C API), we will spur of bunch of interesting third-party experimentations, including possibilities that we on python-dev have not thought about. The appeal of the PEP for experimentations is multiple: 1) ability to concurrently run independent execution environments without spawning child processes (which on some platforms and in some situations may not be very desirable: for example on Windows where the cost of spawning is rather high; also, child processes may crash, and sometimes it is not easy for the parent to recover, especially if a synchronization primitive is left in an unexpected state) 2) the potential for parallelizing CPU-bound pure Python code in a single process, if a per-interpreter GIL is finally implemented 3) easier support for sharing large data between separate execution environments, without the hassle of setting up shared memory or the fragility of relying on fork() semantics (and as I said, I hope people find other applications) As for the argument that we already have asyncio and several other packages, I actually think that combining these different concurrency mechanisms would be interesting complex applications (such as distributed systems). For that, however, I think the PEP as currently written is a bit lacking, see below. Now for the detailed comments. * I think the module should indeed be provisional. Experimentation may discover warts that call for a change in the API or semantics. Let's not prevent ourselves from fixing those issues. * The "association" timing seems quirky and potentially annoying: an interpreter only becomes associated with a channel the first time it calls recv() or send(). How about, instead, associating an interpreter with a channel as soon as that channel is given to it through `Interpreter.run(..., channels=...)` (or received through `recv()`)? * How hard would it be, in the current implementation, to add buffering to channels? It doesn't have to be infinite: you can choose a fixed buffer size (or make it configurable in the create() function, which allows passing 0 for unbuffered). Like Nathaniel, I think unbuffered channels will quickly be annoying to work with (yes, you can create a helper thread... now you have one additional thread per channel, which isn't pretty -- especially with the GIL). * In the same vein, I think channels should allow adding readiness callbacks (that are called whenever a channel becomes ready for sending or receiving, respectively). This would make it easy to plug them into an event loop or other concurrency systems (such as Future-based concurrency). Note that each interpreter "associated" with a channel should be able to set its own readiness callback: so one callback per Python object representing the channel, but potentially multiple callbacks for the underlying channel primitive. (how would the callback be scheduled for execution in the right interpreter? perhaps using `_PyEval_AddPendingCall()` or a similar mechanism?) * I think either `interpreters.get_main()` or `interpreters.is_main()` is desirable. Inevitable, the slight differences between main and non-main interpreters will surface in non-trivial applications (finalization issues in distributed systems can really be hairy). It seems this should be mostly costless to provide, so let's do it. * I do think a minimal synchronization primitive would be nice. Either a Lock (in the Python sense) or a Semaphore: both should be relatively easy to provide, by wrapping an OS-level synchronization primitive. Then you can recreate all high-level synchronization primitives, like the threading and multiprocessing modules do (using a Lock or a Semaphore, respectively). (note you should be able to emulate a semaphore using blocking send() and recv() calls, but that's probably not very efficient, and efficiency is important) Of course, I hope these are all actionable before beta1 :-) If not, here is my preferential priority list: * High priority: fix association timing * High priority: either buffering /or/ readiness callbacks * Middle priority: get_main() /or/ is_main() * Middle / low priority: a simple synchronization primitive But I would stress the more of these we provide, the more we encourage people to experiment without pulling too much of their hair. (also, of course, I hope other people read the PEP and emit feedback) Best regards Antoine.

7 29

[RELEASE] Python 3.8.3rc1 is now ready for testing
by Łukasz Langa April 29, 2020

April 29, 2020

Python 3.8.3rc1 is the release candidate of the third maintenance release of Python 3.8. Go get it here: https://www.python.org/downloads/release/python-383rc1/ <https://www.python.org/downloads/release/python-383rc1/> Assuming no critical problems are found prior to 2020-05-11, the scheduled release date for 3.8.3, no code changes are planned between this release candidate and the final release. That being said, please keep in mind that this is a pre-release and as such its main purpose is testing. Maintenance releases for the 3.8 series will continue at regular bi-monthly intervals, with 3.8.4 planned for mid-July 2020. What’s new? The Python 3.8 series is the newest feature release of the Python language, and it contains many new features and optimizations. See the “What’s New in Python 3.8 <https://docs.python.org/3.8/whatsnew/3.8.html>” document for more information about features included in the 3.8 series. Detailed information about all changes made in version 3.8.3 specifically can be found in its change log <https://docs.python.org/release/3.8.3rc1/whatsnew/changelog.html#python-3-8…>. We hope you enjoy Python 3.8! Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

1 0

PEP 554 for 3.9 or 3.10?
by Eric Snow April 29, 2020

April 29, 2020

Hi all, It was nice to speak with many of you at the online language summit this week. It was the next best thing to seeing you all in person! :) With the 3.9 feature freeze coming up I'm considering options for PEP 554. I'm hopeful to have a pronouncement from Antoine in the near future. If that comes in time for 3.9, we will have the implementation ready to go. It is already nearly complete and the remaining work should be done in the next week or so. I'm looking for feedback on a nagging worry I have with merging PEP 554 into 3.9 (if it is accepted) without having the per-interpreter GIL. See below. I'll update the PEP later with relevant info from this discussion. wishing-you-were-all-here-ly yours, -eric +++++++++++++++++++++++++++++++++++++++++++++++++ (++++++++++ Dilemma ============ Many folks have conflated PEP 554 with having a per-interpreter GIL. In fact, I was careful to avoid any mention of parallelism or the GIL in the PEP. Nonetheless some are expecting that when PEP 554 lands we will reach multi-core nirvana. While PEP 554 might be accepted and the implementation ready in time for 3.9, the separate effort toward a per-interpreter GIL is unlikely to be sufficiently done in time. That will likely happen in the next couple months (for 3.10). So...would it be sufficiently problematic for users if we land PEP 554 in 3.9 without per-interpreter GIL? Options ============ Here are the options as I see them (if the PEP is accepted in time for 3.9): 1. merge PEP 554 into 3.9 even if per-interpreter GIL doesn't get into 3.9 (they get parallelism for free in 3.10) 2. like 1, but mark the module as provisional until per-interpreter GIL lands 3. do not merge PEP 554 until per-interpreter GIL is merged 4. like 3, but publish a 3.9-only module to PyPI in the meantime Context ============ Like I said, PEP 554 purposefully does not talk about parallelism or the GIL. Instead it focuses strictly on the following (with an emphasis on minimalism): * exposing the existing C-API * supporting a new concurrency model (CSP) Regarding that first one, the value is in exposing the existing functionality to a broader group of people. keep in mind that subinterpreters currently have various other limitations (aside from sharing the GIL): * nothing about them has been optimized (e.g. interpreter startup, data sharing, data passing) * extension modules with process-global state may break * some bugs still remain in the corners Currently, there is a chicken-and-egg problem. It is unlikely that there will be sufficient motivation to address those limitations until the community starts really using subinterpreters. Hence PEP 554. My Position ============ At this point I think we should go with option #1 (or possibly #2). IMHO, we would be fine merging PEP 554 without per-interpreter GIL. Here's why: * the two objectives of the PEP have value for the community (as explained in the PEP) * the sooner we can get the functionality of subinterpreters out to the broader world, the better * I don't think the lack of parallelism in 3.9 will trip anyone up My only concern is that folks try them out in 3.9, get frustrated by the limitations, and then there is a mental barrier to trying them again in the future when the limitations have been reduced or eliminated. However, I don't think that will be a problem for people that would benefit from serious use of subinterpreters. Furthermore, if we're clear that "for now" subinterpreters still share the GIL in 3.9 then I'm not so worried about folks getting confused. In fact, as long as we're clear about all the current limitations then there isn't much downside to option #1. Marking the module as "provisional" may help communicate that there are limitations that impact use.

16 60

Comments on PEP 554 (Multiple Interpreters in the Stdlib)
by Mark Shannon April 29, 2020

April 29, 2020

Hi, I'm generally in favour of PEP 554, but I don't think it is ready to be accepted in its current form. My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 provides no value over threading or multi-processing. Multi-processing provides true parallelism and threads provide shared memory concurrency. If per-subinterpeter GILs are possible then, and only then, sub-interpreters will provide true parallelism and (limited) shared memory concurrency. The problem is that we don't know whether we can implement per-subinterpeter GILs without too large a negative performance impact. I think we can, but we can't say so for certain. So, IMO, we should not accept PEP 554 until we know for sure that per-subinterpeter GILs can be implemented efficiently. Detailed critique ----------------- I don't see how `list_all()` can be both safe and accurate. The Java equivalent makes no guarantees of accuracy. Attempting to lock the list is likely to lead to deadlock and not locking it will lead to races; potentially dangerous ones. I think it would be best to drop this. `list_all_channels()`. See `list_all()` above. `.destroy()` is either misleading or unsafe. What does this do? >>> is.destroy() >>> is.run() If `run()` raises an exception then the interpreter must exist. Rename to `close()` perhaps? `Channel.interpreters` see `list_all()` and `list_all_channels()` above. How does `is_shareable()` work? Are you proposing some mechanism to transfer an object from one sub-interpreter to another? How would that work? If objects are not shared but serialized, why not use marshal or pickle instead of inventing a third serialization protocol? It would be clearer if channels only dealt with simple, contiguous binary data. As it stands the PEP doesn't state what form the received object will take. Once channels supporting the transfer of bytes objects work, then it is simple to pass more complex objects using pickle or marshal. Channels need a more detailed description of their lifespan. Ideally a state machine. For example: How does an interpreter detach from the receiving end of a channel that is never empty? What happens if an interpreter deletes the last reference to a non-empty channel? On the receiving end, or on the sending end? Isn't the lack of buffering in channels a recipe for deadlocks? What is the mechanism for reliably copying exceptions from one sub-interpreter to another in the `run()` method? If `run()` can raise an exception, why not let it return values? Cheers, Mark.

5 8

[RELEASE] Python 3.9.0a6 is now available for testing
by Łukasz Langa April 29, 2020

April 29, 2020

On behalf of the entire Python development community, and the currently serving Python release team in particular, I’m pleased to announce the release of Python 3.9.0a6. Get it here: https://www.python.org/downloads/release/python-390a6/ <https://www.python.org/downloads/release/python-390a6/> This is an early developer preview of Python 3.9 Python 3.9 is still in development. This release, 3.9.0a6, is the last out of six planned alpha releases. Alpha releases are intended to make it easier to test the current state of new features and bug fixes and to test the release process. During the alpha phase, features may be added up until the start of the beta phase (2020-05-18) and, if necessary, may be modified or deleted up until the release candidate phase (2020-08-10). Please keep in mind that this is a preview release and its use is not recommended for production environments. Major new features of the 3.9 series, compared to 3.8 Many new features for Python 3.9 are still being planned and written. Among the new major new features and changes so far: PEP 584 <https://www.python.org/dev/peps/pep-0584/>, Union Operators in dict PEP 585 <https://www.python.org/dev/peps/pep-0585/>, Type Hinting Generics In Standard Collections PEP 593 <https://www.python.org/dev/peps/pep-0593/>, Flexible function and variable annotations PEP 602 <https://www.python.org/dev/peps/pep-0602/>, Python adopts a stable annual release cadence PEP 616 <https://www.python.org/dev/peps/pep-0616/>, String methods to remove prefixes and suffixes PEP 617 <https://www.python.org/dev/peps/pep-0617/>, New PEG parser for CPython BPO 38379 <https://bugs.python.org/issue38379>, garbage collection does not block on resurrected objects; BPO 38692 <https://bugs.python.org/issue38692>, os.pidfd_open added that allows process management without races and signals; BPO 39926 <https://bugs.python.org/issue39926>, Unicode support updated to version 13.0.0 BPO 1635741 <https://bugs.python.org/issue1635741>, when Python is initialized multiple times in the same process, it does not leak memory anymore A number of Python builtins (range, tuple, set, frozenset, list) are now sped up using PEP 590 <https://www.python.org/dev/peps/pep-0590> vectorcall A number of standard library modules (audioop, ast, grp, _hashlib, pwd, _posixsubprocess, random, select, struct, termios, zlib) are now using the stable ABI defined by PEP 384 <https://www.python.org/dev/peps/pep-0384/>. (Hey, fellow core developer, if a feature you find important is missing from this list, let Łukasz know <mailto:lukasz@python.org>.) The next pre-release, the first beta release of Python 3.9, will be 3.9.0b1. It is currently scheduled for 2020-05-18. Your friendly release team, Ned Deily @nad <https://discuss.python.org/u/nad> Steve Dower @steve.dower <https://discuss.python.org/u/steve.dower> Łukasz Langa @ambv <https://discuss.python.org/u/ambv>

5 5

killing static types (for sub-interpreters?)
by Jim J. Jewett April 29, 2020

April 29, 2020

Why do sub-interpreters require (separate and) heap-allocated types? It seems types that are statically allocated are a pretty good use for immortal objects, where you never change the refcount ... and then I don't see why you need more than one copy.

4 5

Announcement: pip 20.1b1 beta release
by Sumana Harihareswara April 28, 2020

April 28, 2020

On behalf of the PyPA, I am pleased to announce that a beta release of pip, pip 20.1b1, has been released. I hope you will help us by testing this and checking for new bugs in how pip deals with complicated projects and dependencies. The highlights for this release are: * Significant speedups when building local directories, by changing behavior to perform in-place builds, instead of copying to temporary directories. * Significant speedups in `pip list --outdated`, by parallelizing network access. This is the first instance of parallel code within pip's codebase. * A new `pip cache` command, which makes it possible to introspect and manage pip's cache directory. * Better `pip freeze` for packages installed from direct URLs, enabled by the implementation of PEP 610. We would be grateful for all the testing that users could do to ensure that, when pip 20.1 is released, it's as solid as we can make it. You can upgrade to this beta with `python -m pip install -U --pre pip`. This release also contains an alpha version of pip's next generation resolver. It is **off by default** because it is **unstable and not ready for everyday use**. If you're curious about this, please visit this GitHub issue about the resolver, what doesn't work yet, and what kind of testing would help us out https://github.com/pypa/pip/issues/8099 . The full changelog is available. https://pip.pypa.io/en/latest/news/ As with all pip releases, a significant amount of the work was contributed by pip's user community. Huge thanks to all who have contributed, whether through code, documentation, issue reports and/or discussion. Your help keeps pip improving, and is hugely appreciated. Specific thanks go to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for their support that enabled the work on the new resolver. -- Sumana Harihareswara pip project manager under contract with Python Software Foundation Changeset Consulting https://changeset.nyc

3 6

Improvement to SimpleNamespace
by Raymond Hettinger April 28, 2020

April 28, 2020

SimpleNamespace() is really good at giving attribute style-access. I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works: catalog = json.load(f, object_hook=SimpleNamespace) print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict() print(catalog.clothing.mens.shoes.extra_wide.quantity]) # proposed with SimpleNamespace() print(catalog.clothing.boys['3t'].tops.quantity # would also be supported I've already seen something like this in production; however, people are having to write custom subclasses to do it. This is kind of bummer because the custom subclasses are a pain to write, are non-standard, and are generally somewhat slow. I would like to see a high-quality version this made more broadly available. The core idea is keep the simple attribute access but make it easier to load data programmatically: >>> ns = SimpleNamespace(roses='red', violets='blue') >>> thing = input() sugar >>> quality = input() sweet >>> setattr(ns, thing, quality) # current >>> ns['sugar'] = 'sweet' # proposed If the PEP 584 __ior__ method were supported, updating a SimpleNamespace would be much cleaner: ns |= some_dict I posted an issue on the tracker: https://bugs.python.org/issue40284 . There was a suggestion to create a different type for this, but I don't see the point in substantially duplicating everything SimpleNamespace already does just so we can add some supporting dunder methods. Please add more commentary so we can figure-out the best way to offer this powerful functionality. Raymond

14 19

Request For Review: Add support for CAN_J1939 sockets (bpo-40291)
by Karl Ding April 28, 2020

April 28, 2020

Hi all, Could someone take a look at the following PR to add support for CAN_J1939 to the socket module? I'd like to try landing this for 3.9. This enhancement would be useful for anyone working in automotive and/or dealing with the SAE J1939 CAN protocol. This feature is available on Linux 5.4+ (Ubuntu 20.04 LTS ships with a compatible kernel). PR Link: https://github.com/python/cpython/pull/19538 BPO Link: https://bugs.python.org/issue40291 You may find the following links useful if you want to find out more about how the kernel J1939 implementation is used. J1939 Kernel Docs: https://www.kernel.org/doc/html/latest/networking/j1939.html J1939 header: https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/can/j1939… Thanks! -- Karl Ding

2 1

Collecting questions for a Steering Council Q&A
by Victor Stinner April 27, 2020

April 27, 2020

Copy of Brett Cannon's message: https://discuss.python.org/t/collecting-questions-for-a-steering-council-q-… --- As part of PyCon US 2020 (*), we are planning to record a Q&A with the Steering Council. We are using this topic to solicit questions from the community. We plan to leave this open for about a week. Thanks for all of those who leave a question (which adhere to the PSF Code of Conduct )! P.S. Do make sure to check out the various talks that have been posted for the conference so far! And thanks in advance to Ewa, Emily, and the whole PyCon US 2020 staff for making this Q&A possible. (*) https://www.youtube.com/channel/UCMjMBMGt0WJQLeluw6qNJuA --- Please ask your question at: https://discuss.python.org/t/collecting-questions-for-a-steering-council-q-… Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0

Summary of Python tracker Issues
by Python tracker April 24, 2020

April 24, 2020

ACTIVITY SUMMARY (2020-04-17 - 2020-04-24) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7452 ( +9) closed 44675 (+57) total 52127 (+66) Open issues with patches: 2942 Issues opened (50) ================== #34037: asyncio: BaseEventLoop.close() shutdowns the executor without https://bugs.python.org/issue34037 reopened by vstinner #40286: Add randbytes() method to random.Random https://bugs.python.org/issue40286 reopened by rhettinger #40312: Weakref callbacks running before finalizers in GC collection https://bugs.python.org/issue40312 opened by a-feld #40314: python code order of magnitude faster than equivalent CPython https://bugs.python.org/issue40314 opened by deekay #40316: Add zero function to time, datetime, which acts as the use cas https://bugs.python.org/issue40316 opened by Or Toledano #40317: inspect.getsource() examines incorrect target https://bugs.python.org/issue40317 opened by Grzegorz Krasoń #40318: Migrate to SQLite3 trace v2 API https://bugs.python.org/issue40318 opened by erlendaasland #40320: Add ability to specify instance of contextvars context to Task https://bugs.python.org/issue40320 opened by Jeff.Laughlin #40321: urllib.request does not support HTTP response status code 308 https://bugs.python.org/issue40321 opened by Jochem Schulenklopper #40324: python 3.8.2 idle not opening https://bugs.python.org/issue40324 opened by krishandanushka536(a)gmail.com #40327: list(sys.modules.items()) can throw RuntimeError: dictionary c https://bugs.python.org/issue40327 opened by Markus Mohrhard #40328: Add tools for generating mappings_XX.h https://bugs.python.org/issue40328 opened by corona10 #40331: Increase test coverage for the statistics module https://bugs.python.org/issue40331 opened by tzabal #40333: Request for multi-phase initialization API to run code after i https://bugs.python.org/issue40333 opened by indygreg #40334: PEP 617: new PEG-based parser https://bugs.python.org/issue40334 opened by gvanrossum #40337: builtins.RuntimeError: Caught RuntimeError in pin memory threa https://bugs.python.org/issue40337 opened by shawn #40338: [Security] urllib and anti-slash (\) in the hostname https://bugs.python.org/issue40338 opened by vstinner #40339: Instead of skipping, IPV6 test(s) fail on a non-IPV6 machine https://bugs.python.org/issue40339 opened by UnitedMarsupials #40340: Programming FAQ about "How do I convert a string to a number?" https://bugs.python.org/issue40340 opened by Dominik V. #40341: Programming FAQ includes actively discouraged solutions; Shoul https://bugs.python.org/issue40341 opened by Dominik V. #40342: Programming FAQ about "How do I apply a method to a sequence o https://bugs.python.org/issue40342 opened by Dominik V. #40343: Programming FAQ about "How do I call a method defined in a bas https://bugs.python.org/issue40343 opened by Dominik V. #40344: Programming FAQ about "What is the most efficient way to conca https://bugs.python.org/issue40344 opened by Dominik V. #40345: Programming FAQ about "How do I iterate over a sequence in rev https://bugs.python.org/issue40345 opened by Dominik V. #40346: Add random.BaseRandom to ease implementation of subclasses https://bugs.python.org/issue40346 opened by vstinner #40347: Programming FAQ about "How do you remove duplicates from a lis https://bugs.python.org/issue40347 opened by Dominik V. #40348: Programming FAQ about "What is delegation?": Fix typos https://bugs.python.org/issue40348 opened by Dominik V. #40350: modulefinder chokes on numpy - dereferencing None in spec.load https://bugs.python.org/issue40350 opened by Greg Whiteley #40352: SocketHandler silently drops log messages on re-connect https://bugs.python.org/issue40352 opened by Oleg Nykolyn #40353: Add an optional "strict" check to zip https://bugs.python.org/issue40353 opened by brandtbucher #40355: The ast module fails to reject certain malformed nodes https://bugs.python.org/issue40355 opened by brandtbucher #40356: OverflowError: mktime argument out of range https://bugs.python.org/issue40356 opened by darkman66 #40357: asyncio: will shutdown_default_executor work in single step (s https://bugs.python.org/issue40357 opened by cmeyer #40358: pathlib's relative_to should behave like os.path.relpath https://bugs.python.org/issue40358 opened by d.ragusa #40359: email.parse part.get_filename() fails to unwrap long attachmen https://bugs.python.org/issue40359 opened by matt-davis #40360: Deprecate lib2to3 (and 2to3) for future removal https://bugs.python.org/issue40360 opened by gregory.p.smith #40361: Darwin systems using win settings for webbrowser.py https://bugs.python.org/issue40361 opened by Viraat Das #40362: AbstractBasicAuthHandler does not support the following scheme https://bugs.python.org/issue40362 opened by Paul Stoner #40363: shlex.quote applied to a glob pattern disables glob matching https://bugs.python.org/issue40363 opened by geophile #40364: asyncio: replace _compute_returncode() with os.waitstatus_to_e https://bugs.python.org/issue40364 opened by vstinner #40365: argparse: action "extend" with 1 parameter splits strings into https://bugs.python.org/issue40365 opened by strjan #40366: Remove support for obsolete flags in compile function https://bugs.python.org/issue40366 opened by BTaskaya #40367: ImportError: libffi.so.6 https://bugs.python.org/issue40367 opened by YoSTEALTH #40368: os.path.realpath uppercases Windows drive letter on Python 3.8 https://bugs.python.org/issue40368 opened by saschanaz #40371: Deadlock in logging.config.dictConfig https://bugs.python.org/issue40371 opened by mrc #40372: doctest example programs should exit 1 if any test fails https://bugs.python.org/issue40372 opened by AndiDog #40373: urlunparse does not escape slash (/) for http+unix:// in netlo https://bugs.python.org/issue40373 opened by socketpair #40375: Add the UNSELECT command to imaplib https://bugs.python.org/issue40375 opened by eric.smith #40376: documentation for os.getgrouplist potentially misleading https://bugs.python.org/issue40376 opened by woodfighter #40377: APPDATA location in Microsoft Store version https://bugs.python.org/issue40377 opened by stephtr Most recent 15 issues with no replies (15) ========================================== #40377: APPDATA location in Microsoft Store version https://bugs.python.org/issue40377 #40376: documentation for os.getgrouplist potentially misleading https://bugs.python.org/issue40376 #40375: Add the UNSELECT command to imaplib https://bugs.python.org/issue40375 #40373: urlunparse does not escape slash (/) for http+unix:// in netlo https://bugs.python.org/issue40373 #40372: doctest example programs should exit 1 if any test fails https://bugs.python.org/issue40372 #40371: Deadlock in logging.config.dictConfig https://bugs.python.org/issue40371 #40367: ImportError: libffi.so.6 https://bugs.python.org/issue40367 #40357: asyncio: will shutdown_default_executor work in single step (s https://bugs.python.org/issue40357 #40350: modulefinder chokes on numpy - dereferencing None in spec.load https://bugs.python.org/issue40350 #40348: Programming FAQ about "What is delegation?": Fix typos https://bugs.python.org/issue40348 #40347: Programming FAQ about "How do you remove duplicates from a lis https://bugs.python.org/issue40347 #40345: Programming FAQ about "How do I iterate over a sequence in rev https://bugs.python.org/issue40345 #40343: Programming FAQ about "How do I call a method defined in a bas https://bugs.python.org/issue40343 #40331: Increase test coverage for the statistics module https://bugs.python.org/issue40331 #40328: Add tools for generating mappings_XX.h https://bugs.python.org/issue40328 Most recent 15 issues waiting for review (15) ============================================= #40376: documentation for os.getgrouplist potentially misleading https://bugs.python.org/issue40376 #40372: doctest example programs should exit 1 if any test fails https://bugs.python.org/issue40372 #40366: Remove support for obsolete flags in compile function https://bugs.python.org/issue40366 #40361: Darwin systems using win settings for webbrowser.py https://bugs.python.org/issue40361 #40360: Deprecate lib2to3 (and 2to3) for future removal https://bugs.python.org/issue40360 #40358: pathlib's relative_to should behave like os.path.relpath https://bugs.python.org/issue40358 #40346: Add random.BaseRandom to ease implementation of subclasses https://bugs.python.org/issue40346 #40340: Programming FAQ about "How do I convert a string to a number?" https://bugs.python.org/issue40340 #40334: PEP 617: new PEG-based parser https://bugs.python.org/issue40334 #40331: Increase test coverage for the statistics module https://bugs.python.org/issue40331 #40328: Add tools for generating mappings_XX.h https://bugs.python.org/issue40328 #40327: list(sys.modules.items()) can throw RuntimeError: dictionary c https://bugs.python.org/issue40327 #40321: urllib.request does not support HTTP response status code 308 https://bugs.python.org/issue40321 #40318: Migrate to SQLite3 trace v2 API https://bugs.python.org/issue40318 #40316: Add zero function to time, datetime, which acts as the use cas https://bugs.python.org/issue40316 Top 10 most discussed issues (10) ================================= #40334: PEP 617: new PEG-based parser https://bugs.python.org/issue40334 40 msgs #40346: Add random.BaseRandom to ease implementation of subclasses https://bugs.python.org/issue40346 26 msgs #40312: Weakref callbacks running before finalizers in GC collection https://bugs.python.org/issue40312 16 msgs #40275: test.support has way too many imports https://bugs.python.org/issue40275 12 msgs #40301: zipfile module: new feature (two lines of code), useful for te https://bugs.python.org/issue40301 11 msgs #40255: Fixing Copy on Writes from reference counting https://bugs.python.org/issue40255 9 msgs #40286: Add randbytes() method to random.Random https://bugs.python.org/issue40286 9 msgs #39562: Asynchronous comprehensions don't work in asyncio REPL https://bugs.python.org/issue39562 7 msgs #33065: IDLE debugger: failure stepping through module loading https://bugs.python.org/issue33065 6 msgs #40164: Upgrade Windows and macOS installer builds to OpenSSL 1.1.1g https://bugs.python.org/issue40164 6 msgs Issues closed (55) ================== #11654: errors in atexit hooks don't change process exit code https://bugs.python.org/issue11654 closed by glyph #17986: Alternative async subprocesses (pep 3145) https://bugs.python.org/issue17986 closed by vstinner #23117: Properly codesign Mac python 2.7.9.pkg so it can work thru OS https://bugs.python.org/issue23117 closed by ned.deily #23414: seek(count, whence) accepts bogus whence on windows, python2.7 https://bugs.python.org/issue23414 closed by ned.deily #26826: Expose new copy_file_range() syscall in os module. https://bugs.python.org/issue26826 closed by benjamin.peterson #36890: python-3.7.3-macosx10.6.pkg verification error on macOS 10.6 S https://bugs.python.org/issue36890 closed by ned.deily #37465: Incorrect documentation for `s#` arguments in C API argument p https://bugs.python.org/issue37465 closed by inada.naoki #37626: Documentation：conflict between docs https://bugs.python.org/issue37626 closed by inada.naoki #37790: subprocess.Popen() is extremely slow (with close_fds=True whic https://bugs.python.org/issue37790 closed by vstinner #37846: declare that Text I/O use buffer inside https://bugs.python.org/issue37846 closed by inada.naoki #38061: FreeBSD: Optimize subprocess.Popen(close_fds=True) using close https://bugs.python.org/issue38061 closed by vstinner #38329: macOS python.org installers only add or modify framework Versi https://bugs.python.org/issue38329 closed by ned.deily #38360: single-argument form of -isysroot should be supported https://bugs.python.org/issue38360 closed by ned.deily #38784: ip_network does not clear/update the broadcast_address cache w https://bugs.python.org/issue38784 closed by inada.naoki #38891: ShareableList read and write access is O(N), should be O(1) https://bugs.python.org/issue38891 closed by pitrou #39207: concurrent.futures.ProcessPoolExecutor does not properly reap https://bugs.python.org/issue39207 closed by pitrou #39285: PurePath.match indicates case-sensitive nature and presents a https://bugs.python.org/issue39285 closed by pitrou #39471: Meaning and clarification of PyBuffer_Release() https://bugs.python.org/issue39471 closed by seberg #39674: Keep deprecated features in Python 3.9 to ease migration from https://bugs.python.org/issue39674 closed by vstinner #39849: Compiler warninig: warning: variable ‘res’ set but not use https://bugs.python.org/issue39849 closed by corona10 #39932: test_multiprocessing_fork leaked [0, 2, 0] file descriptors on https://bugs.python.org/issue39932 closed by vstinner #39933: test_gdb fails on AMD64 FreeBSD Shared 3.x: ptrace: Operation https://bugs.python.org/issue39933 closed by vstinner #39939: PEP 616: Add str.removeprefix and str.removesuffix methods https://bugs.python.org/issue39939 closed by vstinner #39942: Making instance of `TypeVar` fails because of missing `__name_ https://bugs.python.org/issue39942 closed by levkivskyi #40028: Math module method to find prime factors for non-negative int https://bugs.python.org/issue40028 closed by trrhodes #40048: _PyEval_EvalFrameDefault() doesn't reset tstate->frame if _PyC https://bugs.python.org/issue40048 closed by vstinner #40089: Add _at_fork_reinit() method to locks https://bugs.python.org/issue40089 closed by vstinner #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 closed by vstinner #40138: Windows implementation of os.waitpid() truncates the exit stat https://bugs.python.org/issue40138 closed by vstinner #40148: Add PurePath.with_stem() https://bugs.python.org/issue40148 closed by pitrou #40179: Argument Clinic incorretly translates #elif https://bugs.python.org/issue40179 closed by serhiy.storchaka #40198: macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH d https://bugs.python.org/issue40198 closed by ned.deily #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 closed by steve.dower #40247: Logged out of user when running Tkinter https://bugs.python.org/issue40247 closed by ned.deily #40257: Improve the use of __doc__ in pydoc https://bugs.python.org/issue40257 closed by serhiy.storchaka #40308: Intermittent failure of test_os.TestScandir.test_attributes on https://bugs.python.org/issue40308 closed by Dennis Sweeney #40311: docs.python.org banner - release blocker for 2.7.18 https://bugs.python.org/issue40311 closed by benjamin.peterson #40313: bytes.hex(sep, bytes_per_sep) is many times slower than manual https://bugs.python.org/issue40313 closed by vstinner #40315: Incorrect stacksize in code object https://bugs.python.org/issue40315 closed by skip.montanaro #40319: dict.update() return the updated dict instead of None https://bugs.python.org/issue40319 closed by SilentGhost #40322: struct.pack adding unexpected byte in 'B' format https://bugs.python.org/issue40322 closed by Robert Bressan #40323: Printing of Single (') and Double (") code in one sentence usi https://bugs.python.org/issue40323 closed by SilentGhost #40325: Random.seed does not affect string hash randomization leading https://bugs.python.org/issue40325 closed by rhettinger #40326: A string contains an empty string? https://bugs.python.org/issue40326 closed by serhiy.storchaka #40329: Faulty dict arg gives ValueError but mostly TypeError https://bugs.python.org/issue40329 closed by serhiy.storchaka #40330: ShareableList size guard incorrect for str data https://bugs.python.org/issue40330 closed by pitrou #40332: RegEx for numbers in documentation (easy fix - solution provid https://bugs.python.org/issue40332 closed by serhiy.storchaka #40335: Regression in multiline SyntaxError offsets https://bugs.python.org/issue40335 closed by pablogsal #40336: Refactor typing._SpecialForm https://bugs.python.org/issue40336 closed by serhiy.storchaka #40349: Python3.9 changes col_offset for some ast nodes https://bugs.python.org/issue40349 closed by serhiy.storchaka #40351: How to edit "python --help"? https://bugs.python.org/issue40351 closed by eric.smith #40354: problem when using yield https://bugs.python.org/issue40354 closed by xsmyqf #40369: Use PEP 590 vectorcall to speed up GenericAlias. https://bugs.python.org/issue40369 closed by corona10 #40370: AIX: ld_so_aix not found during test of test_peg_generator https://bugs.python.org/issue40370 closed by pablogsal #40374: collections.abc docs table: Mapping missing __reversed__ https://bugs.python.org/issue40374 closed by rhettinger

1 0

Accepted PEP 615: Support for the IANA Time Zone Database in the Standard Library
by Barry Warsaw April 21, 2020

April 21, 2020

The Python Steering Council accepts PEP 615 - Support for the IANA Time Zone Database in the Standard Library: https://www.python.org/dev/peps/pep-0615/ Congratulations Paul Ganssle! This is a fantastic, well written PEP, and we appreciate Paul’s engagement with the SC to clear up our last questions and concerns. We look forward to its availability in Python 3.9. Thank you Paul, and everyone who helped contribute to this important new module. Cheers, -Barry (on behalf of the Python Steering Council)

4 3

[RELEASE] Python 2.7.18, the end of an era
by Benjamin Peterson April 21, 2020

April 21, 2020

I'm eudaemonic to announce the immediate availability of Python 2.7.18. Python 2.7.18 is a special release. I refer, of course, to the fact that "2.7.18" is the closest any Python version number will ever approximate e, Euler's number. Simply exquisite! A less transcendent property of Python 2.7.18 is that it is the last Python 2.7 release and therefore the last Python 2 release. It's time for the CPython community to say a fond but firm farewell to Python 2. Users still on Python 2 can use e to compute the instantaneously compounding interest on their technical debt. Download this unique, commemorative Python release on python.org: https://www.python.org/downloads/release/python-2718/ Python 2.7 has been under active development since the release of Python 2.6, more than 11 years ago. Over all those years, CPython's core developers and contributors sedulously applied bug fixes to the 2.7 branch, no small task as the Python 2 and 3 branches diverged. There were large changes midway through Python 2.7's life such as PEP 466's feature backports to the ssl module and hash randomization. Traditionally, these features would never have been added to a branch in maintenance mode, but exceptions were made to keep Python 2 users secure. Thank you to CPython's community for such dedication. Python 2.7 was lucky to have the services of two generations of binary builders and operating system experts, Martin von Löwis and Steve Dower for Windows, and Ronald Oussoren and Ned Deily for macOS. The reason we provided binary Python 2.7 releases for macOS 10.9, an operating system obsoleted by Apple 4 years ago, or why the "Microsoft Visual C++ Compiler for Python 2.7" exists is the dedication of these individuals. I thank the past and present Python release managers, Barry Warsaw, Ned Deily, Georg Brandl, Larry Hastings, and Łukasz Langa for their advice and support over the years. I've learned a lot from them—like don't be the sucker who volunteers to manage the release right before a big compatibility break! Python 3 would be nowhere without the critical work of the wider community. Library maintainers followed CPython by maintaining Python 2 support for many years but also threw their weight behind the Python 3 statement (https://python3statement.org). Linux distributors chased Python 2 out of their archives. Users migrated hundreds of millions of lines of code, developed porting guides, and kept Python 2 in their brain while Python 3 gained 10 years of improvements. Finally, thank you to GvR for creating Python 0.9, 1, 2, and 3. Long live Python 3+! Signing off, Benjamin 2.7 release manager

4 3

Re: Accepting PEP 617: New PEG parser for CPython
by Raymond Hettinger April 20, 2020

April 20, 2020

This will be a nice improvement. Raymond

1 0

Re: PEP 616 "String methods to remove prefixes and suffixes" accepted
by Raymond Hettinger April 20, 2020

April 20, 2020

Please consider adding underscores to the names: remove_prefix() and remove_suffix(). The latter method causes a mental hiccup when first read as removes-uffix, forcing mental backtracking to get to remove-suffix. We had a similar problem with addinfourl initially being read as add-in-four-l before mentally backtracking to add-info-url. The PEP says this alternative was considered, but I disagree with the rationale given in the PEP. The reason that "startswith" and "endswith" don't have underscores is that they aren't needed to disambiguate the text. Our rules are to add underscores and to spell-out words when it improves readability, which in this case it does. Like casing conventions, our rules and preferences for naming evolved after the early modules were created -- the older the module, the more likely that it doesn't follow modern conventions. We only have one chance to get this right (bugs can be fixed, but API choices persist for very long time). Take it from someone with experience with this particular problem. I created imap() but later regretted the naming pattern when if came to ifilter() and islice() which sometimes cause mental hiccups initially being read as if-ilter and is-lice. Raymond

1 0

PEP 616 "String methods to remove prefixes and suffixes" accepted
by Victor Stinner April 20, 2020

April 20, 2020

Hi, The Python Steering Council accepts the PEP 616 "String methods to remove prefixes and suffixes": https://www.python.org/dev/peps/pep-0616/ Congrats Dennis Sweeney! We just have one last request: we expect the documentation to explain well the difference between removeprefix()/removesuffix() and lstrip()/strip()/rstrip(), since it is the rationale of the PEP ;-) You can find the WIP implementation at: * https://github.com/python/cpython/pull/18939 * https://bugs.python.org/issue39939 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

3 2

Accepting PEP 617: New PEG parser for CPython
by Brett Cannon April 20, 2020

April 20, 2020

The steering council is happy to announce that we have accepted PEP 617! Thanks to the PEP authors for all their hard work (which includes sending a PR to update the acceptance of the PEP 😉).

2 1

Further PEP 615 Discussion: Equality and hash of ZoneInfo
by Paul Ganssle April 20, 2020

April 20, 2020

A few weeks ago, I submitted PEP 615 to the steering council <https://github.com/python/steering-council/issues/22> for a decision. There's been a decent amount of discussion there with some very good questions. I think they are mostly resolved (though I'm happy to have other people look over the logic of some of my responses there), but Victor brought up the question of the equality and hash behavior of ZoneInfo, and while I'm leaning one way, I could easily be convinced - particularly if people have /real/ use cases that they are planning to implement that would be affected by this. I've summed up the options on the discourse thread <https://discuss.python.org/t/pep-615-support-for-the-iana-time-zone-databas…>, and would appreciate if anyone is able to give some feedback. Thanks! Paul

3 3

Summary of Python tracker Issues
by Python tracker April 17, 2020

April 17, 2020

ACTIVITY SUMMARY (2020-04-10 - 2020-04-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7443 (+10) closed 44618 (+55) total 52061 (+65) Open issues with patches: 2932 Issues opened (40) ================== #25680: Selector.select() hangs when there is nothing to select https://bugs.python.org/issue25680 reopened by gvanrossum #40247: Logged out of user when running Tkinter https://bugs.python.org/issue40247 opened by Jordan #40249: __import__ doesn't honour globals https://bugs.python.org/issue40249 opened by stefan #40250: unable to comment out r'\u' string with triple quote marks https://bugs.python.org/issue40250 opened by Elmar Bucher #40251: selectors.KqueueSelector hangs on EOF, unlike other selectors https://bugs.python.org/issue40251 opened by russelldavis #40252: selectors.KqueueSelector should not be the default selector https://bugs.python.org/issue40252 opened by russelldavis #40253: Fix .py(w) file association with Pyhon 3 Windows installer https://bugs.python.org/issue40253 opened by virtualnobi #40254: pyspecific directives are not translatable https://bugs.python.org/issue40254 opened by cocoatomo #40255: Fixing Copy on Writes from reference counting https://bugs.python.org/issue40255 opened by eelizondo #40256: Python 3.8 Not Launching on Bootcamp Windows 10. https://bugs.python.org/issue40256 opened by YusufM #40257: Improve the use of __doc__ in pydoc https://bugs.python.org/issue40257 opened by serhiy.storchaka #40259: re.Scanner groups https://bugs.python.org/issue40259 opened by dchron #40260: modulefinder traceback regression starting on Windows https://bugs.python.org/issue40260 opened by barry-scott #40261: Build of Python where make is called from subprocess, within a https://bugs.python.org/issue40261 opened by rdbisme #40262: SSL recv_into requires the object to implement __len__ unlike https://bugs.python.org/issue40262 opened by tzickel #40263: ValueError exception on _winapi.WaitForMultipleObjects https://bugs.python.org/issue40263 opened by Ray Donnelly #40269: Inconsistent complex behavior with (-1j) https://bugs.python.org/issue40269 opened by rushilu #40272: ModuleNotFoundEror thrown by system python while accessing it https://bugs.python.org/issue40272 opened by gavin #40273: mappingproxy isn't reversible https://bugs.python.org/issue40273 opened by jack1142 #40275: test.support has way too many imports https://bugs.python.org/issue40275 opened by vstinner #40278: pathlib Path.replace raises OSError when target exists https://bugs.python.org/issue40278 opened by selik #40279: Documentation example of module init function lacks error hand https://bugs.python.org/issue40279 opened by scoder #40280: Consider supporting emscripten/webassembly as a build target https://bugs.python.org/issue40280 opened by Simon Biggs #40283: Documentation of turtle.circle() https://bugs.python.org/issue40283 opened by guchao #40284: Add mapping methods to types.SimpleNamespace https://bugs.python.org/issue40284 opened by rhettinger #40288: atexit module should not be loaded more than once per interpre https://bugs.python.org/issue40288 opened by vstinner #40289: "highlighting" how to get Python's Scripts directory in the do https://bugs.python.org/issue40289 opened by thautwarm #40291: socket library support for CAN_J1939 https://bugs.python.org/issue40291 opened by karlding #40293: Tag libffi build and sources in cpython-source-deps for 3.9.0b https://bugs.python.org/issue40293 opened by indygreg #40296: help(list[int]) fails https://bugs.python.org/issue40296 opened by serhiy.storchaka #40297: test_socket.CANTest is broken at HEAD on master https://bugs.python.org/issue40297 opened by karlding #40300: logging.Formatter crashes when default_msec_format is None. https://bugs.python.org/issue40300 opened by felixxm #40301: zipfile module: new feature (two lines of code), useful for te https://bugs.python.org/issue40301 opened by massimosala #40304: Classes created using type() don't need to explicitly inherit https://bugs.python.org/issue40304 opened by boris #40305: Fix server_close() method for ThreadingHTTPServer and TCPServe https://bugs.python.org/issue40305 opened by ROUX antoine2 #40306: Enhancement request for SSLContext - flag to handle trailing d https://bugs.python.org/issue40306 opened by bli2020 #40307: multiprocessing.BaseManager does not retain Client type. https://bugs.python.org/issue40307 opened by amol #40308: Intermittent failure of test_os.TestScandir.test_attributes on https://bugs.python.org/issue40308 opened by Dennis Sweeney #40309: “unmatched paren” for space before parenthesis in Py_Build https://bugs.python.org/issue40309 opened by Wrzlprmft #40311: docs.python.org banner - release blocker for 2.7.18 https://bugs.python.org/issue40311 opened by sumanah Most recent 15 issues with no replies (15) ========================================== #40309: “unmatched paren” for space before parenthesis in Py_Build https://bugs.python.org/issue40309 #40305: Fix server_close() method for ThreadingHTTPServer and TCPServe https://bugs.python.org/issue40305 #40304: Classes created using type() don't need to explicitly inherit https://bugs.python.org/issue40304 #40297: test_socket.CANTest is broken at HEAD on master https://bugs.python.org/issue40297 #40291: socket library support for CAN_J1939 https://bugs.python.org/issue40291 #40289: "highlighting" how to get Python's Scripts directory in the do https://bugs.python.org/issue40289 #40283: Documentation of turtle.circle() https://bugs.python.org/issue40283 #40279: Documentation example of module init function lacks error hand https://bugs.python.org/issue40279 #40275: test.support has way too many imports https://bugs.python.org/issue40275 #40272: ModuleNotFoundEror thrown by system python while accessing it https://bugs.python.org/issue40272 #40262: SSL recv_into requires the object to implement __len__ unlike https://bugs.python.org/issue40262 #40256: Python 3.8 Not Launching on Bootcamp Windows 10. https://bugs.python.org/issue40256 #40254: pyspecific directives are not translatable https://bugs.python.org/issue40254 #40252: selectors.KqueueSelector should not be the default selector https://bugs.python.org/issue40252 #40251: selectors.KqueueSelector hangs on EOF, unlike other selectors https://bugs.python.org/issue40251 Most recent 15 issues waiting for review (15) ============================================= #40305: Fix server_close() method for ThreadingHTTPServer and TCPServe https://bugs.python.org/issue40305 #40301: zipfile module: new feature (two lines of code), useful for te https://bugs.python.org/issue40301 #40300: logging.Formatter crashes when default_msec_format is None. https://bugs.python.org/issue40300 #40297: test_socket.CANTest is broken at HEAD on master https://bugs.python.org/issue40297 #40291: socket library support for CAN_J1939 https://bugs.python.org/issue40291 #40288: atexit module should not be loaded more than once per interpre https://bugs.python.org/issue40288 #40273: mappingproxy isn't reversible https://bugs.python.org/issue40273 #40269: Inconsistent complex behavior with (-1j) https://bugs.python.org/issue40269 #40263: ValueError exception on _winapi.WaitForMultipleObjects https://bugs.python.org/issue40263 #40260: modulefinder traceback regression starting on Windows https://bugs.python.org/issue40260 #40257: Improve the use of __doc__ in pydoc https://bugs.python.org/issue40257 #40255: Fixing Copy on Writes from reference counting https://bugs.python.org/issue40255 #40254: pyspecific directives are not translatable https://bugs.python.org/issue40254 #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 Top 10 most discussed issues (10) ================================= #40255: Fixing Copy on Writes from reference counting https://bugs.python.org/issue40255 45 msgs #40260: modulefinder traceback regression starting on Windows https://bugs.python.org/issue40260 15 msgs #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 13 msgs #36780: Interpreter exit blocks waiting for futures of shut-down Threa https://bugs.python.org/issue36780 9 msgs #39481: Implement PEP 585 (Type Hinting Generics In Standard Collectio https://bugs.python.org/issue39481 9 msgs #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 8 msgs #40269: Inconsistent complex behavior with (-1j) https://bugs.python.org/issue40269 8 msgs #40284: Add mapping methods to types.SimpleNamespace https://bugs.python.org/issue40284 8 msgs #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 7 msgs #40257: Improve the use of __doc__ in pydoc https://bugs.python.org/issue40257 7 msgs Issues closed (55) ================== #15904: file,close() can fail assert on Windows in 2.7 https://bugs.python.org/issue15904 closed by benjamin.peterson #29255: selects.KqueueSelector behaves incorrectly when no fds are reg https://bugs.python.org/issue29255 closed by gvanrossum #30140: Binary arithmetic does not always call subclasses first https://bugs.python.org/issue30140 closed by gvanrossum #31758: various refleaks in _elementtree, and crashes when using an un https://bugs.python.org/issue31758 closed by scoder #32033: The pwd module implementation incorrectly sets some attributes https://bugs.python.org/issue32033 closed by vstinner #32894: AST unparsing of infinity numbers https://bugs.python.org/issue32894 closed by vstinner #35967: Better platform.processor support https://bugs.python.org/issue35967 closed by jaraco #38644: Pass explicitly tstate to function calls https://bugs.python.org/issue38644 closed by vstinner #39011: ElementTree attributes replace "\r" with "\n" https://bugs.python.org/issue39011 closed by scoder #39380: ftplib uses latin-1 as default encoding https://bugs.python.org/issue39380 closed by vstinner #39522: AST Unparser with unicode kinded constants https://bugs.python.org/issue39522 closed by pablogsal #39667: Update zipfile.Path with zipp 3.0 https://bugs.python.org/issue39667 closed by jaraco #39793: make_msgid fail on FreeBSD 12.1-RELEASE-p1 with different doma https://bugs.python.org/issue39793 closed by BTaskaya #39865: getattr silences an unrelated AttributeError https://bugs.python.org/issue39865 closed by rhettinger #39894: `pathlib.Path.samefile()` calls `os.stat()` without using acce https://bugs.python.org/issue39894 closed by pitrou #39897: `pathlib.Path.is_mount()` calls `Path(self.parent)` and theref https://bugs.python.org/issue39897 closed by pitrou #39901: `pathlib.Path.owner()` and `group()` use `pwd` and `grp` modul https://bugs.python.org/issue39901 closed by pitrou #39953: Let's update ssl error codes https://bugs.python.org/issue39953 closed by benjamin.peterson #40091: Crash in logging._after_at_fork_child_reinit_locks() https://bugs.python.org/issue40091 closed by vstinner #40129: Add test classes for custom __index__, __int__, __float__ and https://bugs.python.org/issue40129 closed by serhiy.storchaka #40197: Add nanoseconds to timing table in What's new python 3.8 https://bugs.python.org/issue40197 closed by rhettinger #40208: Remove deprecated symtable.SymbolTable.has_exec https://bugs.python.org/issue40208 closed by corona10 #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 closed by pablogsal #40218: sys.executable is a false path if python is executed from gdb https://bugs.python.org/issue40218 closed by terry.reedy #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 closed by corona10 #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 closed by rhettinger #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 closed by corona10 #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 closed by serhiy.storchaka #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 closed by vstinner #40246: Different error messages for same error - invalid string prefi https://bugs.python.org/issue40246 closed by pablogsal #40248: Proposed class for collections: dynamicdict https://bugs.python.org/issue40248 closed by steven.daprano #40258: random module does not have type hints https://bugs.python.org/issue40258 closed by rhettinger #40264: List item inside tuple seemingly allows for item assignment ev https://bugs.python.org/issue40264 closed by eric.smith #40265: argparse.Namespace __repr__ does not handle reserved keywords https://bugs.python.org/issue40265 closed by rhettinger #40266: Failure to build _ssl module on ubuntu xenial https://bugs.python.org/issue40266 closed by SilentGhost #40267: Error message differs when an expression is in an fstring https://bugs.python.org/issue40267 closed by gvanrossum #40268: Reorganize pycore_pystate.h header https://bugs.python.org/issue40268 closed by vstinner #40270: activate (or include) json1 extension in sqlite https://bugs.python.org/issue40270 closed by zach.ware #40271: Allow shell like paths in https://bugs.python.org/issue40271 closed by serhiy.storchaka #40274: 3D plotting library doesn't occlude objects by depth/perspecti https://bugs.python.org/issue40274 closed by eric.smith #40276: Make member objects inspectable. https://bugs.python.org/issue40276 closed by rhettinger #40277: Improve repr for _tuplegetter objects https://bugs.python.org/issue40277 closed by rhettinger #40281: Add pathlib.PurePath.as_uri() https://bugs.python.org/issue40281 closed by pitrou #40282: random.getrandbits(0) should succeed https://bugs.python.org/issue40282 closed by pitrou #40286: Add randbytes() method to random.Random https://bugs.python.org/issue40286 closed by vstinner #40287: SpooledTemporaryFile.seek returns None https://bugs.python.org/issue40287 closed by inada.naoki #40290: Add zscore to statistics.NormalDist https://bugs.python.org/issue40290 closed by rhettinger #40292: Memory leak when defining a new class inside a loop https://bugs.python.org/issue40292 closed by serhiy.storchaka #40294: Use-after-free crash if multiple interpreters import asyncio m https://bugs.python.org/issue40294 closed by vstinner #40295: doctest handling of multiline strings is broken https://bugs.python.org/issue40295 closed by steven.daprano #40298: Type annotation objects (Tuple, List, etc.) register as callab https://bugs.python.org/issue40298 closed by serhiy.storchaka #40299: os.dup seems broken with execvp (LINUX) https://bugs.python.org/issue40299 closed by krsna #40302: Add pycore_byteswap.h internal header file with _Py_bswap32() https://bugs.python.org/issue40302 closed by vstinner #40303: argparse parse_args args parameter bug or intended https://bugs.python.org/issue40303 closed by Gharg #40310: If use element from for in while loop it will have bad iterate https://bugs.python.org/issue40310 closed by Maks Bleo

1 0

The Python 2 death march
by Benjamin Peterson April 16, 2020

April 16, 2020

Hi all, It's finally time to schedule the last releases in Python 2's life. There will be two more releases of Python 2.7: Python 2.7.17 and Python 2.7.18. Python 2.7.17 release candidate 1 will happen on October 5th followed by the final release on October 19th. I'm going to time Python 2.7.18 to coincide with PyCon 2020 in April, so attendees can enjoy some collective catharsis. We'll still say January 1st is the official EOL date. Thanks to Sumana Harihareswara, there's now a FAQ about the Python 2 sunset on the website: https://www.python.org/doc/sunset-python-2/ Regards, Benjamin

24 38

Re: Improvement to SimpleNamespace
by Raymond Hettinger April 15, 2020

April 15, 2020

[Serhiy] > As a workaround you can use > > object_hook=lambda x: SimpleNamespace(**x) That doesn't suffice because some valid JSON keys are not valid identifiers. You still need a way to get past those when they arise: catalog.books.fiction['Paradise Lost'].isbn Also, it still leaves you with using setattr(ns, attrname, attrvalue) or tricks with vars() when doing updates. The AttrDict recipe is popular for a reason. Raymond

3 2

PEP: Modify the C API to hide implementation details
by Victor Stinner April 15, 2020

April 15, 2020

Hi, Here is a first draft a PEP which summarize the research work I'm doing on CPython C API since 2017 and the changes that me and others already made since Python 3.7 towards an "opaque" C API. The PEP is also a collaboration with developers of PyPy, HPy, Rust-CPython and many others! Thanks to everyone who helped me to write it down! Maybe this big document should be reorganized as multiple smaller better defined goals: as multiple PEPs. The PEP is quite long and talks about things which are not directly related. It's a complex topic and I chose to put everything as a single document to have a good starting point to open the discussion. I already proposed some of these ideas in 2017: see the Prior Art section ;-) The PEP can be read on GitHub where it's better formatted: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst If someone wants to work on the PEP itself, the document on GitHub is the current reference. Victor ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PEP xxx: Modify the C API to hide implementation details ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Abstract ======== * Hide implementation details from the C API to be able to `optimize CPython`_ and make PyPy more efficient. * The expectation is that `most C extensions don't rely directly on CPython internals`_ and so will remain compatible. * Continue to support old unmodified C extensions by continuing to provide the fully compatible "regular" CPython runtime. * Provide a `new optimized CPython runtime`_ using the same CPython code base: faster but can only import C extensions which don't use implementation details. Since both CPython runtimes share the same code base, features implemented in CPython will be available in both runtimes. * `Stable ABI`_: Only build a C extension once and use it on multiple Python runtimes and different versions of the same runtime. * Better advertise alternative Python runtimes and better communicate on the differences between the Python language and the Python implementation (especially CPython). Note: Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. Rationale ========= To remain competitive in term of performance with other programming languages like Go or Rust, Python has to become more efficient. Make Python (at least) two times faster --------------------------------------- The C API leaks too many implementation details which prevent optimizing CPython. See `Optimize CPython`_. PyPy's support for Python's C API (pycext) is slow because it has to emulate CPython internals like memory layout and reference counting. The emulation causes memory overhead, memory copies, conversions, etc. See `Inside cpyext: Why emulating CPython C API is so Hard <https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c…>`_ (Sept 2018) by Antonio Cuni. While this PEP may make CPython a little bit slower in the short term, the long-term goal is to make "Python" at least two times faster. This goal is not hypothetical: PyPy is already 4.2x faster than CPython and is fully compatible. C extensions are the bottleneck of PyPy. This PEP proposes a migration plan to move towards opaque C API which would make PyPy faster. Separated the Python language and the CPython runtime (promote alternative runtimes) ------------------------------------------------------------------------------------ The Python language should be better separated from its runtime. It's common to say "Python" when referring to "CPython". Even in this PEP :-) Because the CPython runtime remains the reference implementation, many people believe that the Python language itself has design flaws which prevent it from being efficient. PyPy proved that this is a false assumption: on average, PyPy runs Python code 4.2 times faster than CPython. One solution for separating the language from the implementation is to promote the usage of alternative runtimes: not only provide the regular CPython, but also PyPy, optimized CPython which is only compatible with C extensions using the limited C API, CPython compiled in debug mode to ease debugging issues in C extensions, RustPython, etc. To make alternative runtimes viable, they should be competitive in term of features and performance. Currently, C extension modules remain the bottleneck for PyPy. Most C extensions don't rely directly on CPython internals ---------------------------------------------------------- While the C API is still tidely coupled to CPython internals, in practical, most C extensions don't rely directly on CPython internals. The expectation is that these C extensions will remain compatible with an "opaque" C API and only a minority of C extensions will have to be modified. Moreover, more and more C extensions are implemented in Cython or cffi. Updating Cython and cffi to be compatible with the opaque C API will make all these C extensions without having to modify the source code of each extension. Stable ABI ---------- The idea is to build a C extension only once: the built binary will be usable on multiple Python runtimes and different versions of the same runtime (stable ABI). The idea is not new but is an extension of the `PEP 384: Defining a Stable ABI <https://www.python.org/dev/peps/pep-0384/>`__ implemented in CPython 3.4 with its "limited C API". The limited API is not used by default and is not widely used: PyQt is one of the only few known users. The idea here is that the default C API becomes the limited C API and so all C extensions will benefit of advantages of a stable ABI. Flaws of the C API ================== Borrowed references ------------------- A borrowed reference is a pointer which doesn't “hold” a reference. If the object is destroyed, the borrowed reference becomes a dangling pointer, pointing to freed memory which might be reused by a new object. Borrowed references can lead to bugs and crashes when misused. An example of a CPython bug caused by this is `bpo-25750: crash in type_getattro() <https://bugs.python.org/issue25750>`_. Borrowed references are a problem whenever there is no reference to borrow: they assume that a referenced object already exists (and thus has a positive reference count). Tagged pointers are an example of this problem: since there is no concrete ``PyObject*`` to represent the integer, it cannot easily be manipulated. This issue complicates optimizations like PyPy's list strategies: if a list contains only small integers, it is stored as a compact C array of longs. The equivalent of ``PyObject`` is only created when an item is accessed. (Most of the time the object is optimized away by the JIT, but this is another story.) This makes it hard to support the C API function ``PyList_GetItem()``, which should return a reference borrowed from the list, but the list contains no concrete ``PyObject`` that it could lend a reference to! PyPy's current solution is very bad: the first time ``PyList_GetItem()`` is called, the whole list is de-optimized (converted to a list of ``PyObject*``). See ``cpyext`` ``get_list_storage()``. See also the Specialized list use case, which is the same optimization applied to CPython. Like in PyPy, this optimization is incompatible with borrowed references since the runtime cannot guess when the temporary object should be destroyed. If ``PyList_GetItem()`` returned a strong reference, the ``PyObject*`` could just be allocated on the fly and destroyed when the user decrements its reference count. Basically, by putting borrowed references in the API, we are making it impossible to change the underlying data structure. Functions stealing strong references ------------------------------------ There are functions which steal strong references, for example ``PyModule_AddObject()`` and ``PySet_Discard()``. Stealing references is an issue similar to borrowed references. PyObject** ---------- Some functions of the C API return a pointer to an array of ``PyObject*``: * ``PySequence_Fast_ITEMS()`` * ``PyTuple_GET_ITEM()`` is sometimes abused to get an array of all of the tuple's contents: ``PyObject **items = &PyTuple_GET_ITEM(0);`` In effect, these functions return an array of borrowed references: like with ``PyList_GetItem()``, all callers of ``PySequence_Fast_ITEMS()`` assume the sequence holds references to its elements. Leaking structure members ------------------------- ``PyObject``, ``PyTypeObject``, ``PyThreadState``, etc. structures are currently public: C extensions can directly read and modify the structure members. For example, the ``Py_INCREF()`` macro directly increases ``PyObject.ob_refcnt``, without any abstraction. Hopefully, ``Py_INCREF()`` implementation can be modified without affecting the API. Change the C API ================ This PEP doesn't define an exhaustive list of all C API changes, but define some guidelines of bad patterns which should be avoided in the C API to prevent leaking implementation details. Separate header files of limited and internal C API --------------------------------------------------- In Python 3.6, all headers (.h files) were directly in the ``Include/`` directory. In Python 3.7, work started to move the internal C API into a new subdirectory, ``Include/internal/``. The work continued in Python 3.8 and 3.9. The internal C API is only partially exported: some functions are only declared with ``extern`` and so cannot be used outside CPython (with compilers supporting ``-fvisibility=hidden``, see above), whereas some functions are exported with ``PyAPI_FUNC()`` to make them usable in C extensions. Debuggers and profilers are typical users of the internal C API to inspect Python internals without calling functions (to inspect a coredump for example). Python 3.9 is now built with ``-fvisibility=hidden`` (supported by GCC and clang): symbols which are not declared with ``PyAPI_FUNC()`` or ``PyAPI_DATA()`` are no longer exported by the dynamical library (libpython). Another change is to separate the limited C API from the "CPython" C API: Python 3.8 has a new ``Include/cpython/`` sub-directory. It should not be used directly, but it is used automatically from the public headers when the ``Py_LIMITED_API`` macro is not defined. **Backward compatibility:** fully backward compatible. **Status:** basically completed in Python 3.9. Changes without API changes and with minor performance overhead --------------------------------------------------------------- * Replace macros with static inline functions. Work started in 3.8 and made good progress in Python 3.9. * Modify macros to avoid directly accessing structures fields. For example, the `Hide implementation detail of trashcan macros <https://github.com/python/cpython/commit/38965ec5411da60d312b59be281f3510d5…>`_ commit modifies ``Py_TRASHCAN_BEGIN_CONDITION()`` macro to call a new ``_PyTrash_begin()`` function rather than accessing directly ``PyThreadState.trash_delete_nesting`` field. **Backward compatibility:** fully backward compatible. **Status:** good progress in Python 3.9. Changes without API changes but with performance overhead --------------------------------------------------------- Replace macros or inline functions with regular functions. Work started in 3.9 on a limited set of functions. Converting macros to function calls can have a small overhead on performances. For example, ``Py_INCREF()`` macro modifies directly ``PyObject.ob_refcnt``: this macro would become an alias to the opaque ``Py_IncRef()`` function. It is possible that the regular CPython runtime keeps the ``Py_INCREF()`` macro which modifies directly ``PyObject.ob_refcnt`` to avoid any performance overhead. A tradeoff should be defined to limit differences between the regular and the new optimized CPython runtimes, without hurting too much performances of the regular CPython runtime. **Backward compatibility:** fully backward compatible. **Status:** not started. The performance overhead must be measured with benchmarks and this PEP should be accepted. API and ABI incompatible changes -------------------------------- * Make structures opaque: move them to the internal C API. * Remove functions from the public C API which are tied to CPython internals. Maybe begin by marking these functions as private (rename ``PyXXX`` to ``_PyXXX``) or move them to the internal C API. * Ban statically allocated types (by making ``PyTypeObject`` opaque): enforce usage of ``PyType_FromSpec()``. Examples of issues to make structures opaque: * ``PyGC_Head``: https://bugs.python.org/issue40241 * ``PyObject``: https://bugs.python.org/issue39573 * ``PyTypeObject``: https://bugs.python.org/issue40170 * ``PyThreadState``: https://bugs.python.org/issue39573 Another example are ``Py_REFCNT()`` and ``Py_TYPE()`` macros which can currently be used l-value to modify an object reference count or type. Python 3.9 has new ``Py_SET_REFCNT()`` and ``Py_SET_TYPE()`` macros which should be used instead. ``Py_REFCNT()`` and ``Py_TYPE()`` macros should be converted to static inline functions to prevent their usage as l-value. **Backward compatibility:** backward incompatible on purpose. Break the limited C API and the stable ABI, with the assumption that `Most C extensions don't rely directly on CPython internals`_ and so will remain compatible. CPython specific behavior ========================= Some C functions and some Python functions have a behavior which is closely tied to the current CPython implementation. is operator ----------- The "x is y" operator is closed tied to how CPython allocates objects and to ``PyObject*``. For example, CPython uses singletons for numbers in [-5; 256] range:: >>> x=1; (x + 1) is 2 True >>> x=1000; (x + 1) is 1001 False Python 3.8 compiler now emits a ``SyntaxWarning`` when the right operand of the ``is`` and ``is not`` operators is a literal (ex: integer or string), but don't warn if it is ``None``, ``True``, ``False`` or ``Ellipsis`` singleton (`bpo-34850 <https://bugs.python.org/issue34850>`_). Example:: >>> x=1 >>> x is 1 <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="? True CPython PyObject_RichCompareBool -------------------------------- CPython considers that two objects are identical if their memory address are equal: ``x is y`` in Python (``IS_OP`` opcode) is implemented internally in C as ``left == right`` where ``left`` and ``right`` are ``PyObject *`` pointers. The main function to implement comparison in CPython is ``PyObject_RichCompareBool()``. This function considers that two objects are equal if the two ``PyObject*`` pointers are equal (if the two objects are "identical"). For example, ``PyObject_RichCompareBool(obj1, obj2, Py_EQ)`` doesn't call ``obj1.__eq__(obj2)`` if ``obj1 == obj2`` where ``obj1`` and ``obj2`` are ``PyObject*`` pointers. This behavior is an optimization to make Python more efficient. For example, the ``dict`` lookup avoids ``__eq__()`` if two pointers are equal. Another example are Not-a-Number (NaN) floating pointer numbers which are not equal to themselves:: >>> nan = float("nan") >>> nan is nan True >>> nan == nan False The ``list.__contains__(obj)`` and ``list.index(obj)`` methods are implemented with ``PyObject_RichCompareBool()`` and so rely on objects identity:: >>> lst = [9, 7, nan] >>> nan in lst True >>> lst.index(nan) 2 >>> lst[2] == nan False In CPython, ``x == y`` is implemented with ``PyObject_RichCompare()`` which don't make the assumption that identical objects are equal. That's why ``nan == nan`` or ``lst[2] == nan`` return ``False``. Issues for other Python implementations --------------------------------------- The Python language doesn't require to be implemented with ``PyObject`` structure and use ``PyObject*`` pointers. PyPy doesn't use ``PyObject`` nor ``PyObject*``. If CPython is modified to use `Tagged Pointers`_, CPython would have the same issue. Alternative Python implementations have to mimick CPython to reduce incompatibilities. For example, PyPy mimicks CPython behavior for the ``is`` operator with CPython small integer singletons:: >>>> x=1; (x + 1) is 2 True It also mimicks CPython ``PyObject_RichCompareBool()``. Example with the Not-a-Number (NaN) float:: >>>> nan=float("nan") >>>> nan == nan False >>>> lst = [9, 7, nan] >>>> nan in lst True >>>> lst.index(nan) 2 >>>> lst[2] == nan False Better advertise alternative Python runtimes ============================================ Currently, PyPy and other "alternative" Python runtimes are not well advertised on the `Python website <https://www.python.org/>`_. They are only listed as the last choice in the Download menu. Once enough C extensions will be compatible with the limited C API, PyPy and other Python runtimes should be better advertised on the Python website and in the Python documentation, to no longer introduce them as as first-class citizen. Obviously, CPython is likely to remain the most feature-complete implementation in mid-term, since new PEPs are first implemented in CPython. Limitations can be simply documented, and users should be free to make their own choice, depending on their use cases. HPy project =========== The `HPy project <https://github.com/pyhandle/hpy>`__ is a brand new C API written from scratch. It is designed to ease migration from the current C API and to be efficient on PyPy. HPy hides all implementation details: it is based on "handles" so objects cannot be inspected with direct memory access: only opaque function calls are allowed. This abstraction has many benefits: * No more ``PyObject`` emulation needed: smaller memory footprint in PyPy cpyext, no more expensive conversions. * It is possible to have multiple handles pointing to the same object. It helps to better track the object lifetime and makes the PyPy implementation easier. PyPy doesn't use reference counting but a tracing garbage collector. When the PyPy GC moves objects in memory, handles don't change! HPy uses an array mapping handle to objects: only this array has to be updated. It is way more efficient. * The Python runtime is free to modify deep internals compared to CPython. Many optimizations become possible: see `Optimize CPython`_ section. * It is easy to add a debug wrapper to add checks before and after the function calls. For example, ensure that that GIL is held when calling CPython. HPy is developed outside CPython, is implemented on top of the existing Python C API, and so can support old Python versions. By default, binaries compiled in "universal" HPy ABI mode can be used on CPython and PyPy. HPy can also target CPython ABI which has the same performance than native C extensions. See HPy documentation of `Target ABIs documentation <https://github.com/pyhandle/hpy/blob/feature/improve-docs/docs/overview.rst…>`_. The PEP moves the C API towards HPy design and API. New optimized CPython runtime ============================== Backward incompatible changes is such a pain for the whole Python community. To ease the migration (accelerate adoption of the new C API), one option is to provide not only one but two CPython runtimes: * Regular CPython: fully backward compatible, support direct access to structures like ``PyObject``, etc. * New optimized CPython: incompatible, cannot import C extensions which don't use the limited C API, has new optimizations, limited to the C API. Technically, both runtimes would have the same code base, to ease maintenance: CPython. The new optimized CPython would be a ./configure flag to build a different Python. On Windows, it would be a different project of the Visual Studio solution reusing pythoncore project, but define a macro to build enable optimization and change the C API. The new optimized CPython runtime remains compatible with CPython 3.8 `stable ABI`_. CPython code base remains 30 years old. Many technical choices made 30 years ago are no longer relevant today. This PEP should ease the development of new Python implementation which would be even more efficient, like PyPy! Cython and cffi =============== Cython and cffi should be preferred to write new C extensions. This PEP is about existing C extensions which cannot be rewritten with Cython. Cython may be modified to add a new build mode where only the "limited C API" is used. Use Cases ========= Optimize CPython ---------------- The new optimized runtime can implement new optimizations since it only supports C extension modules which don't access Python internals. Tagged pointers ............... `Tagged pointer <https://en.wikipedia.org/wiki/Tagged_pointer>`_. Avoid ``PyObject`` for small objects (ex: small integers, short Latin-1 strings, None and True/False singletons): store the content directly in the pointer, with a tag for the object type. Tracing garbage collector ......................... Experiment with a tracing garbage collector inside CPython. Keep reference counting for the C API. Rewriting CPython with a tracing garbage collector is large project which is out of the scope of this PEP. This PEP fix some blockers issues which prevent to start such project today. One of the issue are functions of the C API which return a pointer like ``PyBytes_AsString()``. Python doesn't know when the caller stops using the pointer, and so cannot move the object in memory (for a moving garbage collector). API like ``PyBuffer`` is better since it requires the caller to call ``PyBuffer_Release()`` when it is done. Specialized list ................ Specialize lists of small integers: if a list only contains numbers which fit into a C ``int32_t``, a Python list object could use a more efficient ``int32_t`` array to reduce the memory footprint (avoid ``PyObject`` overhead for these numbers). Temporary ``PyObject`` objects would be created on demand for backward compatibility. This optimization is less interesting if tagged pointers are implemented. PyPy already implements this optimization. O(1) bytearray to bytes conversion .................................. Convert bytearray to bytes without memory copy. Currently, bytearray is used to build a bytes string, but it's usually converted into a bytes object to respect an API. This conversion requires to allocate a new memory block and copy data (O(n) complexity). It is possible to implement O(1) conversion if it would be possible to pass the ownership of the bytearray object to bytes. That requires modifying the ``PyBytesObject`` structure to support multiple storages (support storing content into a separate memory block). Fork and "Copy-on-Read" problem ............................... Solve the "Copy on read" problem with fork: store reference counter outside ``PyObject``. Currently, when a Python object is accessed, its ``ob_refcnt`` member is incremented temporarily to hold a "strong reference" to it (ensure that it cannot be destroyed while we use it). Many operating system implement fork() using copy-on-write ("CoW"). A memory page (ex: 4 KB) is only copied when a process (parent or child) modifies it. After Python is forked, modifying ``ob_refcnt`` copies the memory page, even if the object is only accessed in "read only mode". `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-i…>`_ (Jan 2017) by Instagram Engineering. Instagram contributed `gc.freeze() <https://docs.python.org/dev/library/gc.html#gc.freeze>`_ to Python 3.7 which works around the issue. One solution for that would be to store reference counters outside ``PyObject``. For example, in a separated hash table (pointer to reference counter). Changing ``PyObject`` structures requires that C extensions don't access them directly. Debug runtime and remove debug checks in release mode ..................................................... If the C extensions are no longer tied to CPython internals, it becomes possible to switch to a Python runtime built in debug mode to enable runtime debug checks to ease debugging C extensions. If using such a debug runtime becomes harder, indirectly it means that runtime debug checks can be removed from the release build. CPython code base is still full of runtime checks calling ``PyErr_BadInternalCall()`` on failure. Removing such checks in release mode can make Python more efficient. PyPy ---- ujson is 3x faster on PyPy when using HPy instead of the Python C API. See `HPy kick-off sprint report <https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.html>`_ (December 2019). This PEP should help to make PyPy cpyext more efficient, or at least ease the migration of C extensions to HPy. GraalPython ----------- `GraalPython <https://github.com/graalvm/graalpython>`_ is a Python 3 implementation built on `GraalVM <https://www.graalvm.org/>`_ ("Universal VM for a polyglot world"). It is interested in supporting HPy. See `Leysin 2020 Sprint Report <https://morepypy.blogspot.com/2020/03/leysin-2020-sprint-report.html>`_. It would also benefit of this PEP. RustPython, Rust-CPython and PyO3 --------------------------------- Rust-CPython is interested in supporting HPy. See `Leysin 2020 Sprint Report <https://morepypy.blogspot.com/2020/03/leysin-2020-sprint-report.html>`_. RustPython and PyO3 would also benefit of this PEP. Links: * `PyO3 <https://github.com/PyO3/pyo3>`_: Rust bindings for the Python (CPython) interpreter * `rust-cpython <https://github.com/dgrunwald/rust-cpython>`_: Rust <-> Python (CPython) bindings * `RustPython <https://github.com/RustPython/RustPython>`_: A Python Interpreter written in Rust Rejected Ideas ============== Drop the C API -------------- One proposed alternative to a new better C API is to drop the C API at all. The reasoning is that since existing solutions are already available, complete and reliable, like Cython and cffi. What about the long tail of C extensions on PyPI which still use the C API? Would a Python without these C extensions would remain relevant? Lots of project do not use those solution, and the C API is part of Python success. For example, there would be no numpy without the C API. It doesn't sound like a workable solution. Bet on HPy, leave the C API unchanged ------------------------------------- The HPy project is developed outside CPython and so doesn't cause any backward incompatibility in CPython. HPy API was designed with efficiency in mind. The problem is the long tail of C extensions on PyPI which are written with the C API and will not be converted soon or will never be converted to HPy. The transition from Python 2 to Python 3 showed that migrations are very slow and never fully complete. The PEP also rely on the assumption that `Most C extensions don't rely directly on CPython internals`_ and so will remain compatible with the new opaque C API. The concept of HPy is not new: CPython has a limited C API which provides a stable ABI since Python 3.4, see `PEP 384: Defining a Stable ABI <https://www.python.org/dev/peps/pep-0384/>`_. Since it is an opt-in option, most users simply use the **default** C API. Prior Art ========= * `pythoncapi.readthedocs.io <https://pythoncapi.readthedocs.io/>`_: Research project behind this PEP * July 2019: Keynote `Python Performance: Past, Present, Future <https://github.com/vstinner/talks/raw/master/2019-EuroPython/python_perform…>`_ (slides) by Victor Stinner at EuroPython 2019 * [python-dev] `Make the stable API-ABI usable <https://mail.python.org/pipermail/python-dev/2017-November/150607.html>`_ (November 2017) by Victor Stinner * [python-ideas] `PEP: Hide implementation details in the C API <https://mail.python.org/pipermail/python-ideas/2017-July/046399.html>`_ (July 2017) by Victor Stinner. Old PEP draft which proposed to add an option to build C extensions. * `A New C API for CPython <https://vstinner.github.io/new-python-c-api.html>`_ (Sept 2017) article by Victor Stinner * `Python Performance <https://github.com/vstinner/conf/raw/master/2017-PyconUS/summit.pdf>`_ (May 2017 at the Language Summit) by Victor Stinner: early discusssions on reorganizing header files, promoting PyPy, fix the C API, etc. Discussion summarized in `Keeping Python competitive <https://lwn.net/Articles/723949/>`_ article. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -- Night gathers, and now my watch begins. It shall not end until my death.

16 43

Providing fix for modulefinder.py regression on Windows
by Barry Scott April 15, 2020

April 15, 2020

modulefinder.py does not open source files in "rb" which prevents compile() from applying the encoding rules. This first showed up for me on Windows with Python 3.8. I'd like to provide patches for 3.8 (if any more releases) and 3.9. What is the process I should follow? I have signed the Contributor Agreement. Here is my test case and the results on Windows with 3.7. import modulefinder with open('import_functools.py', 'w') as f: f.write('import functools\n') mf = modulefinder.ModuleFinder() mf.run_script('import_functools.py') print('Test passed') py -3.8-32 modulefinder_test.py Traceback (most recent call last): File "modulefinder_test.py", line 7, in <module> mf.run_script('import_functools.py') File "C:\Python38.win32\lib\modulefinder.py", line 165, in run_script self.load_module('__main__', fp, pathname, stuff) File "C:\Python38.win32\lib\modulefinder.py", line 360, in load_module self.scan_code(co, m) File "C:\Python38.win32\lib\modulefinder.py", line 433, in scan_code self._safe_import_hook(name, m, fromlist, level=0) File "C:\Python38.win32\lib\modulefinder.py", line 378, in _safe_import_hook self.import_hook(name, caller, level=level) File "C:\Python38.win32\lib\modulefinder.py", line 177, in import_hook q, tail = self.find_head_package(parent, name) File "C:\Python38.win32\lib\modulefinder.py", line 233, in find_head_package q = self.import_module(head, qname, parent) File "C:\Python38.win32\lib\modulefinder.py", line 326, in import_module m = self.load_module(fqname, fp, pathname, stuff) File "C:\Python38.win32\lib\modulefinder.py", line 343, in load_module co = compile(fp.read()+'\n', pathname, 'exec') File "C:\Python38.win32\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 308: character maps to <undefined> Barry

5 6

Do we need port some extension modules to the modules directory?
by Hai Shi April 13, 2020

April 13, 2020

Hi, I notice some modules not in modules directory(for example: _warnings、marshal in python directory). Do we need port those modules to modules directory? Sorry for my noisy info. Bests Wishs, Hai Shi

3 4

Python2 as 𝑣 → 𝑒
by Mike Miller April 12, 2020

April 12, 2020

Unless I've read something wrong, it looks like the final Python 2 release (2.7.18) should approximate the math constant e: >>> import math >>> math.e 2.718281828459045 Aka: Python as 𝑣 → 𝑒 Python ≈ 𝑒 Would be fun to note that in the announcement/docs somehow. -Mike

4 3

Fixing Copy on Writes with Immortal Instances
by Eddie Elizondo April 11, 2020

April 11, 2020

Copy on writes are a big problem in large Python application that rely on multiple processes sharing the same memory. I just sent out a patch introduces Immortal Instances which allows the user to bypass reference counting on specific objects to avoid CoW on forked processes. This has helped in reducing the memory constraints in some of our applications. The PR has a more in-depth explanation about the change. Looking forward to get your thoughts. Github PR: https://github.com/python/cpython/pull/19474 Bug Report: https://bugs.python.org/issue40255 - Eddie E.

1 0

Summary of Python tracker Issues
by Python tracker April 10, 2020

April 10, 2020

ACTIVITY SUMMARY (2020-04-03 - 2020-04-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7433 (+32) closed 44563 (+37) total 51996 (+69) Open issues with patches: 2934 Issues opened (50) ================== #39984: Move pending calls from _PyRuntime to PyInterpreterState https://bugs.python.org/issue39984 reopened by vstinner #40178: Convert the remaining os funtions to Argument Clinic https://bugs.python.org/issue40178 opened by serhiy.storchaka #40179: Argument Clinic incorretly translates #elif https://bugs.python.org/issue40179 opened by serhiy.storchaka #40180: isinstance(cls_with_metaclass, non_type) raises KeyError https://bugs.python.org/issue40180 opened by terry.reedy #40181: IDLE: remove positional-only note from calltips https://bugs.python.org/issue40181 opened by terry.reedy #40183: AC_COMPILE_IFELSE doesn't work in all cases https://bugs.python.org/issue40183 opened by jerome.hamm #40186: test_notify_all hangs forever in sparc64 https://bugs.python.org/issue40186 opened by BTaskaya #40188: Azure Pipelines jobs failing randomly with: Unable to connect https://bugs.python.org/issue40188 opened by vstinner #40191: tempfile.mkstemp() | Documentation Error https://bugs.python.org/issue40191 opened by Howard Waterfall #40192: time.thread_time isn't outputting in nanoseconds in AIX https://bugs.python.org/issue40192 opened by BTaskaya #40195: multiprocessing.Queue.put can fail silently due to pickle erro https://bugs.python.org/issue40195 opened by Sander Land #40197: Add nanoseconds to timing table in What's new python 3.8 https://bugs.python.org/issue40197 opened by mchels #40198: macOS Python builds from Python.org ignore DYLD_LIBRARY_PATH d https://bugs.python.org/issue40198 opened by dgelessus #40199: Invalid escape sequence DeprecationWarnings don't trigger by d https://bugs.python.org/issue40199 opened by Numerlor #40202: Misleading grammatically of ValueError Message? https://bugs.python.org/issue40202 opened by Jacob RR #40203: Warn about invalid PYTHONUSERBASE https://bugs.python.org/issue40203 opened by Volker Weißmann #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 opened by xtreak #40205: Profile 'builtins' parameter documentation missing https://bugs.python.org/issue40205 opened by bar.harel #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 opened by BTaskaya #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 opened by BTaskaya #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 opened by hakancelik #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 opened by Nikolai Ehrhardt #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 opened by BTaskaya #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 opened by BTaskaya #40213: contextlib.aclosing() https://bugs.python.org/issue40213 opened by John Belmonte #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 opened by aeros #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 opened by Óscar García Amor #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 opened by vstinner #40218: sys.executable is a non existing path if python is executed fr https://bugs.python.org/issue40218 opened by Volker Weißmann #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 opened by Stephen Bell #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 opened by vstinner #40222: "Zero cost" exception handling https://bugs.python.org/issue40222 opened by Mark.Shannon #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 opened by skrah #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 opened by brendon-zhang(a)hotmail.com #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 opened by iivanyuk #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 opened by Mark.Shannon #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 opened by Steven Lu #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 opened by Henry Carscadden #40231: Fix pending calls in subinterpreters https://bugs.python.org/issue40231 opened by vstinner #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 opened by vstinner #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 opened by eric.snow #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 opened by dholth #40236: datetime.datetime.strptime get day error https://bugs.python.org/issue40236 opened by zhanying #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 opened by vstinner #40238: os.stat() on Windows succeeds for nonexistent paths with trail https://bugs.python.org/issue40238 opened by laloch #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 opened by Eric Cousineau #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 opened by vstinner #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 opened by wmeehan #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 opened by Michael.Felt #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 opened by 0az Most recent 15 issues with no replies (15) ========================================== #40245: Add description meta tags to docs.python.org https://bugs.python.org/issue40245 #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40235: confusing documentation for IOBase.__exit__ https://bugs.python.org/issue40235 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40227: SSLError is not passed to the client during handshake https://bugs.python.org/issue40227 #40223: Add -fwrapv for new icc versions https://bugs.python.org/issue40223 #40221: Use new _at_fork_reinit() lock method in multiprocessing https://bugs.python.org/issue40221 #40219: ttk LabeledScale: label covered by hidden element https://bugs.python.org/issue40219 #40215: Use xdg basedir spec on linux https://bugs.python.org/issue40215 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40210: ttk.Combobox focus-out event inheritage https://bugs.python.org/issue40210 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 Most recent 15 issues waiting for review (15) ============================================= #40243: Unicode 3.2 numeric uses decimal_changed instead of numeric_ch https://bugs.python.org/issue40243 #40241: [C API] Make PyGC_Head structure opaque, or even move it to th https://bugs.python.org/issue40241 #40237: Test code coverage (C) job of Travis CI fails on test_distutil https://bugs.python.org/issue40237 #40234: Disallow daemon threads in subinterpreters optionally. https://bugs.python.org/issue40234 #40232: PyOS_AfterFork_Child() should use _PyThread_at_fork_reinit() https://bugs.python.org/issue40232 #40229: tty unblocking setraw and save-restore features https://bugs.python.org/issue40229 #40228: Make setting line number in frame more robust. https://bugs.python.org/issue40228 #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 #40212: Re-enable posix_fallocate and posix_fadvise on AIX https://bugs.python.org/issue40212 #40211: Clarify preadv and pwritev is supported AIX 7.1 and newer. https://bugs.python.org/issue40211 #40209: read_pyfile function refactor in Lib/test/test_unparse.py https://bugs.python.org/issue40209 #40208: Remove deprecated SymbolTable.has_exec https://bugs.python.org/issue40208 #40207: Expose NCURSES_EXT_FUNCS under curses https://bugs.python.org/issue40207 #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 Top 10 most discussed issues (10) ================================= #40214: test_ctypes.test_load_dll_with_flags Windows failure https://bugs.python.org/issue40214 16 msgs #40217: The garbage collector doesn't take in account that objects of https://bugs.python.org/issue40217 16 msgs #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 13 msgs #39481: Implement PEP 585 (Type Hinting Generics In Standard Collectio https://bugs.python.org/issue39481 12 msgs #40225: recursive call within generator expression is O(depth) https://bugs.python.org/issue40225 12 msgs #40204: Docs build error with Sphinx 3.0 due to invalid C declaration https://bugs.python.org/issue40204 11 msgs #40240: Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALI https://bugs.python.org/issue40240 9 msgs #40060: socket.TCP_NOTSENT_LOWAT is missing in official macOS builds https://bugs.python.org/issue40060 8 msgs #40230: Itertools.product() Out of Memory Errors https://bugs.python.org/issue40230 7 msgs #40244: AIX: build: _PyObject_GC_TRACK Asstertion failure https://bugs.python.org/issue40244 7 msgs Issues closed (36) ================== #5654: Add C hook in PyDict_SetItem for debuggers https://bugs.python.org/issue5654 closed by vstinner #25780: Add support for CAN_RAW_JOIN_FILTERS https://bugs.python.org/issue25780 closed by pablogsal #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 closed by vstinner #37127: Handling pending calls during runtime finalization may cause p https://bugs.python.org/issue37127 closed by vstinner #38501: multiprocessing.Pool hangs atexit (and garbage collection some https://bugs.python.org/issue38501 closed by pablogsal #38689: IDLE crashes when KeyError is raised during calltip generation https://bugs.python.org/issue38689 closed by terry.reedy #39019: Missing class getitems in standard library classes https://bugs.python.org/issue39019 closed by BTaskaya #39689: struct and memoryview tests rely on undefined behavior (as rev https://bugs.python.org/issue39689 closed by petr.viktorin #40082: trip_signal() gets NULL tstate on Windows on CTRL+C https://bugs.python.org/issue40082 closed by vstinner #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 closed by ammar2 #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 closed by terry.reedy #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 closed by pablogsal #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 closed by SilentGhost #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 closed by steve.dower #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 closed by benjamin.peterson #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 closed by nergall2 #40177: Python Language Reference Documentation https://bugs.python.org/issue40177 closed by mark.dickinson #40182: Remove the _field_types attribute of NamedTuple https://bugs.python.org/issue40182 closed by serhiy.storchaka #40184: Compiler warnings under sparc64 https://bugs.python.org/issue40184 closed by benjamin.peterson #40185: Refactor typing.NamedTuple https://bugs.python.org/issue40185 closed by serhiy.storchaka #40187: Refactor typing.TypedDict https://bugs.python.org/issue40187 closed by serhiy.storchaka #40189: MultiProcessing Subclass - overrides run method issue https://bugs.python.org/issue40189 closed by Python_dev_IL #40190: Add support for _SC_AIX_REALMEM in posix.sysconf https://bugs.python.org/issue40190 closed by vstinner #40193: thread.get_native_id() support for solaris https://bugs.python.org/issue40193 closed by BTaskaya #40194: Document special meanings of a single underscore https://bugs.python.org/issue40194 closed by SilentGhost #40196: symtable.Symbol.is_local() can be True for global symbols https://bugs.python.org/issue40196 closed by pablogsal #40200: count 0 crash https://bugs.python.org/issue40200 closed by BTaskaya #40201: Last digit count error https://bugs.python.org/issue40201 closed by josh.r #40206: Multiplying 4.6*100 will result in 459.99999999999994 https://bugs.python.org/issue40206 closed by eric.smith #40216: Support --runstatedir in configure https://bugs.python.org/issue40216 closed by vstinner #40220: Modules/gcmodule.c:434: update_refs: Assertion "gc_get_refs(gc https://bugs.python.org/issue40220 closed by gregory.p.smith #40224: delete_me https://bugs.python.org/issue40224 closed by ronaldoussoren #40226: Leak in tstate->interp->ceval.pending https://bugs.python.org/issue40226 closed by vstinner #40233: Awkward to set compression on writeable ZipFile.open() https://bugs.python.org/issue40233 closed by dholth #40239: Add a function for merging sorted iterables https://bugs.python.org/issue40239 closed by serhiy.storchaka #40242: zmq mysql core dump https://bugs.python.org/issue40242 closed by eric.smith

1 0

Need help with test_ctypes failing on Windows (test_load_dll_with_flags)
by Guido van Rossum April 7, 2020

April 7, 2020

I have a large PR (https://github.com/python/cpython/pull/18239, for PEP 585) that's failing in the Azures pipeline on Win32 and Win64 only. My trusty assistant who has a Windows laptop couldn't reproduce the failure. Can I buy a hint from someone? Steve? The relevant failure output is: ====================================================================== ERROR: test_load_dll_with_flags (ctypes.test.test_loading.LoaderTest) [WinDLL('_sqlite3.dll', winmode=0)] ---------------------------------------------------------------------- Traceback (most recent call last): File "d:\a\1\s\lib\ctypes\test\test_loading.py", line 140, in should_pass subprocess.check_output( File "d:\a\1\s\lib\subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "d:\a\1\s\lib\subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['d:\\a\\1\\s\\PCbuild\\win32\\python.exe', '-c', "from ctypes import *; import nt;WinDLL('_sqlite3.dll', winmode=0)"]' returned non-zero exit status 1. ---------------------------------------------------------------------- -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

4 5

Problem with instantiating a C extension class from another class
by Musbur April 6, 2020

April 6, 2020

Hello, (I've asked this question before on python-list but only got scarce and ultimately unhelpful answers. Although this isn't about Python development itself, I'm assuming that there is more CPython knowledge on this list than on the other, so please bear with me.) I've written a C extension module, each defining a class (called Series and CLHist, respectively). I can import both modules in Python and use the classes. Everything works fine, also according to valgrind. Now I want to return a Series instance directly from a CLHist method, and that immediately crashes Python with a segfault. Here's a bit of code and just the _new, _init, and _finalize methods which I've sprinkled with debugging code. When I instantiate a Series object from Python, I see output from all three methods. When I call pyseries_new() from C, none of the three functions are called and it crashes. At some point I thought maybe I should call PyType_Ready() from the C side before instantiating Series. In a "standard" (system) Python installation, it printed the desired debugging output and crashed later. Under a debugging Python installation I built it outputs nothing except an interesting post-mortem "Fatal Python error: UNREF invalid object" (see below). Here's my C code with the boilerplate stuff left out. typedef struct { PyObject_HEAD struct clh_series *series; } Series; static PyObject *Series_new(PyTypeObject *type, PyObject *args, PyObject *kw) { Series *self; self = (Series *) type->tp_alloc(type, 0); fprintf(stderr, "Series_new(%p)\n", self); self->series = NULL; return (PyObject*)self; } static int Series_init(Series *self, PyObject *args, PyObject *kw) { fprintf(stderr, "Series_init(%p)\n", self); self->series = NULL; return 0; } static void Series_finalize(PyObject *self) { fprintf(stderr, "Series_finalize(%p)\n", self); clh_series_free(((Series*)self)->series); } /* To create a new Series object directly from C */ PyObject *pyseries_new(struct clh_series *series) { Series *pyseries; pyseries = PyObject_New(Series, &series_type); PyObject_Init((PyObject *)pyseries, &series_type); pyseries->series = series; return (PyObject *) pyseries; } Here's the debugging output: * ob object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 * op->_ob_prev->_ob_next object : <_Series object at 0x7f1a4e2ee9d0> type : _Series refcount: 1 address : 0x7f1a4e2ee9d0 * op->_ob_next->_ob_prev object : <refcnt 0 at 0x7f1a4e2f97a8> type : tuple refcount: 0 address : 0x7f1a4e2f97a8 Fatal Python error: UNREF invalid object

3 3

Question about bytecode stability
by Jonathan Goble April 6, 2020

April 6, 2020

I'm getting ready to get back into an old personal project with some new ideas. Before I do, I need to ask a question regarding the stability of CPython bytecode. Before you say it, I am well aware and fully understand that the opcodes may change in value and semantics between 3.x and 3.(x+1) without notice, and that's acceptable to me. My question is, are the opcodes guaranteed stable across the lifetime of a single 3.x release? In other words, are they guaranteed to not change values or semantics between 3.x.y and 3.x.(y+1)? Reading through the list of opcodes in the dis documentation, it seems that all changes have occurred in 3.x.0, so it seems the answer would be yes, but then the "CPython implementation detail" paragraph at the top doesn't specify that and is a little vague on whether that's true or not. (For context, my personal project is a BASIC implementation in Python that would compile to Python bytecode.)

5 6

[RELEASE] Python 2.7.18 release candidate 1
by Benjamin Peterson April 6, 2020

April 6, 2020

Greetings, 2.7.18 release candidate 1, a testing release for the last release of the Python 2.7 series, is now available for download. The CPython core developers stopped applying routine bugfixes to the 2.7 branch on January 1. 2.7.18 will includes fixes that were made between the release of 2.7.17 and the end of 2019. A final—very final!—release is expected in 2 weeks. Downloads are at: https://www.python.org/downloads/release/python-2718rc1/ The full changelog is at https://raw.githubusercontent.com/python/cpython/v2.7.18rc1/Misc/NEWS.d/2.7… Test it out, and let us know if there are any critical problems at https://bugs.python.org/ (This is the last chance!) All the best, Benjamin

1 0

How to enable tracemalloc for the test suite?
by Skip Montanaro April 6, 2020

April 6, 2020

I've got a memory issue in my modified Python interpreter I'm trying to debug. Output at the end of the problematic unit test looks like this: ... == Tests result: FAILURE then SUCCESS == 1 test OK. 1 re-run test: test_rattlesnake Total duration: 2.9 sec Tests result: FAILURE then SUCCESS Debug memory block at address p=0x55a227969080: API '' 0 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd): at p-7: 0x00 *** OUCH at p-6: 0x00 *** OUCH at p-5: 0x00 *** OUCH at p-4: 0x00 *** OUCH at p-3: 0x00 *** OUCH at p-2: 0x00 *** OUCH at p-1: 0x00 *** OUCH Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0x55a227969080 are not all FORBIDDENBYTE (0xfd): at tail+0: 0x00 *** OUCH at tail+1: 0x00 *** OUCH at tail+2: 0x00 *** OUCH at tail+3: 0x00 *** OUCH at tail+4: 0x00 *** OUCH at tail+5: 0x00 *** OUCH at tail+6: 0x00 *** OUCH at tail+7: 0x00 *** OUCH The block was made by call #0 to debug malloc/realloc. Enable tracemalloc to get the memory block allocation traceback Looking at the tracemalloc module docs and trying various command line args (-X tracemalloc=5) or environment variables (PYTHONTRACEMALLOC=5) I'm unable to provoke any different output. The module docs give a tantalizing suggestion: Example of output of the Python test suite: but I see no command line args related to running the test suite with tracemalloc enabled. Pointers appreciated. Skip

3 8

Any thoughts about a control flow optimizer for CPython?
by joannah nanjekye April 6, 2020

April 6, 2020

Hey all, From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. -- Best, Joannah Nanjekye *"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis*

4 7

Set PyEval_SetTrace to method from class
by Leandro Müller April 5, 2020

April 5, 2020

Hello. I need to set a PyEval_SetTrace with a new class instance. All examples use simple function. PyEval_SetTrace(trace_trampoline, obj); Following my problem: I start three threads, but I need to check trace C just one thread. If I run a while on C trace on thread, it stops all threads. So, I've been thinking to run Trace with a new instance class C. Any suggestions? Att. Leandro Müller

2 1

Summary of Python tracker Issues
by Python tracker April 3, 2020

April 3, 2020

ACTIVITY SUMMARY (2020-03-27 - 2020-04-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7401 (+44) closed 44526 (+40) total 51927 (+84) Open issues with patches: 2909 Issues opened (65) ================== #23574: datetime: support leap seconds https://bugs.python.org/issue23574 reopened by vstinner #35455: Solaris thread_time doesn't work with current implementation https://bugs.python.org/issue35455 reopened by BTaskaya #37207: Use PEP 590 vectorcall to speed up calls to range(), list() an https://bugs.python.org/issue37207 reopened by petr.viktorin #39562: Asynchronous comprehensions don't work in asyncio REPL https://bugs.python.org/issue39562 reopened by vstinner #39951: Ignore specific errors when closing ssl connections https://bugs.python.org/issue39951 reopened by Dima.Tisnek #40068: test_threading: ThreadJoinOnShutdown.test_reinit_tls_after_for https://bugs.python.org/issue40068 reopened by vstinner #40093: ThreadPoolExecutor with wait=True shuts down too early https://bugs.python.org/issue40093 opened by fireattack #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 opened by vstinner #40095: Incorrect st_ino returned for ReFS on Windows 10 https://bugs.python.org/issue40095 opened by mbrijun(a)gmail.com #40098: dir() does not return the list of valid attributes for the obj https://bugs.python.org/issue40098 opened by serhiy.storchaka #40100: IDLE: Undesired behavior of emoji in editor. https://bugs.python.org/issue40100 opened by Alex-Python-Programmer #40101: lib2to3 fails on default convert functionality https://bugs.python.org/issue40101 opened by BTaskaya #40102: Improve XLC support for function attributes https://bugs.python.org/issue40102 opened by BTaskaya #40103: ZipFile.extractall is not multiprocess safe with regard to dir https://bugs.python.org/issue40103 opened by nathansilberman #40105: Updating zip comment doesn't truncate the zip file https://bugs.python.org/issue40105 opened by nergall2 #40106: multiprocessor spawn https://bugs.python.org/issue40106 opened by mouse07410 #40107: pathlib: make `_Accessor.open()` return a file object and not https://bugs.python.org/issue40107 opened by barneygale #40110: multiprocessing.Pool.imap() should be lazy https://bugs.python.org/issue40110 opened by kousu #40111: Introspect ssl context: read ALPN and NPN protocols https://bugs.python.org/issue40111 opened by Dima.Tisnek #40113: Turtle demo https://bugs.python.org/issue40113 opened by Davide Golinelli #40116: Regression in memory use of shared key dictionaries for "compa https://bugs.python.org/issue40116 opened by Mark.Shannon #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 opened by omer sela #40119: ensurepip should use different pattern for pip/setuptool wheel https://bugs.python.org/issue40119 opened by anthon #40120: Undefined C behavior going beyond end of struct via a [1] arra https://bugs.python.org/issue40120 opened by gregory.p.smith #40124: Clearer assertion error https://bugs.python.org/issue40124 opened by pgjones #40126: Incorrect error handling in unittest.mock https://bugs.python.org/issue40126 opened by bmclarnon #40127: Documentation of SSL library https://bugs.python.org/issue40127 opened by Christophe Nanteuil #40128: IDLE Show completions pop-up not working on macOS https://bugs.python.org/issue40128 opened by darthur90 #40129: Add test classes for custom __index__, __int__, __float__ and https://bugs.python.org/issue40129 opened by serhiy.storchaka #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 opened by ChrisRands #40133: Provide additional matchers for unittest.mock https://bugs.python.org/issue40133 opened by Flameeyes #40134: Inconsistent ANSI escape code handling on Windows 10 https://bugs.python.org/issue40134 opened by daverove #40135: multiprocessing: test_shared_memory_across_processes() cannot https://bugs.python.org/issue40135 opened by vstinner #40137: TODO list when PEP 573 "Module State Access from C Extension M https://bugs.python.org/issue40137 opened by vstinner #40138: Windows implementation of os.waitpid() truncates the exit stat https://bugs.python.org/issue40138 opened by vstinner #40139: mimetypes module racy https://bugs.python.org/issue40139 opened by ukl #40142: Modify _PyObject_GC_TRACK() to ensure that newly tracked objec https://bugs.python.org/issue40142 opened by vstinner #40143: shutil.rmtree will frequently fail on Windows under heavy load https://bugs.python.org/issue40143 opened by Alexander Riccio #40145: Pyshellext room for binary size improvement https://bugs.python.org/issue40145 opened by Alexander Riccio #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 opened by pablogsal #40148: Add PurePath.with_stem() https://bugs.python.org/issue40148 opened by timhoffm #40149: test_threading leaked [38, 38, 38] references, sum=114 https://bugs.python.org/issue40149 opened by vstinner #40150: (minor) mismatched argument in overlapped_RegisterWaitWithQueu https://bugs.python.org/issue40150 opened by Alexander Riccio #40151: _overlapped room for improvement https://bugs.python.org/issue40151 opened by Alexander Riccio #40152: Coroutine hangs if it performs async operations when handling https://bugs.python.org/issue40152 opened by salgado #40153: json dump with repeated key https://bugs.python.org/issue40153 opened by facundobatista #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 opened by ferferga #40157: SMTP email policy does not encode non-ASCII characters https://bugs.python.org/issue40157 opened by aepsil0n #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 opened by sydefekt #40160: documentation example of os.walk should be less destructive https://bugs.python.org/issue40160 opened by jftuga #40161: Name collisions in pythoncore, preventing unity/jumbo build https://bugs.python.org/issue40161 opened by Alexander Riccio #40163: multissl doesn't support tarballs in /source/old/ https://bugs.python.org/issue40163 opened by vstinner #40164: Upgrade Windows and macOS installer builds to OpenSSL 1.1.1f https://bugs.python.org/issue40164 opened by ned.deily #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 opened by BTaskaya #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 opened by amaajemyfren #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 opened by Beier Liu #40168: import pandas error[python 3.8.] https://bugs.python.org/issue40168 opened by Harish Gajjar #40169: `dis.findlabels()` should accept a code object https://bugs.python.org/issue40169 opened by laike9m #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 opened by vstinner #40171: Attempting to import inaccessible package imports an empty pac https://bugs.python.org/issue40171 opened by xfix #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 opened by nergall2 #40173: test.support.import_fresh_module fails to correctly block subm https://bugs.python.org/issue40173 opened by p-ganssle #40174: HAVE_CLOCK_GETTIME not repected in pytime.c https://bugs.python.org/issue40174 opened by jerome.hamm #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 opened by nergall2 #40176: unterminated string literal tokenization error messages could https://bugs.python.org/issue40176 opened by benjamin.peterson Most recent 15 issues with no replies (15) ========================================== #40176: unterminated string literal tokenization error messages could https://bugs.python.org/issue40176 #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 #40173: test.support.import_fresh_module fails to correctly block subm https://bugs.python.org/issue40173 #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 #40170: [C API] Make PyTypeObject structure an opaque structure in the https://bugs.python.org/issue40170 #40169: `dis.findlabels()` should accept a code object https://bugs.python.org/issue40169 #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 #40164: Upgrade Windows and macOS installer builds to OpenSSL 1.1.1f https://bugs.python.org/issue40164 #40161: Name collisions in pythoncore, preventing unity/jumbo build https://bugs.python.org/issue40161 #40157: SMTP email policy does not encode non-ASCII characters https://bugs.python.org/issue40157 #40154: embedded null byte when connecting to sqlite database using a https://bugs.python.org/issue40154 #40152: Coroutine hangs if it performs async operations when handling https://bugs.python.org/issue40152 #40150: (minor) mismatched argument in overlapped_RegisterWaitWithQueu https://bugs.python.org/issue40150 Most recent 15 issues waiting for review (15) ============================================= #40175: Add support for removing zip entry in ZipInfo https://bugs.python.org/issue40175 #40172: ZipInfo corrupts file names in some old zip archives https://bugs.python.org/issue40172 #40167: Cython files don't compile on Mingw-w64 64-bit https://bugs.python.org/issue40167 #40166: UNICODE HOWTO: Change the total number of code points in the i https://bugs.python.org/issue40166 #40165: Hide stderror from the user if command failes https://bugs.python.org/issue40165 #40163: multissl doesn't support tarballs in /source/old/ https://bugs.python.org/issue40163 #40160: documentation example of os.walk should be less destructive https://bugs.python.org/issue40160 #40158: MSBuild Extensions in CPython NuGet Package has Bad Expression https://bugs.python.org/issue40158 #40151: _overlapped room for improvement https://bugs.python.org/issue40151 #40148: Add PurePath.with_stem() https://bugs.python.org/issue40148 #40147: Move checking for duplicated keywords to the compiler https://bugs.python.org/issue40147 #40145: Pyshellext room for binary size improvement https://bugs.python.org/issue40145 #40142: Modify _PyObject_GC_TRACK() to ensure that newly tracked objec https://bugs.python.org/issue40142 #40137: TODO list when PEP 573 "Module State Access from C Extension M https://bugs.python.org/issue40137 #40133: Provide additional matchers for unittest.mock https://bugs.python.org/issue40133 Top 10 most discussed issues (10) ================================= #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 29 msgs #37207: Use PEP 590 vectorcall to speed up calls to range(), list() an https://bugs.python.org/issue37207 13 msgs #40120: Undefined C behavior going beyond end of struct via a [1] arra https://bugs.python.org/issue40120 13 msgs #40098: dir() does not return the list of valid attributes for the obj https://bugs.python.org/issue40098 10 msgs #17005: Add a topological sort algorithm https://bugs.python.org/issue17005 8 msgs #39837: Remove Azure Pipelines from GitHub PRs https://bugs.python.org/issue39837 8 msgs #39966: mock 3.9 bug: Wrapped objects without __bool__ raise exception https://bugs.python.org/issue39966 8 msgs #40077: Convert static types to PyType_FromSpec() https://bugs.python.org/issue40077 8 msgs #33725: Python crashes on macOS after fork with no exec https://bugs.python.org/issue33725 7 msgs #39503: [security][CVE-2020-8492] Denial of service in urllib.request. https://bugs.python.org/issue39503 7 msgs Issues closed (42) ================== #15140: PEP 384 inconsistent with implementation https://bugs.python.org/issue15140 closed by gvanrossum #38527: configure script fails to detect "float word ordering" on Sola https://bugs.python.org/issue38527 closed by pablogsal #38636: IDLE regression: toggle tabs and change indent width functions https://bugs.python.org/issue38636 closed by terry.reedy #38972: [venv] Link to instructions to change PowerShell execution pol https://bugs.python.org/issue38972 closed by brett.cannon #39133: threading lib. working improperly on idle window https://bugs.python.org/issue39133 closed by terry.reedy #39682: pathlib.Path objects can be used as context managers https://bugs.python.org/issue39682 closed by pitrou #39780: Add HTTP Response code 103 https://bugs.python.org/issue39780 closed by corona10 #39812: Avoid daemon threads in concurrent.futures https://bugs.python.org/issue39812 closed by pitrou #39982: FreeBSD: SCTP tests of test_socket fails on AMD64 FreeBSD Shar https://bugs.python.org/issue39982 closed by koobs #40018: test_ssl fails with OpenSSL 1.1.1e https://bugs.python.org/issue40018 closed by benjamin.peterson #40019: test_gdb should better detect when Python is optimized https://bugs.python.org/issue40019 closed by vstinner #40020: growable_comment_array_add leaks, causes crash https://bugs.python.org/issue40020 closed by Alexander Riccio #40024: Add PyModule_AddType helper function https://bugs.python.org/issue40024 closed by vstinner #40071: test__xxsubinterpreters leaked [1, 1, 1] references: test_ids https://bugs.python.org/issue40071 closed by vstinner #40086: test_etree is skipped in test_typing due to cElementTree remov https://bugs.python.org/issue40086 closed by xtreak #40087: How to Uninstall Python3.7.3 using cmd? https://bugs.python.org/issue40087 closed by terry.reedy #40088: list.reverse(): slow sublist reverse https://bugs.python.org/issue40088 closed by rhettinger #40096: Support _Py_NO_RETURN on XLC https://bugs.python.org/issue40096 closed by pablogsal #40097: Queue.Empty issue - Python3.8 https://bugs.python.org/issue40097 closed by Python_dev_IL #40099: modify code format of json library for pep7,8 https://bugs.python.org/issue40099 closed by rhettinger #40104: ElementTree does not find elements in a default namespace with https://bugs.python.org/issue40104 closed by Myron Walker #40108: Improve error message for -m option when .py is present https://bugs.python.org/issue40108 closed by pablogsal #40109: List index doesn't work with multiple assignment https://bugs.python.org/issue40109 closed by benjamin.peterson #40112: AIX: xlc - default path changed and no longer recognized https://bugs.python.org/issue40112 closed by vstinner #40114: support maxsize argument for lru_cache's user_function option https://bugs.python.org/issue40114 closed by brendon-zhang(a)hotmail.com #40115: test_asyncio leaked [3, 3, 3] references, sum=9 https://bugs.python.org/issue40115 closed by vstinner #40117: __round__ doesn't behave well with return NotImplemented https://bugs.python.org/issue40117 closed by mark.dickinson #40121: socket module missing audit events https://bugs.python.org/issue40121 closed by steve.dower #40122: The implementation and documentation of "dis.findlables" don't https://bugs.python.org/issue40122 closed by serhiy.storchaka #40123: append() works not correct https://bugs.python.org/issue40123 closed by steven.daprano #40125: update OpenSSL 1.1.1 in multissltests.py to 1.1.1f https://bugs.python.org/issue40125 closed by benjamin.peterson #40130: Remove _PyUnicode_AsKind from private C API https://bugs.python.org/issue40130 closed by serhiy.storchaka #40131: Zipapp example has parameters in the wrong order https://bugs.python.org/issue40131 closed by xtreak #40136: add warning to datetime.replace documentation to not use it fo https://bugs.python.org/issue40136 closed by p-ganssle #40140: test_builtin crashes when runned in parallel mode on solaris https://bugs.python.org/issue40140 closed by vstinner #40141: Add line and column information for keywords in the AST https://bugs.python.org/issue40141 closed by pablogsal #40144: resource: RLIMIT_xxx and RUSAGE_xxx constants are missing on m https://bugs.python.org/issue40144 closed by vstinner #40146: Upgrade Azure Pipelines to OpenSSL 1.1.1f https://bugs.python.org/issue40146 closed by vstinner #40155: AIX: test_builtin.test_input_no_stdout_fileno() hangs https://bugs.python.org/issue40155 closed by vstinner #40156: CodeCov/patch job stills runs on pull requests on 3.5 and 3.6 https://bugs.python.org/issue40156 closed by vstinner #40159: Make python -V -V output arguments of configure https://bugs.python.org/issue40159 closed by Volker Weißmann #40162: Upgrade Travis CI to OpenSSL 1.1.1f https://bugs.python.org/issue40162 closed by vstinner

1 0

PEP 616 -- String methods to remove prefixes and suffixes
by Dennis Sweeney April 2, 2020

April 2, 2020

Browser Link: https://www.python.org/dev/peps/pep-0616/ PEP: 616 Title: String methods to remove prefixes and suffixes Author: Dennis Sweeney <sweeney.dennis650(a)gmail.com> Sponsor: Eric V. Smith <eric(a)trueblade.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 19-Mar-2020 Python-Version: 3.9 Post-History: 30-Aug-2002 Abstract ======== This is a proposal to add two new methods, ``cutprefix`` and ``cutsuffix``, to the APIs of Python's various string objects. In particular, the methods would be added to Unicode ``str`` objects, binary ``bytes`` and ``bytearray`` objects, and ``collections.UserString``. If ``s`` is one these objects, and ``s`` has ``pre`` as a prefix, then ``s.cutprefix(pre)`` returns a copy of ``s`` in which that prefix has been removed. If ``s`` does not have ``pre`` as a prefix, an unchanged copy of ``s`` is returned. In summary, ``s.cutprefix(pre)`` is roughly equivalent to ``s[len(pre):] if s.startswith(pre) else s``. The behavior of ``cutsuffix`` is analogous: ``s.cutsuffix(suf)`` is roughly equivalent to ``s[:-len(suf)] if suf and s.endswith(suf) else s``. Rationale ========= There have been repeated issues [#confusion]_ on the Bug Tracker and StackOverflow related to user confusion about the existing ``str.lstrip`` and ``str.rstrip`` methods. These users are typically expecting the behavior of ``cutprefix`` and ``cutsuffix``, but they are surprised that the parameter for ``lstrip`` is interpreted as a set of characters, not a substring. This repeated issue is evidence that these methods are useful, and the new methods allow a cleaner redirection of users to the desired behavior. As another testimonial for the usefulness of these methods, several users on Python-Ideas [#pyid]_ reported frequently including similar functions in their own code for productivity. The implementation often contained subtle mistakes regarding the handling of the empty string (see `Specification`_). Specification ============= The builtin ``str`` class will gain two new methods with roughly the following behavior:: def cutprefix(self: str, pre: str, /) -> str: if self.startswith(pre): return self[len(pre):] return self[:] def cutsuffix(self: str, suf: str, /) -> str: if suf and self.endswith(suf): return self[:-len(suf)] return self[:] The only difference between the real implementation and the above is that, as with other string methods like ``replace``, the methods will raise a ``TypeError`` if any of ``self``, ``pre`` or ``suf`` is not an instace of ``str``, and will cast subclasses of ``str`` to builtin ``str`` objects. Note that without the check for the truthyness of ``suf``, ``s.cutsuffix('')`` would be mishandled and always return the empty string due to the unintended evaluation of ``self[:-0]``. Methods with the corresponding semantics will be added to the builtin ``bytes`` and ``bytearray`` objects. If ``b`` is either a ``bytes`` or ``bytearray`` object, then ``b.cutsuffix()`` and ``b.cutprefix()`` will accept any bytes-like object as an argument. Note that the ``bytearray`` methods return a copy of ``self``; they do not operate in place. The following behavior is considered a CPython implementation detail, but is not guaranteed by this specification:: >>> x = 'foobar' * 10**6 >>> x.cutprefix('baz') is x is x.cutsuffix('baz') True >>> x.cutprefix('') is x is x.cutsuffix('') True That is, for CPython's immutable ``str`` and ``bytes`` objects, the methods return the original object when the affix is not found or if the affix is empty. Because these types test for equality using shortcuts for identity and length, the following equivalent expressions are evaluated at approximately the same speed, for any ``str`` objects (or ``bytes`` objects) ``x`` and ``y``:: >>> (True, x[len(y):]) if x.startswith(y) else (False, x) >>> (True, z) if x != (z := x.cutprefix(y)) else (False, x) The two methods will also be added to ``collections.UserString``, where they rely on the implementation of the new ``str`` methods. Motivating examples from the Python standard library ==================================================== The examples below demonstrate how the proposed methods can make code one or more of the following: Less fragile: The code will not depend on the user to count the length of a literal. More performant: The code does not require a call to the Python built-in ``len`` function. More descriptive: The methods give a higher-level API for code readability, as opposed to the traditional method of string slicing. refactor.py ----------- - Current:: if fix_name.startswith(self.FILE_PREFIX): fix_name = fix_name[len(self.FILE_PREFIX):] - Improved:: fix_name = fix_name.cutprefix(self.FILE_PREFIX) c_annotations.py: ----------------- - Current:: if name.startswith("c."): name = name[2:] - Improved:: name = name.cutprefix("c.") find_recursionlimit.py ---------------------- - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) deccheck.py ----------- This is an interesting case because the author chose to use the ``str.replace`` method in a situation where only a prefix was intended to be removed. - Current:: if funcname.startswith("context."): self.funcname = funcname.replace("context.", "") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Improved:: if funcname.startswith("context."): self.funcname = funcname.cutprefix("context.") self.contextfunc = True else: self.funcname = funcname self.contextfunc = False - Arguably further improved:: self.contextfunc = funcname.startswith("context.") self.funcname = funcname.cutprefix("context.") test_i18n.py ------------ - Current:: if test_func_name.startswith("test_"): print(test_func_name[5:]) else: print(test_func_name) - Improved:: print(test_finc_name.cutprefix("test_")) - Current:: if creationDate.endswith('\\n'): creationDate = creationDate[:-len('\\n')] - Improved:: creationDate = creationDate.cutsuffix('\\n') shared_memory.py ---------------- - Current:: reported_name = self._name if _USE_POSIX and self._prepend_leading_slash: if self._name.startswith("/"): reported_name = self._name[1:] return reported_name - Improved:: if _USE_POSIX and self._prepend_leading_slash: return self._name.cutprefix("/") return self._name build-installer.py ------------------ - Current:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if ((retval.startswith('tcl') or retval.startswith('tk')) and retval.endswith('-src')): retval = retval[:-4] - Improved:: if archiveName.endswith('.tar.gz'): retval = os.path.basename(archiveName[:-7]) if retval.startswith(('tcl', 'tk')): retval = retval.cutsuffix('-src') Depending on personal style, ``archiveName[:-7]`` could also be changed to ``archiveName.cutsuffix('.tar.gz')``. test_core.py ------------ - Current:: if output.endswith("\n"): output = output[:-1] - Improved:: output = output.cutsuffix("\n") cookiejar.py ------------ - Current:: def strip_quotes(text): if text.startswith('"'): text = text[1:] if text.endswith('"'): text = text[:-1] return text - Improved:: def strip_quotes(text): return text.cutprefix('"').cutsuffix('"') - Current:: if line.endswith("\n"): line = line[:-1] - Improved:: line = line.cutsuffix("\n") fixdiv.py --------- - Current:: def chop(line): if line.endswith("\n"): return line[:-1] else: return line - Improved:: def chop(line): return line.cutsuffix("\n") test_concurrent_futures.py -------------------------- In the following example, the meaning of the code changes slightly, but in context, it behaves the same. - Current:: if name.endswith(('Mixin', 'Tests')): return name[:-5] elif name.endswith('Test'): return name[:-4] else: return name - Improved:: return name.cutsuffix('Mixin').cutsuffix('Tests').cutsuffix('Test') msvc9compiler.py ---------------- - Current:: if value.endswith(os.pathsep): value = value[:-1] - Improved:: value = value.cutsuffix(os.pathsep) test_pathlib.py --------------- - Current:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r[len(clsname) + 1 : -1] - Improved:: self.assertTrue(r.startswith(clsname + '('), r) self.assertTrue(r.endswith(')'), r) inner = r.cutprefix(clsname + '(').cutsuffix(')') Rejected Ideas ============== Expand the lstrip and rstrip APIs --------------------------------- Because ``lstrip`` takes a string as its argument, it could be viewed as taking an iterable of length-1 strings. The API could therefore be generalized to accept any iterable of strings, which would be successively removed as prefixes. While this behavior would be consistent, it would not be obvious for users to have to call ``'foobar'.cutprefix(('foo,))`` for the common use case of a single prefix. Allow multiple prefixes ----------------------- Some users discussed the desire to be able to remove multiple prefixes, calling, for example, ``s.cutprefix('From: ', 'CC: ')``. However, this adds ambiguity about the order in which the prefixes are removed, especially in cases like ``s.cutprefix('Foo', 'FooBar')``. After this proposal, this can be spelled explicitly as ``s.cutprefix('Foo').cutprefix('FooBar')``. Remove multiple copies of a prefix ---------------------------------- This is the behavior that would be consistent with the aforementioned expansion of the ``lstrip/rstrip`` API -- repeatedly applying the function until the argument is unchanged. This behavior is attainable from the proposed behavior via the following:: >>> s = 'foo' * 100 + 'bar' >>> while s != (s := s.cutprefix("foo")): pass >>> s 'bar' The above can be modififed by chaining multiple ``cutprefix`` calls together to achieve the full behavior of the ``lstrip``/``rstrip`` generalization, while being explicit in the order of removal. While the proposed API could later be extended to include some of these use cases, to do so before any observation of how these methods are used in practice would be premature and may lead to choosing the wrong behavior. Raising an exception when not found ----------------------------------- There was a suggestion that ``s.cutprefix(pre)`` should raise an exception if ``not s.startswith(pre)``. However, this does not match with the behavior and feel of other string methods. There could be ``required=False`` keyword added, but this violates the KISS principle. Alternative Method Names ------------------------ Several alternatives method names have been proposed. Some are listed below, along with commentary for why they should be rejected in favor of ``cutprefix`` (the same arguments hold for ``cutsuffix``) ``ltrim`` "Trim" does in other languages (e.g. JavaScript, Java, Go, PHP) what ``strip`` methods do in Python. ``lstrip(string=...)`` This would avoid adding a new method, but for different behavior, it's better to have two different methods than one method with a keyword argument that select the behavior. ``cut_prefix`` All of the other methods of the string API, e.g. ``str.startswith()``, use ``lowercase`` rather than ``lower_case_with_underscores``. ``cutleft``, ``leftcut``, or ``lcut`` The explicitness of "prefix" is preferred. ``removeprefix``, ``deleteprefix``, ``withoutprefix``, etc. All of these might have been acceptable, but they have more characters than ``cut``. Some suggested that the verb "cut" implies mutability, but the string API already contains verbs like "replace", "strip", "split", and "swapcase". ``stripprefix`` Users may benefit from the mnemonic that "strip" means working with sets of characters, while other methods work with substrings, so re-using "strip" here should be avoided. Reference Implementation ======================== See the pull request on GitHub [#pr]_. References ========== .. [#pr] GitHub pull request with implementation (https://github.com/python/cpython/pull/18939) .. [#pyid] Discussion on Python-Ideas (https://mail.python.org/archives/list/python-ideas@python.org/thread/RJARZS…) .. [#confusion] Comment listing Bug Tracker and StackOverflow issues (https://mail.python.org/archives/list/python-ideas@python.org/message/GRGAF…) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

33 137

Debug C++
by Leandro Müller April 2, 2020

April 2, 2020

Hi Are there any away to debug C module during python debug? Att. Leandro Müller

4 3

Moving threadstate to thread-local storage.
by Mark Shannon April 2, 2020

April 2, 2020

Hi, As an experiment, I thought I would try moving the thread state (what you get from _PyThreadState_GET() ) to TLS. https://github.com/python/cpython/compare/master...markshannon:threadstate_… It works, passing all the tests, and seems sound. It is a small patch (< 50 lines) and doesn't increase the overall code size. My branch is GCC/Clang only, so will need a bit of extra code for Windows. It should only need a few more lines; I haven't done it as I don't have a Windows machine to test it on. This is a *much* cleaner approach to removing the global variable than adding lots of extra parameters all over the place. Cheers, Mark.

3 4

Re: Python-Dev Digest, Vol 201, Issue 1
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > reversed(enumerate('abc', 1)) > Should that yield...? > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > Something else? > My preference would be to treat the starting value as an ending value. My idea is largely that as long as any iterable has defined length and order, it should be reversible, and this should yield the same as if I did this reverse countdown with some pointer. It just so happens that `enumerate` depends fully in this regard on the iterable it enumerates, if it has defined length, then enumerate will also have defined length. Maybe enumerate should also start defining `len` whenever underlying iterable supports it. Regarding the starting value, I think it is important to keep symmetry: enumerate('abc', 1) is [ (1, 'a'), (2, 'b'), (3, 'c') ], therefore reversed(enumerate('abc', 1)) should be [(3, 'c'), (2, 'b'), (1, 'a')] Best Regards, -- Ilya Kamen On Thu, Apr 2, 2020 at 5:55 AM <python-dev-request(a)python.org> wrote: > Send Python-Dev mailing list submissions to > python-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman3/lists/python-dev.python.org/ > or, via email, send a message with subject or body 'help' to > python-dev-request(a)python.org > > You can reach the person managing the list at > python-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > Today's Topics: > > 1. Re: reversed enumerate (Steven D'Aprano) > 2. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > 3. Re: [Python-ideas] Re: reversed enumerate (Andrew Barnert) > > > ---------------------------------------------------------------------- > > Date: Thu, 2 Apr 2020 13:20:02 +1100 > From: Steven D'Aprano <steve(a)pearwood.info> > Subject: [Python-Dev] Re: reversed enumerate > To: python-dev(a)python.org > Cc: python-ideas(a)python.org > Message-ID: <20200402022000.GC16830(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Ilya, > > I'm not sure that this mailing list (Python-Dev) is the right place for > this discussion, I think that Python-Ideas (CCed) is the correct place. > > For the benefit of Python-Ideas, I have left your entire post below, to > establish context. > > [Ilya] > > I needed reversed(enumerate(x: list)) in my code, and have discovered > > that it wound't work. This is disappointing because operation is well > > defined. > > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. Consider: > > def values(): > while True: > yield random.random() > > a, b = reversed(enumerate(values()) > > What should the first pair of (a, b) be? > > However, having said that, I think that your idea is not unreasonable. > `enumerate(it)` in the most general case isn't reversable, but if `it` > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > be too. > > My personal opinion is that this is a fairly obvious and straightforward > enhancement, one which (hopefully!) shouldn't require much, if any, > debate. I don't think we need a new class for this, I think enhancing > enumerate to be reversable if its underlying iterator is reversable > makes good sense. > > But if you can show some concrete use-cases, especially one or two from > the standard library, that would help your case. Or some other languages > which offer this functionality as standard. > > On the other hand, I think that there is a fairly lightweight work > around. Define a helper function: > > def countdown(n): > while True: > yield n > n -= 1 > > then call it like this: > > # reversed(enumerate(seq)) > zip(countdown(len(seq)-1), reversed(seq))) > > So it isn't terribly hard to work around this. But I agree that it would > be nice if enumerate encapsulated this for the caller. > > One potentially serious question: what should `enumerate.__reversed__` > do when given a starting value? > > reversed(enumerate('abc', 1)) > > Should that yield...? > > # treat the start value as a start value > (1, 'c'), (0, 'b'), (-1, 'a') > > # treat the start value as an end value > (3, 'c'), (2, 'b'), (1, 'a') > > Something else? > > My preference would be to treat the starting value as an ending value. > > > Steven > > > On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote: > > Hi, > > > > I needed reversed(enumerate(x: list)) in my code, and have discovered > that > > it wound't work. This is disappointing because operation is well defined. > > It is also well defined for str type, range, and - in principle, but not > > yet in practice - on dictionary iterators - keys(), values(), items() as > > dictionaries are ordered now. > > It would also be well defined on any user type implementing __iter__, > > __len__, __reversed__ - think numpy arrays, some pandas dataframes, > tensors. > > > > That's plenty of usecases, therefore I guess it would be quite useful to > > avoid hacky / inefficient solutions like described here: > > https://code.activestate.com/lists/python-list/706205/. > > > > If deemed useful, I would be interested in implementing this, maybe > > together with __reversed__ on dict keys, values, items. > > > > Best Regards, > > -- > > Ilya Kamen > > > > ----------- > > p.s. > > > > *Sketch* of what I am proposing: > > > > class reversible_enumerate: > > > > def __init__(self, iterable): > > self.iterable = iterable > > self.ctr = 0 > > > > def __iter__(self): > > for e in self.iterable: > > yield self.ctr, e > > self.ctr += 1 > > > > def __reversed__(self): > > try: > > ri = reversed(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > try: > > l = len(self.iterable) > > except Exception as e: > > raise Exception( > > "enumerate can only be reversed if iterable to > > enumerate can be reversed and has defined length." > > ) from e > > > > indexes = range(l-1, -1, -1) > > for i, e in zip(indexes, ri): > > yield i, e > > > > for i, c in reversed(reversible_enumerate("Hello World")): > > print(i, c) > > > > for i, c in reversed(reversible_enumerate([11, 22, 33])): > > > > print(i, c) > > > _______________________________________________ > > Python-Dev mailing list -- python-dev(a)python.org > > To unsubscribe send an email to python-dev-leave(a)python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD… > > Code of Conduct: http://python.org/psf/codeofconduct/ > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Date: Wed, 1 Apr 2020 20:42:12 -0700 > From: Andrew Barnert <abarnert(a)yahoo.com> > Subject: [Python-Dev] Re: [Python-ideas] Re: reversed enumerate > To: Steven D'Aprano <steve(a)pearwood.info> > Cc: python-dev(a)python.org, python-ideas(a)python.org > Message-ID: <647BE271-D7ED-4E50-A807-3D748B00FF1D(a)yahoo.com> > Content-Type: text/plain; charset=utf-8 > > Before jumping in: > > In many cases, when you want to reverse an enumerate, it’s small and > fixed-sized, so there’s a trivial way to do this: Just store the enumerate > iterator in a tuple, and tuples are reversible. > > for idx, value in reversed(tuple(enumerate(stuff))): > > But of course there are some cases where this isn’t appropriate, like > enumerating a fixed-size but huge input. > > > On Apr 1, 2020, at 19:23, Steven D'Aprano <steve(a)pearwood.info> wrote: > > > > [Ilya] > >> I needed reversed(enumerate(x: list)) in my code, and have discovered > >> that it wound't work. This is disappointing because operation is well > >> defined. > > > > It isn't really well-defined, since enumerate can operate on infinite > > iterators, and you cannot reverse an infinite stream. > > ... > > > However, having said that, I think that your idea is not unreasonable. > > `enumerate(it)` in the most general case isn't reversable, but if `it` > > is reversable and sized, there's no reason why `enumerate(it)` shouldn't > > be too. > > Agreed—but this is just a small piece of a much wider issue. Today, > enumerate is always an Iterator. It’s never reversible. But it’s also not > sized, or subscriptable, or in-testable, even if you give it inputs that > are. And it’s not just enumerate—the same is true for map, filter, zip, > itertools.islice, itertools.dropwhile, etc. > > There’s no reason these things couldn’t all be views, just like the > existing dict views (and other things like memoryview and third-party > things like numpy array slices). In fact, they already are in Swift, and > will be in C++20. > > > My personal opinion is that this is a fairly obvious and straightforward > > enhancement, one which (hopefully!) shouldn't require much, if any, > > debate. I don't think we need a new class for this, I think enhancing > > enumerate to be reversable if its underlying iterator is reversable > > makes good sense. > > Actually, that doesn’t work—it has to be Sized as well. > > More generally, it’s rarely _quite_ as simple as just “views support the > same operations as the things they view”. An enumerate can be a Sequence if > its input is, but a filter can’t. A map with multiple inputs isn’t > Reversible unless they’re all not just Reversible but Sized, although a map > with only one input doesn’t need it to be Sized. And so on. But none of > these things are hard, it’s just a bunch of work to go through all the > input types for all the view types and write up the rules. (Or steal them > from another language or library that already did that work…) > > > But if you can show some concrete use-cases, especially one or two from > > the standard library, that would help your case. Or some other languages > > which offer this functionality as standard. > > Agreed. I don’t think we need to wait until someone designs and writes a > complete viewtools library and submits it for stdlib inclusion before we > can consider adding just one extension to one iterator. But I do think we > want to add the one(s) that are most useful if any, not just whichever ones > people think of first. I’ve personally wanted to reverse a map or a filter > more often than an enumerate, but examples would easily convince me that > that’s just me, and reversing enumerate is more needed. > > > One potentially serious question: what should `enumerate.__reversed__` > > do when given a starting value? > > > > reversed(enumerate('abc', 1)) > > I don’t think this is a problem. When you reversed(tuple(enumerate('abc', > 1))) today, what do you get? You presumably don’t even need to look that up > or try it out. It would be pretty confusing if it were different without > the tuple. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > > ------------------------------ > > End of Python-Dev Digest, Vol 201, Issue 1 > ****************************************** >

1 0

Re: reversed enumerate
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

Re: steve(a)pearwood.info > It isn't really well-defined, since enumerate can operate on infinite > iterators, and you cannot reverse an infinite stream. It is well defined on any iterable that itself is reversible and has defined length. For standard types that's lists, strings, dictionary iterators and some of the collections (deque, dictionary-based counter). Re: storchaka(a)gmail.com > Could you please provide evidence that this feature would be quite > useful? How much usecases can you find in the stdlib? I needed this while solving a CS problem https://leetcode.com/problems/container-with-most-water/, which, I agree, does not tell how useful it is in production projects. Couldn't find any usecase in stdlib. I could find a usecase in scikit-learn: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/pipeline.p… in case file gets changed, here is a copy: for i in reversed(range(len(estimators))): name = names[i] if name in namecount: names[i] += "-%d" % namecount[name] namecount[name] -= 1 It can indeed be that one doesn't have to reverse enumeration very often in practice. It can also be that people just express their ideas differently knowing it doesn't work in straight-forward way. Best Regards, -- Ilya Kamenshchikov

1 0

reversed enumerate
by Ilya Kamenshchikov April 2, 2020

April 2, 2020

Hi, I needed reversed(enumerate(x: list)) in my code, and have discovered that it wound't work. This is disappointing because operation is well defined. It is also well defined for str type, range, and - in principle, but not yet in practice - on dictionary iterators - keys(), values(), items() as dictionaries are ordered now. It would also be well defined on any user type implementing __iter__, __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors. That's plenty of usecases, therefore I guess it would be quite useful to avoid hacky / inefficient solutions like described here: https://code.activestate.com/lists/python-list/706205/. If deemed useful, I would be interested in implementing this, maybe together with __reversed__ on dict keys, values, items. Best Regards, -- Ilya Kamen ----------- p.s. *Sketch* of what I am proposing: class reversible_enumerate: def __init__(self, iterable): self.iterable = iterable self.ctr = 0 def __iter__(self): for e in self.iterable: yield self.ctr, e self.ctr += 1 def __reversed__(self): try: ri = reversed(self.iterable) except Exception as e: raise Exception( "enumerate can only be reversed if iterable to enumerate can be reversed and has defined length." ) from e try: l = len(self.iterable) except Exception as e: raise Exception( "enumerate can only be reversed if iterable to enumerate can be reversed and has defined length." ) from e indexes = range(l-1, -1, -1) for i, e in zip(indexes, ri): yield i, e for i, c in reversed(reversible_enumerate("Hello World")): print(i, c) for i, c in reversed(reversible_enumerate([11, 22, 33])): print(i, c)

5 4