Mailman 3 February 2019 - Python-Dev

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

Boundaries between numbers and identifiers
by Serhiy Storchaka 15 Apr '21

15 Apr '21

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

8 10

About "python-porting" mail list
by Facundo Batista 20 Jan '20

20 Jan '20

Hello! This list (which I co-admin, with Georg) is getting less and less traffic as months pass by. See: https://mail.python.org/pipermail/python-porting/ The interwebs has been collecting ton of resources about porting py2 to 3 during these years. Any not-yet-answered question surely can be done in a list with more participants. Can we kill this list? Thanks! Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org.ar/ Twitter: @facundobatista

4 7

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton 15 Sep '19

15 Sep '19

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

why is not 64-bit installer the default download link for Windows?
by Cosimo Lupo 19 Jun '19

19 Jun '19

If one goes to httWhps://www.python.org/downloads <https://www.python.org/downloads> from a Windows browser, the default download URL is for the 32-bit installer instead of the 64-bit one. I wonder why is this still the case? Shouldn't we encourage new Windows users (who may not even know the distinction between the two architectures) to use the 64-bit version of Python, since most likely they can? If this is not the correct forum for this, please let me know where I can direct my question/feature request, thanks. Cosimo -- Cosimo Lupo

14 22

int() and math.trunc don't accept objects that only define __index__
by Rémi Lapeyre 15 Mar '19

15 Mar '19

Hi, I open this thread to discuss the proposal by Nick Coghlan in https://bugs.python.org/issue33039 to add __int__ and __trunc__ to a type when __index__ is defined. Currently __int__ does not default to __index__ during class initialisation so both must be defined to get a coherant behavior: (cpython-venv) ➜ cpython git:(add-key-argument-to-bisect) ✗ python3 Python 3.8.0a1+ (heads/add-key-argument-to-bisect:b7aaa1adad, Feb 18 2019, 16:10:22) [Clang 10.0.0 (clang-1000.10.44.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import math >>> class MyInt: ... def __index__(self): ... return 4 ... >>> int(MyInt()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: int() argument must be a string, a bytes-like object or a number, not 'MyInt' >>> math.trunc(MyInt()) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: type MyInt doesn't define __trunc__ method >>> hex(MyInt()) '0x4' >>> len("a"*MyInt()) 4 >>> MyInt.__int__ = MyInt.__index__ >>> int(MyInt()) 4 The difference in behavior is espacially weird in builtins like int() and hex(). The documentation mentions at https://docs.python.org/3/reference/datamodel.html#object.__index__ the need to always define both __index__ and __int__: Note: In order to have a coherent integer type class, when __index__() is defined __int__() should also be defined, and both should return the same value. Nick Coghlan proposes to make __int__ defaults to __index__ when only the second is defined and asked to open a discussion on python-dev before making any change "as the closest equivalent we have to this right now is the "negative" derivation, where overriding __eq__ without overriding __hash__ implicitly marks the derived class as unhashable (look for "type->tp_hash = PyObject_HashNotImplemented;").". I think the change proposed makes more sense than the current behavior and volunteer to implement it if it is accepted. What do you think about this?

4 7

Possible performance regression
by Raymond Hettinger 15 Mar '19

15 Mar '19

I'll been running benchmarks that have been stable for a while. But between today and yesterday, there has been an almost across the board performance regression. It's possible that this is a measurement error or something unique to my system (my Mac installed the 10.14.3 release today), so I'm hoping other folks can run checks as well. Raymond -- Yesterday ------------------------------------------------------------------------ $ ./python.exe Tools/scripts/var_access_benchmark.py Variable and attribute read access: 4.0 ns read_local 4.5 ns read_nonlocal 13.1 ns read_global 17.4 ns read_builtin 17.4 ns read_classvar_from_class 15.8 ns read_classvar_from_instance 24.6 ns read_instancevar 19.7 ns read_instancevar_slots 18.5 ns read_namedtuple 26.3 ns read_boundmethod Variable and attribute write access: 4.6 ns write_local 4.8 ns write_nonlocal 17.5 ns write_global 39.1 ns write_classvar 34.4 ns write_instancevar 25.3 ns write_instancevar_slots Data structure read access: 17.5 ns read_list 18.4 ns read_deque 19.2 ns read_dict Data structure write access: 19.0 ns write_list 22.0 ns write_deque 24.4 ns write_dict Stack (or queue) operations: 55.5 ns list_append_pop 46.3 ns deque_append_pop 46.7 ns deque_append_popleft Timing loop overhead: 0.3 ns loop_overhead -- Today --------------------------------------------------------------------------- $ ./python.exe py Tools/scripts/var_access_benchmark.py Variable and attribute read access: 5.0 ns read_local 5.3 ns read_nonlocal 14.7 ns read_global 18.6 ns read_builtin 19.9 ns read_classvar_from_class 17.7 ns read_classvar_from_instance 26.1 ns read_instancevar 21.0 ns read_instancevar_slots 21.7 ns read_namedtuple 27.8 ns read_boundmethod Variable and attribute write access: 6.1 ns write_local 7.3 ns write_nonlocal 18.9 ns write_global 40.7 ns write_classvar 36.2 ns write_instancevar 26.1 ns write_instancevar_slots Data structure read access: 19.1 ns read_list 19.6 ns read_deque 20.6 ns read_dict Data structure write access: 22.8 ns write_list 23.5 ns write_deque 27.8 ns write_dict Stack (or queue) operations: 54.8 ns list_append_pop 49.5 ns deque_append_pop 49.4 ns deque_append_popleft Timing loop overhead: 0.3 ns loop_overhead

15 37

[bpo-35155] Requesting a review
by Denton Liu 11 Mar '19

11 Mar '19

Hello all, A couple months back, I reported bpo-35155[1] and I submitted a PR for consideration[2]. After a couple of reviews, it seems like progress has stalled. Would it be possible for someone to review this? Thanks, Denton [1]: https://bugs.python.org/issue35155 [2]: https://github.com/python/cpython/pull/10313

2 4

Compact ordered set
by INADA Naoki 07 Mar '19

07 Mar '19

Hello, folks. I'm working on compact and ordered set implementation. It has internal data structure similar to new dict from Python 3.6. It is still work in progress. Comments, tests, and documents should be updated. But it passes existing tests excluding test_sys and test_gdb (both tests checks implementation detail) https://github.com/methane/cpython/pull/16 Before completing this work, I want to evaluate it. Following is my current thoughts about the compact ordered set. ## Preserving insertion order Order is not fundamental for set. There are no order in set in the math world. But it is convenient sometime in real world. For example, it makes doctest easy. When writing set to logs, we can use "grep" command if print order is stable. pyc is stable without PYTHONHASHSEED=0 hack. Additionally, consistency with dict is desirable. It removes one pitfall for new Python users. "Remove duplicated items from list" idiom become `list(set(duplicated))` from `list(dict.fromkeys(duplicated))`. ## Memory efficiency Hash table has dilemma. To reduce collision rate, hash table should be sparse. But it wastes memory. Since current set is optimized for both of hit and miss cases, it is more sparse than dict. (It is bit surprise that set typically uses more memory than same size dict!) New implementation partially solve this dilemma. It has sparse "index table" which items are small (1byte when table size <= 256, 2bytes when table size <= 65536), and dense entry table (each item has key and hash, which is 16bytes on 64bit system). I use 1/2 for capacity rate for now. So new implementation is memory efficient when len(s) <= 32768. But memory efficiency is roughly equal to current implementation when 32768 < len(s) <= 2**31, and worse than current implementation when len(s) > 2**31. Here is quick test about memory usage. https://gist.github.com/methane/98b7f43fc00a84964f66241695112e91 # Performance pyperformance result: $ ./python -m perf compare_to master.json oset2.json -G --min-speed=2 Slower (3): - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower (+52%) - unpickle: 29.6 us +- 2.5 us -> 44.1 us +- 2.5 us: 1.49x slower (+49%) - regex_dna: 448 ms +- 3 ms -> 462 ms +- 2 ms: 1.03x slower (+3%) Faster (4): - meteor_contest: 189 ms +- 1 ms -> 165 ms +- 1 ms: 1.15x faster (-13%) - telco: 15.8 ms +- 0.2 ms -> 15.3 ms +- 0.2 ms: 1.03x faster (-3%) - django_template: 266 ms +- 6 ms -> 259 ms +- 3 ms: 1.03x faster (-3%) - unpickle_pure_python: 818 us +- 6 us -> 801 us +- 9 us: 1.02x faster (-2%) Benchmark hidden because not significant (49) unpickle and unpickle_list shows massive slowdown. I suspect this slowdown is not caused from set change. Linux perf shows many pagefault is happened in pymalloc_malloc. I think memory usage changes hit weak point of pymalloc accidentally. I will try to investigate it. On the other hand, meteor_contest shows 13% speedup. It uses set. Other doesn't show significant performance changes. I need to write more benchmarks for various set workload. I expect new set is faster on simple creation, iteration and destruction. Especially, sequential iteration and deletion will reduce cache misses. (e.g. https://bugs.python.org/issue32846 ) On the other hand, new implementation will be slow on complex (heavy random add & del) case. ----- Any comments are welcome. And any benchmark for set workloads are very welcome. Regards, -- INADA Naoki <songofacandy(a)gmail.com>

12 22

Summary of Python tracker Issues
by Python tracker 05 Mar '19

05 Mar '19

ACTIVITY SUMMARY (2019-02-01 - 2019-02-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6998 (+13) closed 40696 (+47) total 47694 (+60) Open issues with patches: 2783 Issues opened (43) ================== #35885: configparser: indentation https://bugs.python.org/issue35885 opened by mrs.red #35886: Move PyInterpreterState into Include/internal/pycore_pystate.h https://bugs.python.org/issue35886 opened by eric.snow #35887: Doc string for updating the frozen version of importlib in _bo https://bugs.python.org/issue35887 opened by nnja #35888: ssl module - could not get the server certificate w/o complete https://bugs.python.org/issue35888 opened by Lee Eric #35889: sqlite3.Row doesn't have useful repr https://bugs.python.org/issue35889 opened by vlad #35891: urllib.parse.splituser has no suitable replacement https://bugs.python.org/issue35891 opened by jaraco #35892: Fix awkwardness of statistics.mode() for multimodal datasets https://bugs.python.org/issue35892 opened by rhettinger #35893: distutils fails to build extension on windows when it is a pac https://bugs.python.org/issue35893 opened by ronaldoussoren #35898: The TARGETDIR variable must be provided when invoking this ins https://bugs.python.org/issue35898 opened by Thomas Trummer #35899: '_is_sunder' function in 'enum' module fails on empty string https://bugs.python.org/issue35899 opened by Maxpxt #35900: Add pickler hook for the user to customize the serialization o https://bugs.python.org/issue35900 opened by pierreglaser #35901: json.dumps infinite recurssion https://bugs.python.org/issue35901 opened by MultiSosnooley #35903: Build of posixshmem.c should probe for required OS functions https://bugs.python.org/issue35903 opened by nascheme #35904: Add statistics.fmean(seq) https://bugs.python.org/issue35904 opened by rhettinger #35905: macOS build docs need refresh (2019) https://bugs.python.org/issue35905 opened by jaraco #35906: Header Injection in urllib https://bugs.python.org/issue35906 opened by push0ebp #35907: Unnecessary URL scheme exists to allow file:// reading file i https://bugs.python.org/issue35907 opened by push0ebp #35912: _testembed.c fails to compile when using --with-cxx-main in th https://bugs.python.org/issue35912 opened by pablogsal #35913: asyncore: allow handling of half closed connections https://bugs.python.org/issue35913 opened by Isaac Boukris #35915: re.search extreme slowness (looks like hang/livelock), searchi https://bugs.python.org/issue35915 opened by benspiller #35918: multiprocessing's SyncManager.dict.has_key() method is broken https://bugs.python.org/issue35918 opened by giampaolo.rodola #35919: multiprocessing: shared manager Pool fails with AttributeError https://bugs.python.org/issue35919 opened by giampaolo.rodola #35920: Windows 10 ARM32 platform support https://bugs.python.org/issue35920 opened by steve.dower #35921: Use ccache by default https://bugs.python.org/issue35921 opened by pitrou #35922: robotparser crawl_delay and request_rate do not work with no m https://bugs.python.org/issue35922 opened by joseph_myers #35923: Update the BuiltinImporter in importlib to use loader._ORIGIN https://bugs.python.org/issue35923 opened by nnja #35924: curses segfault resizing window https://bugs.python.org/issue35924 opened by Josiah Ulfers #35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster https://bugs.python.org/issue35925 opened by pablogsal #35926: Need openssl 1.1.1 support on Windows for ARM and ARM64 https://bugs.python.org/issue35926 opened by Paul Monson #35927: Intra-package References Documentation Incomplete https://bugs.python.org/issue35927 opened by ADataGman #35928: socket makefile read-write discards received data https://bugs.python.org/issue35928 opened by pravn #35930: Raising an exception raised in a "future" instance will create https://bugs.python.org/issue35930 opened by jcea #35931: pdb: "debug print(" crashes with SyntaxError https://bugs.python.org/issue35931 opened by blueyed #35933: python doc does not say that the state kwarg in Pickler.save_r https://bugs.python.org/issue35933 opened by pierreglaser #35934: Add socket.bind_socket() utility function https://bugs.python.org/issue35934 opened by giampaolo.rodola #35935: threading.Event().wait() not interruptable with Ctrl-C on Wind https://bugs.python.org/issue35935 opened by Chris Billington #35936: Give modulefinder some much-needed updates. https://bugs.python.org/issue35936 opened by brandtbucher #35937: Add instancemethod to types.py https://bugs.python.org/issue35937 opened by bup #35939: Remove urllib.parse._splittype from mimetypes.guess_type https://bugs.python.org/issue35939 opened by corona10 #35940: multiprocessing manager tests fail in the Refleaks buildbots https://bugs.python.org/issue35940 opened by pablogsal #35941: ssl.enum_certificates() regression https://bugs.python.org/issue35941 opened by schlenk #35942: posixmodule.c:path_converter() returns an invalid exception me https://bugs.python.org/issue35942 opened by lukasz.langa #35943: PyImport_GetModule() can return partially-initialized module https://bugs.python.org/issue35943 opened by pitrou Most recent 15 issues with no replies (15) ========================================== #35942: posixmodule.c:path_converter() returns an invalid exception me https://bugs.python.org/issue35942 #35940: multiprocessing manager tests fail in the Refleaks buildbots https://bugs.python.org/issue35940 #35939: Remove urllib.parse._splittype from mimetypes.guess_type https://bugs.python.org/issue35939 #35936: Give modulefinder some much-needed updates. https://bugs.python.org/issue35936 #35934: Add socket.bind_socket() utility function https://bugs.python.org/issue35934 #35931: pdb: "debug print(" crashes with SyntaxError https://bugs.python.org/issue35931 #35930: Raising an exception raised in a "future" instance will create https://bugs.python.org/issue35930 #35928: socket makefile read-write discards received data https://bugs.python.org/issue35928 #35927: Intra-package References Documentation Incomplete https://bugs.python.org/issue35927 #35926: Need openssl 1.1.1 support on Windows for ARM and ARM64 https://bugs.python.org/issue35926 #35924: curses segfault resizing window https://bugs.python.org/issue35924 #35920: Windows 10 ARM32 platform support https://bugs.python.org/issue35920 #35919: multiprocessing: shared manager Pool fails with AttributeError https://bugs.python.org/issue35919 #35918: multiprocessing's SyncManager.dict.has_key() method is broken https://bugs.python.org/issue35918 #35912: _testembed.c fails to compile when using --with-cxx-main in th https://bugs.python.org/issue35912 Most recent 15 issues waiting for review (15) ============================================= #35936: Give modulefinder some much-needed updates. https://bugs.python.org/issue35936 #35934: Add socket.bind_socket() utility function https://bugs.python.org/issue35934 #35931: pdb: "debug print(" crashes with SyntaxError https://bugs.python.org/issue35931 #35926: Need openssl 1.1.1 support on Windows for ARM and ARM64 https://bugs.python.org/issue35926 #35922: robotparser crawl_delay and request_rate do not work with no m https://bugs.python.org/issue35922 #35921: Use ccache by default https://bugs.python.org/issue35921 #35920: Windows 10 ARM32 platform support https://bugs.python.org/issue35920 #35913: asyncore: allow handling of half closed connections https://bugs.python.org/issue35913 #35906: Header Injection in urllib https://bugs.python.org/issue35906 #35903: Build of posixshmem.c should probe for required OS functions https://bugs.python.org/issue35903 #35900: Add pickler hook for the user to customize the serialization o https://bugs.python.org/issue35900 #35887: Doc string for updating the frozen version of importlib in _bo https://bugs.python.org/issue35887 #35886: Move PyInterpreterState into Include/internal/pycore_pystate.h https://bugs.python.org/issue35886 #35878: ast.c: end_col_offset may be used uninitialized in this functi https://bugs.python.org/issue35878 #35876: test_start_new_session for posix_spawnp fails https://bugs.python.org/issue35876 Top 10 most discussed issues (10) ================================= #35813: shared memory construct to avoid need for serialization betwee https://bugs.python.org/issue35813 19 msgs #35904: Add statistics.fmean(seq) https://bugs.python.org/issue35904 11 msgs #35913: asyncore: allow handling of half closed connections https://bugs.python.org/issue35913 11 msgs #35921: Use ccache by default https://bugs.python.org/issue35921 9 msgs #35706: Make it easier to use a venv with an embedded Python interpret https://bugs.python.org/issue35706 7 msgs #35893: distutils fails to build extension on windows when it is a pac https://bugs.python.org/issue35893 7 msgs #30670: pprint for dict in sorted order or insert order? https://bugs.python.org/issue30670 6 msgs #35907: Unnecessary URL scheme exists to allow file:// reading file i https://bugs.python.org/issue35907 6 msgs #34572: C unpickling bypasses import thread safety https://bugs.python.org/issue34572 5 msgs #35933: python doc does not say that the state kwarg in Pickler.save_r https://bugs.python.org/issue35933 5 msgs Issues closed (46) ================== #20001: pathlib inheritance diagram too large https://bugs.python.org/issue20001 closed by inada.naoki #22474: No explanation of how a task gets destroyed in asyncio 'task' https://bugs.python.org/issue22474 closed by cheryl.sabella #24087: Documentation doesn't explain the term "coroutine" (PEP 342) https://bugs.python.org/issue24087 closed by paul.moore #24209: Allow IPv6 bind in http.server https://bugs.python.org/issue24209 closed by jaraco #26256: Fast decimalisation and conversion to other bases https://bugs.python.org/issue26256 closed by skrah #27344: zipfile *does* support utf-8 filenames https://bugs.python.org/issue27344 closed by cheryl.sabella #29734: os.stat handle leak https://bugs.python.org/issue29734 closed by steve.dower #30130: array.array is not an instance of collections.MutableSequence https://bugs.python.org/issue30130 closed by cheryl.sabella #32560: [EASY C] inherit the py launcher's STARTUPINFO https://bugs.python.org/issue32560 closed by steve.dower #33316: Windows: PyThread_release_lock always fails https://bugs.python.org/issue33316 closed by steve.dower #33895: LoadLibraryExW called with GIL held can cause deadlock https://bugs.python.org/issue33895 closed by steve.dower #34691: _contextvars missing in xmaster branch Windows build? https://bugs.python.org/issue34691 closed by steve.dower #35299: LGHT0091: Duplicate symbol 'File:include_pyconfig.h' found https://bugs.python.org/issue35299 closed by steve.dower #35358: Document that importlib.import_module accepts names that are n https://bugs.python.org/issue35358 closed by matrixise #35485: tkinter windows turn black while resized using Tk 8.6.9.1 on m https://bugs.python.org/issue35485 closed by ned.deily #35605: backported patch requires new sphinx, minimum sphinx version w https://bugs.python.org/issue35605 closed by ned.deily #35606: Add prod() function to the math module https://bugs.python.org/issue35606 closed by rhettinger #35615: "RuntimeError: Dictionary changed size during iteration" when https://bugs.python.org/issue35615 closed by pitrou #35642: _asynciomodule.c compiled in both pythoncore.vcxproj and _asyn https://bugs.python.org/issue35642 closed by steve.dower #35686: BufferError with memory.release() https://bugs.python.org/issue35686 closed by skrah #35692: pathlib.Path.exists() on non-existent drive raises WinError in https://bugs.python.org/issue35692 closed by steve.dower #35758: Disable x87 control word for MSVC ARM compiler https://bugs.python.org/issue35758 closed by Minmin.Gong #35851: Make search result in online docs keep their position when sea https://bugs.python.org/issue35851 closed by xtreak #35861: test_named_expressions raises SyntaxWarning https://bugs.python.org/issue35861 closed by emilyemorehouse #35862: Change the environment for a new process https://bugs.python.org/issue35862 closed by steve.dower #35872: Creating venv from venv no longer works in 3.7.2 https://bugs.python.org/issue35872 closed by steve.dower #35873: Controlling venv from venv no longer works in 3.7.2 https://bugs.python.org/issue35873 closed by steve.dower #35877: parenthesis is mandatory for named expressions in while statem https://bugs.python.org/issue35877 closed by emilyemorehouse #35879: test_type_comments leaks references https://bugs.python.org/issue35879 closed by gvanrossum #35884: Add variable access benchmark to Tools/Scripts https://bugs.python.org/issue35884 closed by rhettinger #35890: Cleanup some non-consistent API callings https://bugs.python.org/issue35890 closed by steve.dower #35894: Apparent regression in 3.8-dev: 'TypeError: required field "ty https://bugs.python.org/issue35894 closed by gvanrossum #35895: the test suite of pytest failed with 3.8.0a1 https://bugs.python.org/issue35895 closed by gvanrossum #35896: sysconfig.get_platform returns wrong value when Python 32b is https://bugs.python.org/issue35896 closed by steve.dower #35897: Support list as argument to .startswith() https://bugs.python.org/issue35897 closed by rhettinger #35902: Forking from background thread https://bugs.python.org/issue35902 closed by pitrou #35908: build with building extension modules as builtins is broken in https://bugs.python.org/issue35908 closed by doko #35909: Zip Slip Vulnerability https://bugs.python.org/issue35909 closed by christian.heimes #35910: Curious problem with my choice of variables https://bugs.python.org/issue35910 closed by matrixise #35911: add a cell construtor, and expose the cell type in Lib/types.p https://bugs.python.org/issue35911 closed by pitrou #35914: [2.7] PyStructSequence objects not behaving like nametuple https://bugs.python.org/issue35914 closed by eric.snow #35916: 3.6.5 try/except/else/finally block executes code with typos, https://bugs.python.org/issue35916 closed by SilentGhost #35917: multiprocessing: provide unit-tests for manager classes and sh https://bugs.python.org/issue35917 closed by pitrou #35929: Spam https://bugs.python.org/issue35929 closed by Mariatta #35932: Interpreter gets stuck while applying a regex pattern https://bugs.python.org/issue35932 closed by tim.peters #35938: crash of METADATA file cannot be fixed by reinstall of python https://bugs.python.org/issue35938 closed by steven.daprano

8 14

Another update for PEP 394 -- The "python" Command on Unix-Like Systems
by Petr Viktorin 01 Mar '19

01 Mar '19

PEP 394 says: > This recommendation will be periodically reviewed over the next few > years, and updated when the core development team judges it > appropriate. As a point of reference, regular maintenance releases > for the Python 2.7 series will continue until at least 2020. I think it's time for another review. I'm especially worried about the implication of these: - If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line. So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :( Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893

25 64

before I open an issue re: posix.stat and/or os.stat
by Michael 01 Mar '19

01 Mar '19

My focus is AIX - and I believe I found a bug in AIX include files in 64-bit mode. I'll take that up with IBM and AIX support. However, this issue might also be valid in Python3. The following is from Centos, not AIX Python 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.maxsize 9223372036854775807 >>> import posix >>> posix.stat("/tmp/xxx") posix.stat_result(st_mode=33188, st_ino=33925869, st_dev=64768L, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1550742595, st_mtime=1550742595, st_ctime=1550742595) >>> st=posix.stat("/tmp/xxx") >>> dev=st.st_dev >>> min=posix.minor(dev) >>> maj=posix.major(dev) >>> min,max (0, <built-in function max>) >>> min 0 >>> max <built-in function max> >>> maj 253 >>> posix.minor(dev) 0 >>> posix.major(655536) 2560 >>> posix.major(65536) 256 >>> posix.major(256) 1 >>> import os >>> os.major(256) 1 >>> In AIX - 64-bit mode Python 3.8.0a1+ (heads/master:e7a4bb554e, Feb 20 2019, 18:40:08) [C] on aix7 Type "help", "copyright", "credits" or "license" for more information. >>> import sys,os,posix >>> sys.maxsize 9223372036854775807 >>> posix.major(256) 0 >>> posix.major(65536) 1 >>> posix.stat("/tmp/xxx") os.stat_result(st_mode=33188, st_ino=12, st_dev=-9223371993905102841, st_nlink=1, st_uid=202, st_gid=1954, st_size=0, st_atime=1550690105, st_mtime=1550690105, st_ctime=1550690105) AIX 32-bit: root@x066:[/data/prj/python/git/python3-3.8.0.66]./python Python 3.8.0a1+ (heads/master:e7a4bb554e, Feb 19 2019, 11:22:56) [C] on aix6 Type "help", "copyright", "credits" or "license" for more information. >>> import os,sys,posix >>> sys.maxsize 2147483647 >>> posix.major(65536) 1 >>> posix.stat("/tmp/xxx") os.stat_result(st_mode=33188, st_ino=149, st_dev=655367, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1550743517, st_mtime=1550743517, st_ctime=1550743517) To make it easier to view: buildbot@x064:[/home/buildbot]cat osstat.c #include <sys/types.h> #include <sys/stat.h> #include <sys/sysmacros.h> #include <stdio.h> main() { dev_t dev; char *path = "/tmp/xxx"; struct stat st; int minor,major; lstat(path,&st); printf("size: %d\n", sizeof(st.st_dev)); dev = st.st_dev; minor = minor(dev); major = major(dev); printf("%016lx %ld %ld\n",dev,dev, (unsigned) dev); printf("%d,%d\n",major,minor); } buildbot@x064:[/home/buildbot]OBJECT_MODE=32 cc osstat.c -o osstat-32 && ./osstat-32 size: 4 00000000000a0007 655367 655367 10,7 And here is the AIX behavior (and bug - major() macro!) buildbot@x064:[/home/buildbot]OBJECT_MODE=64 cc osstat.c -o osstat-64 && ./osstat-64 size: 8 8000000a00000007 -9223371993905102841 7 0,7 The same on AIX 6 (above is AIX7) - and also with gcc: root@x068:[/data/prj]gcc -maix64 osstat.c -o osstat-64 && ./osstat-64 size: 8 8000000a00000007 -9223371993905102841 42949672967 0,7 root@x068:[/data/prj]gcc -maix32 osstat.c -o osstat-32 && ./osstat-32 size: 4 00000000000a0007 655367 0 10,7 root@x068:[/data/prj] So, the AIX 'bug' with the macro major() has been around for ages - but ALSO setting the MSB of the st_dev. +++++ Now my question: Will this continue to be enough space - i.e., is the Dev size going to be enough? +2042 #ifdef MS_WINDOWS +2043 PyStructSequence_SET_ITEM(v, 2, PyLong_FromUnsignedLong(st->st_dev)); +2044 #else +2045 PyStructSequence_SET_ITEM(v, 2, _PyLong_FromDev(st->st_dev)); +2046 #endif +711 #define _PyLong_FromDev PyLong_FromLongLong It seems so - however, Is there something such as PyUnsignedLong and is that large enough for a "long long"? and if it exists, would that make the value positive (for the first test). posix.major and os.major will need to mask away the MSB and posix.makedev and os.makedev will need to add it back. OR - do I need to make the PyStat values "the same" in both 32-bit and 64-bit? Puzzled on what you think is the correct approach. Michael

3 2

[RELEASE] Python 3.8.0a1 is now available for testing
by Łukasz Langa 28 Feb '19

28 Feb '19

I packaged another release. Go get it here: https://www.python.org/downloads/release/python-380a2/ Python 3.8.0a2 is the second of four planned alpha releases of Python 3.8, the next feature release of Python. During the alpha phase, Python 3.8 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The next preview release, 3.8.0a3, is planned for 2019-03-25. This time around the stable buildbots were a bit less green than they should have. This early in the cycle, I didn't postpone the release and I didn't use the revert hammer. But soon enough, I will. Let's make sure future changes keep the buildbots happy. - Ł

4 4

datetime.timedelta total_microseconds
by Richard Belleville 28 Feb '19

28 Feb '19

In a recent code review, the following snippet was called out as reinventing the wheel: _MICROSECONDS_PER_SECOND = 1000000 def _timedelta_to_microseconds(delta): return int(delta.total_seconds() * _MICROSECONDS_PER_SECOND) The reviewer thought that there must already exist a standard library function that fulfills this functionality. After we had both satisfied ourselves that we hadn't simply missed something in the documentation, we decided that we had better raise the issue with a wider audience. Does this functionality already exist within the standard library? If not, would a datetime.timedelta.total_microseconds function be a reasonable addition? I would be happy to submit a patch for such a thing. Richard Belleville

12 27

Can I get a review for PR 10437?
by Kevin Adler 28 Feb '19

28 Feb '19

2 1

Proposed dates for Python 3.4.10 and Python 3.5.7
by Larry Hastings 27 Feb '19

27 Feb '19

Howdy howdy! It's time to make the next bugfix release of 3.5--and the /final/ release /ever/ of Python 3.4. Here's the schedule I propose: 3.4.10rc1 and 3.5.7rc1 - Saturday March 2 2019 3.4.10 final and 3.5.7 final - Saturday March 16 2019 What's going in these releases? Not much. I have two outstanding PRs against 3.5: bpo-33127 GH-10994: Compatibility patch for LibreSSL 2.7.0 bpo-34623 GH-9933: XML_SetHashSalt in _elementtree and one PR against 3.4: bpo-34623 GH-9953: Use XML_SetHashSalt in _elementtree I expect to merge all three of those, I just need to get around to it. There's one more recent security fix (bpo-35746, GH-11569) that I want in these releases that still needs backporting. And that's the entire list. bpo-34623 is the only current release blocker for either 3.4 or 3.5--I'm not aware of anything else in the pipeline. If you have anything you think needs to go into the next 3.5, or the final 3.4, and it's /not/ listed above, please either file a GitHub PR, file a release-blocker bug on bpo, or email me directly. Good night sweet Python 3.4, and flights of angels sing thee to thy rest! //arry/

3 4

Announcing: signups are open for the 2019 Python Language Summit
by Łukasz Langa 27 Feb '19

27 Feb '19

The Python Language Summit is an event for the developers of Python implementations (CPython, PyPy, Jython, and so on) to share information, discuss our shared problems, and — hopefully — solve them. These issues might be related to the language itself, the standard library, the development process, status of Python 3.8 (or plans for 3.9), the documentation, packaging, the website, et cetera. The Summit focuses on discussion more than on presentations. If you’d like to attend **and actively participate** in the discussions during the Language Summit, please fill in this form by March 21st 2019: https://goo.gl/forms/pexfOGDjpV0BWMer2 We will be evaluating all applications and confirm your attendance by April 15th. Note: **your attendance is not confirmed** until you heard back from us. You don't need to be registered for PyCon in order to attend the summit. One of the goals of the Language Summit is to speed up the discussions and decision making process. Communication over Discourse (or mailing lists!) is generally more time consuming. As part of efforts to make this event more open and less mysterious, we are not requiring invitations by core developers anymore. However, please understand that we will have to be selective as space and time are limited. In particular, we are prioritizing active core contributors, as well as those who we believe will be able to improve the quality of the discussions at the event and bring a more diverse perspective to core Python developers. As for other changes this year, A. Jesse Jiryu Davis will be covering the event and will post a detailed write up on the official blog of the PSF shortly after the conference. We hope to see you at the Summit! - Mariatta and Łukasz PS. If you have any questions, the Users section of our Discourse instance (https://discuss.python.org/c/users) is the best place to ask. For private communication, write to mariatta(a)python.org and/or lukasz(a)python.org.

1 1

OpenSSL 1.1.1 update for 3.7/3.8
by Christian Heimes 26 Feb '19

26 Feb '19

Hi, today's OpenSSL release of 1.0.2r and 1.1.1b reminded me of OpenSSL's release strategy [1]. OpenSSL 1.0.2 will reach EOL on 2019-12-31, 1.1.0 will reach EOL on 2019-09-11 (one year after release of OpenSSL 1.1.1). First the good news: There is no need to take any action for 2.7 to 3.6. As of today, Python 2.7, 3.5, and 3.6 are using OpenSSL 1.0.2. Python 3.6.8 (2018-12-24) and 3.5.5 (2018-02-05) were the last regular update with binary packages. 3.5.6 is a source-only security release. 3.6.9 will be the first source-only security release of the 3.6 series. Python 2.7 will reach EOL just a day after OpenSSL 1.0.2 reaches EOL. IMHO it's fine to ship the last 2.7 build with an OpenSSL version that was EOLed just 24h earlier. Python 3.7 and master (3.8) are affected. As of now, both branches use OpenSSL 1.1.0 and must be updated to 1.1.1 soonish. Ned has scheduled 3.7.3 release for 2019-03-25. That's still well within the release schedule for 1.1.0. I suggest that we update to 1.1.1 directly after the release of Python 3.7.3 and target 3.7.4 as first builds with TLS 1.3 support. That gives Victor, Steve, and me enough time to sort out the remaining issues. In worst case we could revert the update and postpone the update to 3.7.5. Or we disable TLS 1.3 support by default in Mac and Windows builds. Christian [1] https://www.openssl.org/policies/releasestrat.html

2 3

New member
by Aratz Manterola Lasa 26 Feb '19

26 Feb '19

I do not know if I should be doing this, maybe I have been cheated, but this is my introduction: Hello, my name is Aratz, I am a Computer Engineering Bachelor's (not, not science) student on my 3rd grade. I love Python's gramatic efficiency, but most of all, how much I learn thanks to its community. I want to learn more, and if it is possible from the best ones. So... I ended up here. Thanks for your attention.

1 0

Asking for reversion
by Antoine Pitrou 25 Feb '19

25 Feb '19

Hello, I'd like to ask for the reversion of the changes done in https://github.com/python/cpython/pull/11664 The reason is simple: the PR isn't complete, it lacks docs and tests. It also didn't pass any review (this was pointed by Ronald), even though it adds 1300 lines of code. No programmer is perfect, so it's statistically likely that the PR is defective. With git, forks and branches, we definitely don't need to commit unfinished PRs to the main repo. It's perfectly fine to maintain some non-trivial piece of work in a separate fork. People do it on a regular basis (for example I have currently two such long-lived branches: one for PEP 556 and one for PEP 574). Also, this is not the first time this happened. Another multiprocessing PR was merged some years ago without any docs or tests: https://bugs.python.org/issue28053 Today that work /still/ lacks docs or tests, and there is a suspicion that it doesn't work as intended (see issue comments). It's probably too late to revert it, but it's definitely a slippery slope. Regards Antoine.

13 30

"Good first issues" on the bug tracker
by Cheryl Sabella 25 Feb '19

25 Feb '19

Hello! Due to the upcoming sprints at PyCon, there will be issues marked in the bug tracker as 'Good first issues'. Please do *not* submit pull requests for these as it is difficult to find suitable issues. Instead of working on these issues, I'd like to propose a much more difficult challenge to current contributors looking for something to work on. Please find issues which can be tackled at the sprint! They are out there, but the trick is finding them. :-) Here's the link to the mentored sprint page to understand the audience: https://us.pycon.org/2019/hatchery/mentoredsprints/ I don't think this needs to be limited to documentation changes, but those are the obvious choice (even the devguide suggests it). The goal is really workflow, meaning the new contributor doesn't necessarily have to solve the problem themselves - if an issue says: x function in y module needs to have z applied, then there's a lot of hurdles there for a newcomer which may not involve solving the problem. If you find anything you think is suitable, please add a comment with 'good first issue' and nosy me or Mariatta on it. If you're unsure, then nosy us anyway. It would be awesome to have too many issues to choose from than not enough. If an issue isn't worked on at the sprint, then it would still be available to new contributors afterwards, so the exercise of flagging the issues wouldn't be wasted effort. Thanks! Cheryl

9 14

Add minimal information with a new issue?
by Stephane Wirtel 21 Feb '19

21 Feb '19

Hi, What do you think if we suggest a "template" for the new bugs? For example: * Python version (exact version) * Operating System * on Linux -> Distribution (python can have some patches) * Add a script for the reproduction of the bug * Eventually, try with the docker images (on i386,x86_64, ...) * etc... We can lost a lot of time just trying to find the right information for the reproduction of the issue. Cheers, Stéphane -- Stéphane Wirtel - https://wirtel.be - @matrixise

6 7

Making PyInterpreterState an opaque type
by Eric Snow 21 Feb '19

21 Feb '19

Hi all, I've been working on the runtime lately, particularly focused on my multi-core Python project. One thing that would help simplify changes in this area is if PyInterpreterState were defined in Include/internal. This would mean the type would be opaque unless Py_BUILD_CORE were defined. The docs [1] already say none of the struct's fields are public. Furthermore, Victor already moved it into Include/cpython (i.e. not in the stable ABI) in November. Overall, the benefit of making internal types like this opaque is realized in our ability to change the internal details without further breaking C-API users. Realistically, there may be extension modules out there that use PyInterpreterState fields directly. They would break. I expect there to be few such modules and fixing them would not involve great effort. We'd have to add any missing accessor functions to the public C-API, which I see as a good thing. I have an issue [2] open for the change and a PR open. My PR already adds an entry to the porting section of the 3.8 What's New doc about dealing with PyInterpreterState. Anyway, I just wanted to see if there are any objections to making PyInterpreterState an opaque type outside of core use. -eric p.s. I'd like to do the same with PyThreadState, but that's a bit trickier [3] and not one of my immediate needs. :) [1] https://docs.python.org/3/c-api/init.html#c.PyInterpreterState [2] https://bugs.python.org/issue35886 [3] https://bugs.python.org/issue35949

15 40

Making PyInterpreterState an opaque type
by Stefan Krah 21 Feb '19

21 Feb '19

Victor Stinner wrote: > Premature optimization is the root of all evil. Most C extensions use > premature optimization which are causing us a lot of troubles nowadays > when we want to make the C API evolve and cause issues to PyPy which > has issues to reimplement the C API on top of their different object > model with a different GC. I'm intimately familiar with several C extensions in the math/science space and this is not the impression I get at all. Most people are happy if they get things to work, because the actual problems are much harder than focusing on private vs. public API. As for PyPy, if I understood correctly, Armin Rigo was skeptical of the proposed plan and favored publishing an API as a third party package. Stefan Krah

1 0

Question - Bug Triage for 3.4 & 3.5
by Stephane Wirtel 21 Feb '19

21 Feb '19

Hi, As you know, Python 3.4 and 3.5 are in security mode and the EOL for these versions are respectively 2019-03-16 and 2020-09-13. Number of issues 3.4: 1530 issues 3.5: 1901 issues But some issues are not related to the security. Could we update these issues (non-security) to 3.6/3.7 & 3.8? Cheers, Stéphane -- Stéphane Wirtel - https://wirtel.be - @matrixise

5 7

Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the "target" platform
by Xavier de Gaye 20 Feb '19

20 Feb '19

> [Any reason for dropping python-dev?] Sorry. just clicked the wrong button. > And the answer is a resounding "yes, it returns the target platform"? It seems you're saying this, but the wording of your email sounds just enough of a question that I'm not sure whether you are definitively answering it or not. The answer is yes indeed, it returns the target platform. This is a listing of the non-obvious steps that lead to this conclusion. Xavier On 2/19/19 8:45 PM, Steve Dower wrote: > [Any reason for dropping python-dev?] > > On 19Feb2019 1139, Xavier de Gaye wrote: >> Is distutils.util.get_platform() the "current" or the "target" platform ? > > I *think* you're answering this below, yes? > >> * When cross-compiling on posix platforms using autoconf, configure is >> run with the '--host=host-type' [1] command line option to specify >> the target platform. >> >> * The AC_CANONICAL_HOST macro is used by configure.ac to get the >> canonical variables `host' and `host_cpu' [2]. >> Those variables are used to compute _PYTHON_HOST_PLATFORM. >> >> * The Makefile generated by configure runs setup.py, >> generate-posix-vars, etc... on the build platform using >> PYTHON_FOR_BUILD, a native python interpreter that is set >> to run with the _PYTHON_HOST_PLATFORM environment variable. >> >> * get_platform() in setup.py and in Lib/distutils/util.py returns the >> value of _PYTHON_HOST_PLATFORM when cross-compiling. >> >> So the process of cross-compilation on posix platforms has >> get_platform() return the target ('host' in autoconf terminology) platform. >> >> [1] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/ht… >> [2] https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/ht… > > And the answer is a resounding "yes, it returns the target platform"? It seems you're saying this, but the wording of your email sounds just enough of a question that I'm not sure whether you are definitively answering it or not. > > Cheers, > Steve

2 1

buildbottest on Android emulator with docker
by Xavier de Gaye 20 Feb '19

20 Feb '19

The following command runs the buildbottest on an Android emulator with docker (it will use a little bit more than 11 GB): $ docker run -it --privileged xdegaye/abifa:r14b-24-x86_64-master This command does: * pull an image from the Docker hub (only the first time that the command is run, note that this a 2 GB download !) and start a container * pull the latest changes from the GitHub cpython repository and cross-compile python * start an Android emulator and install python on it * run the buildbottest target of cpython Makefile The image is built from a Dockerfile [2]. This same image can also be used with the 'bash' command line argument to enter bash in the container and run python interactively on the emulator [1]. If the 'docker run' command also sets a bind mount to a local cpython repository, then it is possible to develop/debug/fix python on the emulator running in this container using one's own clone of cpython. [1] documentation at https://xdegaye.gitlab.io/abifa/docker.html [2] Dockerfile at https://gitlab.com/xdegaye/abifa/blob/master/docker/Dockerfile.r14b-24-x86_… Xavier

3 5

new binary wheels PEP idea
by Alexander Revin 19 Feb '19

19 Feb '19

Hi all, I have an idea regarding Python binary wheels on non-glibc platforms, and it seems that initially I've posted it to the wrong list ([1]) Long story short, the proposal is to use platform tuples (like compiler ones) for wheel names, which will allow much broader platform support, for example: package-1.0-cp36-cp36m-amd64_linux_gnu.whl package-1.0-cp36-cp36m-amd64_linux_musl.whl So eventually only {platform tag} part will be modified. Glibc/musl detection is quite trivial and eventually will be based on existing one in PEP 513 [2]. Let me know what you think. Best regards, Alex [1] https://mail.python.org/pipermail/python-list/2019-February/739524.html [2] https://www.python.org/dev/peps/pep-0513/#id49

3 3

Adding test.support.safe_rmpath()
by Giampaolo Rodola' 19 Feb '19

19 Feb '19

Hello, after discovering os.makedirs() has no unit-tests ( https://bugs.python.org/issue35982) I was thinking about working on a PR to increase the test coverage of fs-related os.* functions. In order to do so I think it would be useful to add a convenience function to "just delete something if it exists", regardless if it's a file, directory, directory tree, etc., and include it into test.support module. Basically it would be very similar to "rm -rf". I use something like this into psutil: https://github.com/giampaolo/psutil/blob/3ea94c1b8589891a8d1a5781f0445cb508… I find this paradigm especially useful when testing functions involving two files ("src" and "dst"). E.g. in case of os.renames() unit-tests I would write something like this: class RenamesTest(unittest.TestCase): srcname = support.TESTFN dstname = support.TESTFN + '2' def setUp(self): test.support.rmpath(self.srcname) test.support.rmpath(self.dstname) tearDown = setUp def test_rename_file(self): ... def test_rename_dir(self): ... def test_rename_failure(self): # both src and dst will not exist ... With the current utilities included in test.support the setUp function above would be written as such: def setUp(self): for path in (self.srcname, self.dstname): if os.path.isdir(path): test.support.rmtree(path) elif os.path.exists(path): test.support.unlink(path) Extra: one may argue whether this utility could be included into shutil module instead. The extra advantage of test.support.rmtree and test.support.unlink though, is that on Windows they use a timeout, catching "file is currently in use" exceptions for some time before giving up. That IMO would probably make this utility function not palatable for inclusion into shutil module, so test.support would probably be a better landing place. Thoughts? -- Giampaolo - http://grodola.blogspot.com

11 17

Request review for bpo-35153
by Cédric Krier 17 Feb '19

17 Feb '19

Hi, A few months ago, I submitted bpo-35153 with a PR to allow to set headers from xmlrpc.client.ServerProxy. Is there a core developer willing to review it? It will be great to have it for Python 3.8. https://bugs.python.org/issue35153 https://github.com/python/cpython/pull/10308 Thanks, -- Cédric Krier - B2CK SPRL Email/Jabber: cedric.krier(a)b2ck.com Tel: +32 472 54 46 59 Website: http://www.b2ck.com/

1 0

[RELEASE] Python 2.7.16 release candidate 1
by Benjamin Peterson 16 Feb '19

16 Feb '19

I'm pleased to announce the immediate availability of Python 2.7.16 release candidate 1. This is a prerelease for yet another bug fix release in the Python 2.7.x series. It includes over 100 fixes over Python 2.7.15. See the changelog at https://raw.githubusercontent.com/python/cpython/baacaac06f93dd624c9d7b3bac… for full details. Downloads are at: https://www.python.org/downloads/release/python-2716rc1/ Please test your software against the new release and report any issues to https://bugs.python.org/ If all goes according to plan, Python 2.7.16 final will be released on March 2. All the best, Benjamin

1 0

Is distutils.util.get_platform() the "current" or the "target" platform
by Steve Dower 16 Feb '19

16 Feb '19

As part of adding ARM32 support for Windows, we need to enable cross-compilation in distutils. This is easy enough, though it requires somehow getting the target platform as well as the current platform. Right now, the change at https://github.com/python/cpython/pull/11774 adds a get_target_platform() function for this and updates (as far as I can tell) all uses of get_platform() to use this instead. I would rather just change get_platform() to return the target platform. The current docs are somewhat vague on exactly what this function does, and I suspect have been mostly written from an "always build from source" mentality that may have implied, but not explicitly considered cross-compilation. https://docs.python.org/3/distutils/apiref.html#distutils.util.get_platform "Return a string that identifies the current platform. This is used mainly to distinguish platform-specific build directories and platform-specific built distributions." So it says "current" platform, explicitly says that "os.uname()" is the source, but then goes on to say: "For non-POSIX platforms, currently just returns sys.platform." Which is incorrect, as sys.platform is always "win32" always but get_platform() already returns "win-amd64" for 64-bit builds. And also: "For Mac OS X systems the OS version reflects the minimal version on which binaries will run (that is, the value of MACOSX_DEPLOYMENT_TARGET during the build of Python), not the OS version of the current system." So it seems like this function is already returning "the default platform that should be used when building extensions" - ignoring bugs in libraries that monkeypatch distutils, the "--plat-name" option should entirely override the return value of this function. Given this, does it seem to be okay to have it determine and return the target platform rather than the host platform? Right now, that would only affect the new target of building for win-arm32, but I would also like to update the documentation to make it more about how this value should be used rather than where it comes from. Any objections or concerns? Cheers, Steve

3 5

OpenSSL 1.1.1 fixes merged into Python 2.7
by Victor Stinner 15 Feb '19

15 Feb '19

Hi, I reviewed and merged pull requests written by my colleague Charalampos Stratakis to backport OpenSSL 1.1.1 fixes into the future Python 2.7.16. Benjamin Peterson (Python 2.7 release manager) wrote me: "I would very much like to see 1.1.1 support in a Python 2.7 release." These changes are backports of Python 3.6 changes written by Christian Heimes. With these changes, Python 2.7 becomes more secure and should be closer to Python 3.6 security. I apologize for merging these changes late in 2.7.16 devcycle, but we were very busy with higher priority issues :-( I hope that 2.7.16 release candidate will provide enough time to test properly these changes (and fix regressions if any). So far, I'm only aware of one issue on one specific buildbot worker, but I'm not sure that the test failures are regressions caused by merged ssl changes (the worker was offline for 1 month for an unknown reason): https://bugs.python.org/issue33570 Summary of the ssl changes: (*) ssl.SSLContext is now created with secure default values. The options OP_NO_COMPRESSION, OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE, OP_NO_SSLv2 (except for PROTOCOL_SSLv2), OP_NO_SSLv3 (except for PROTOCOL_SSLv3) are set by default. The initial cipher suite list contains only "HIGH" ciphers, no "NULL" ciphers and no "MD5" ciphers (except for PROTOCOL_SSLv2). (*) OpenSSL 1.1.1 has TLS 1.3 cipher suites enabled by default. The suites cannot be disabled with SSLContext.set_ciphers(). (*) Add a new ssl.OP_ENABLE_MIDDLEBOX_COMPAT constant (*) Tools/ssl/multissltests.py has been updated for OpenSSL 1.1.1. I merged 4 changes into 2.7: commit c49f63c1761ce03df7850b9e0b31a18c432dac64 Author: stratakis <cstratak(a)redhat.com> Date: Fri Feb 15 14:17:12 2019 +0100 [2.7] bpo-33570: TLS 1.3 ciphers for OpenSSL 1.1.1 (GH-6976) (GH-8760) (GH-10607) Change TLS 1.3 cipher suite settings for compatibility with OpenSSL 1.1.1-pre6 and newer. OpenSSL 1.1.1 will have TLS 1.3 cipers enabled by default. Also update multissltests to test with latest OpenSSL. Signed-off-by: Christian Heimes <christian(a)python.org>. (cherry picked from commit 3e630c541b35c96bfe5619165255e559f577ee71) Co-authored-by: Christian Heimes <christian(a)python.org> commit b8eaec697a2b5d9d2def2950a0aa50e8ffcf1059 Author: stratakis <cstratak(a)redhat.com> Date: Fri Feb 15 15:24:11 2019 +0100 [2.7] bpo-28043: improved default settings for SSLContext (GH-10608) The options OP_NO_COMPRESSION, OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE, OP_NO_SSLv2 (except for PROTOCOL_SSLv2), and OP_NO_SSLv3 (except for PROTOCOL_SSLv3) are set by default. The initial cipher suite list contains only HIGH ciphers, no NULL ciphers and MD5 ciphers (except for PROTOCOL_SSLv2). (cherry picked from commit 358cfd426ccc0fcd6a7940d306602138e76420ae) commit 28eb87f4f558952f259fada7be1ab5b31b8a91ef (upstream/2.7, 2.7) Author: stratakis <cstratak(a)redhat.com> Date: Fri Feb 15 17:18:58 2019 +0100 Fixup from test_ssl test_default_ecdh_curve (GH-11877) Partial backport from cb5b68abdeb1b1d56c581d5b4d647018703d61e3 Co-authored-by: Christian Heimes <christian(a)python.org> commit 2149a9ad7a9d39d7d680ec0fb602042c91057484 (HEAD -> 2.7, upstream/2.7) Author: stratakis <cstratak(a)redhat.com> Date: Fri Feb 15 18:27:44 2019 +0100 [2.7] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761) (GH-11876) Backport of TLS 1.3 related fixes from 3.7. Misc fixes and workarounds for compatibility with OpenSSL 1.1.1 from git master and TLS 1.3 support. With OpenSSL 1.1.1, Python negotiates TLS 1.3 by default. Some test cases only apply to TLS 1.2. OpenSSL 1.1.1 has added a new option OP_ENABLE_MIDDLEBOX_COMPAT for TLS 1.3. The feature is enabled by default for maximum compatibility with broken middle boxes. Users should be able to disable the hack and CPython's test suite needs it to verify default options Signed-off-by: Christian Heimes <christian(a)python.org> (cherry picked from commit 2a4ee8aa01d61b6a9c8e9c65c211e61bdb471826) And there is a minor multissltests update that's going will be merged as well: https://github.com/python/cpython/pull/11879 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 0

Python 3.8.0a1 with sqlite 3.27.1 -> OK
by Stephane Wirtel 15 Feb '19

15 Feb '19

Hi all, I wanted to test with the new version of SQLite3 3.27.1 and there is no issue (compiled with a debian:latest docker image and the last version of 3.27.1). Sorry, it's not a bug, I wanted to inform you there is no issue with the last stable version of SQLite3. Have a nice week-end, Stéphane root@3b7c342683ff:/src# ./python -m test test_sqlite -v == CPython 3.8.0a1+ (default, Feb 15 2019, 16:05:54) [GCC 6.3.0 20170516] == Linux-4.20.7-200.fc29.x86_64-x86_64-with-glibc2.17 little-endian == cwd: /src/build/test_python_15721 == CPU count: 4 == encodings: locale=UTF-8, FS=utf-8 Run tests sequentially 0:00:00 load avg: 1.99 [1/1] test_sqlite test_sqlite: testing with version '2.6.0', sqlite_version '3.27.1' ... Ran 288 tests in 0.553s OK (skipped=3) == Tests result: SUCCESS == 1 test OK. Total duration: 654 ms Tests result: SUCCESS -- Stéphane Wirtel - https://wirtel.be - @matrixise

1 0

An update on heap-allocated types
by Eddie Elizondo 14 Feb '19

14 Feb '19

I’ll be adding a change to move the Py_INCREF of heap allocated types from PyType_GenericAlloc to PyObject_Init. You can follow the discussion and/or add comments to: https://bugs.python.org/issue35810. This change will make types created through PyType_FromSpec behave like classes in managed code. Thus, making CPython much safer. Without this change, there are a couple of edge cases where the use of PyObject_{,GC}_New{,Var} does not correctly increase refcount. This leads to weird behavior, especially when migrating types from PyType_Ready to PyType_FromSpec. For example, consider a static type with tp_new = NULL and tp_dealloc = NULL. This type initializes instances through PyObject_New and never increases the type’s refcount. tp_dealloc will be a no-op since it's NULL and it's a static type. When this type is migrated to PyType_FromSpec, tp_dealloc will now inherit subtype_dealloc which decrefs the type. This leads to a crash. For the vast majority of existing code this should not have a visible side effect. And, at worst, this will only cause some type to become immortal. I’ve added instructions in the “Porting to Python 3.8” to correctly account for this new incref along with examples. In general, there are only two cases that would require any modification: 1. If the type creates instances through PyObject_{,GC}_New{,Var} and the type manually increfs afterwards. The fix here is to remove that manual incref. 2. If the type has a custom tp_dealloc and it’s not decrefing the type. The fix here is that a custom tp_dealloc should ALWAYS decref the type. Open to feedback/discussion so feel free to reply if you have any questions! - Eddie Elizondo

1 0

2.7.16 release dates
by Benjamin Peterson 12 Feb '19

12 Feb '19

Greetings, I've set the dates for the 2.7.16 release in PEP 373. The release candidate will happen on February 16 with a final release 2 weeks later on March 2 if all goes well. Servus, Benjamin

1 0

Reviewing PEP 580
by Jeroen Demeyer 12 Feb '19

12 Feb '19

Hello, I would like to propose to the new steering council to review PEP 580. Is there already a process for that? Thanks, Jeroen.

3 2

[docs] [issue35155] Requesting a review
by Denton Liu 12 Feb '19

12 Feb '19

Hello all, A couple months back, I reported bpo-35155[1] and I submitted a PR for it[2]. After a couple of reviews, it seems like progress has stalled. Would it be possible for someone to review this? Thanks, Denton [1]: https://bugs.python.org/issue35155 [2]: https://github.com/python/cpython/pull/10313

1 0

find_library and issue21622
by Javier Castillo II 11 Feb '19

11 Feb '19

Ran into some issues trying to deploy in an alpine container, where I wound up coming across the issue. I found a solution ( not sure if an ideal solution can exist ) that walks the paths in the environment variable LD_LIBRARY_PATH. This was submitted in github PR 10460, but not sure if there were any technical issues with this impacting its review.

2 2

Python 3.8 alpha and AIX buildbot “support” moving forward.
by Michael Felt (aixtools) 09 Feb '19

09 Feb '19

Congratulations on the official begin of the alpha phase of Python3-3.8. I hope there will be time to consider three of my PRs so that this phase has at least one of the AIX buildbots (not mine I fear) is passing all the tests and can finally serve it’s real purpose and signal when a change toggles it’s status from PASS to FAIL. Next week I hope to have some time to dig deeper and try to establish why my bot fails additional tests (in the multiprocessing module(s)) as well why AIX fails test_bdb when utf8 support is (additionally) installed but passes when utf8 support is not installed. While I am also concerned about AIX status I also hope that my inspection is helping to improve Python. Sincerely, Michael Sent from my iPhone

1 0

About the future of multi-process Python
by Antoine Pitrou 08 Feb '19

08 Feb '19

Hello, For the record there are number of initiatives currently to boost the usefulness and efficiency of multi-process computation in Python. One of them is PEP 574 (zero-copy pickling with out-of-band buffers), which I'm working on. Another is Pierre Glaser's work on allowing pickling of dynamic functions and classes with the C-accelerated _pickle module (rather than the slow pure Python implementation): https://bugs.python.org/issue35900 https://bugs.python.org/issue35911 Another is Davin's work on shared memory managers. There are also emerging standards like Apache Arrow that provide a shared, runtime-agnostic, compute-friendly representation for in-memory tabular data, and third-party frameworks like Dask which are potentially able to work on top of that and expose nice end-user APIs. For maximum synergy between these initiatives and the resulting APIs, it is better if things are done in the open ;-) Regards Antoine.

3 3

[RELEASE] Python 3.8.0a1 is now available for testing
by Łukasz Langa 08 Feb '19

08 Feb '19

I packaged my first release. *wipes sweat off of face* Go get it here: https://www.python.org/downloads/release/python-380a1/ Python 3.8.0a1 is the first of four planned alpha releases of Python 3.8, the next feature release of Python. During the alpha phase, Python 3.8 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The next preview release, 3.8.0a2, is planned for 2019-02-24. Apart from building the Mac installers, Ned helped me a lot with the process, thank you! Ernest was super quick providing me with all required access and fixing a Unicode problem I found in Salt, thank you! Finally, this release was made on a train to Düsseldorf. There's a PyPy sprint there. The train is pretty cool, makes this "Wasm! Wasm!" sound. - Ł

5 14

CPython on Windows ARM32
by Paul Monson 08 Feb '19

08 Feb '19

Hi Python Developers, I'm Paul Monson, I've spent about 20 years working with embedded software. Since 2010 I've worked for Microsoft as a developer. Our team is working with CPython on Azure IoT Edge devices that run on x64-based devices. We would like to extend that support to Windows running on ARM 32-bit devices and have a working proof-of-concept. Our team is prepared to provide support for CPython for Windows on ARM32 for 10 years, and to provide build bots for ARM32. I like to propose that the initial sequence of PRs could be: - Update to OpenSSL 1.1.1 (without anything ARM specific) - ready to go - Migrate to libffi directly (finish https://github.com/python/cpython/pull/3806<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co…>) - Build file updates for OpenSSL ARM and check into cpython-bin-deps - Build file updates for CPython ARM - ctypes updates for ARM - Test module skips for ARM - Library updates and related test fixes for ARM Updating OpenSSL and libffi are independent of ARM support but need to be done as prerequisites. OpenSSL 1.1.0 doesn't have support for ARM32 on Windows but OpenSSL 1.1.1 does. I have OpenSSL 1.1.1a ready to check in to master with all tests passing on x86 and x64 on Windows. Since work has already been done on this for other platforms only very small changes were needed for Windows. I have also integrated and tested the current libffi on Windows x64. Some additonal porting of x86 assembler to MSVC tools will need to be done. I have a working port of ARM32 assembler for MSVC but it may need to be brought up to date and cleaned up. The last four all need to go in together, but can be reviewed separately. We are not planning to support Tk/Tcl on ARM32 because Windows IoT Core, Windows containers don't support GDI, which is a depenency of Tk/Tcl. Since Window IoT Core and Windows container don't support the .msi or .exe installers found on python.org my team at Microsoft will build the CPython for Windows ARM32 from the official repo and distribute it. Thanks in advance, Paul

6 10

bpo-32972: Add unittest.AsyncioTestCase review (for 3.8?)
by David Shawley 05 Feb '19

05 Feb '19

Hi everyone, I added a PR to add a sub-class of unittest.TestCase that makes it possible to write async test methods. I wrote this a few months ago and it is waiting on core review. Is there a core dev that can take up this review? I would love to have this functionality in the core. Lukasz - should we add this to Python 3.8 or is it too late for feature additions? BPO link: https://bugs.python.org/issue32972 <https://bugs.python.org/issue32972> Github PR: https://github.com/python/cpython/pull/10296 <https://github.com/python/cpython/pull/10296> cheers, dave. -- "State and behavior. State and behavior. If it doesn’t bundle state and behavior in a sensible way, it should not be an object, and there should not be a class that produces it." eevee

3 2

Return type of datetime subclasses added to timedelta
by Paul Ganssle 04 Feb '19

04 Feb '19

Happy New Year everyone! I would like to start a thread here for wider feedback on my proposal to change the return type of the addition operation between a datetime subclass and a timedelta. Currently, adding a timedelta to a subclass of datetime /always/ returns a datetime rather than an instance of the datetime subclass. I have an open PR implementing this, PR #10902 <https://github.com/python/cpython/pull/10902>, but I know it's a major change so I did not want to move forward without more discussion. I first brought this up on datetime-SIG <https://mail.python.org/archives/list/datetime-sig@python.org/thread/TGB3VZ…> [1], and we decided to move the discussion over here because the people most likely to object to the change would be on this list and not on datetime-SIG. In addition to the datetime-SIG thread, you may find a detailed rationale for the change in bpo-35364 <https://bugs.python.org/issue35364#msg331065> [2], and a rationale for why we would want to (and arguably already /do/) support subclassing datetime in bpo-32417 <https://bugs.python.org/issue32417#msg331353> [3]. A short version of the strongest rationale for changing how this works is that it is causing inconsistencies in how subclassing is handled in alternate constructors of datetime. For a given subclass of datetime (which I will call DateTimeSub), nearly all alternate constructors already support subclasses correctly - DateTimeSub.fromtimestamp(x) will return a DateTimeSub, for example. However, because DateTimeSub + timedelta returns datetime, any alternate constructor implemented in terms of timedelta additions will leak that implementation detail by returning a datetime object instead of the subclass. The biggest problem is that datetime.fromutc is defined in terms of timedelta addition, so DateTimeSub.now() returns a DateTimeSub object, but DateTimeSub.now(timezone.utc) returns a datetime object! This is one of the most annoying things to work around when building a datetime subclass, and I don't know of any situation where someone /wants/ their subclass to be lost on addition with a timedelta. From my understanding, this has been discussed before and the original objection was that this implementation assumes that the datetime subclass has a constructor with the same (or a sufficiently similar) signature as datetime. This may be a legitimate gripe, but unfortunately that ship has sailed long ago. All of datetime's alternate constructors make this assumption. Any subclass that does not meet this requirement must have worked around it long ago (or they don't care about alternate constructors). Thanks for your attention, I look forward to your replies. Best, Paul [1] https://mail.python.org/archives/list/datetime-sig@python.org/thread/TGB3VZ… [2] https://bugs.python.org/issue35364#msg331065 [3] https://bugs.python.org/issue32417#msg331353

5 14

About multiprocessing maintainership
by Antoine Pitrou 04 Feb '19

04 Feb '19

Hello, In a recent message, Raymond dramatically pretends that I would have "edited out" Davin of the maintainers list for the multiprocessing module. What I did (*) is different: I asked to mark Davin inactive and to stop auto-assigning him on bug tracker issues. Davin was /still/ listed in the experts list, along with me and others. IOW, there was no "editing out". (*) https://github.com/python/devguide/pull/435 The reason I did this is simple: Davin does not do, and has almost never done, any actual maintenance work on multiprocessing (if you are not convinced, just go through the git history, and the PRs that were merged in the ~4 last years). He usually does not respond to tracker issues opened by users. He does not review PRs. The only sizable piece of work he committed is, as I mentioned in the previous thread, still untested and undocumented. Auto-assigning someone who never (AFAICT) responds to issues ultimately does a disservice to users, whose complaints go unanswered; while other people, who /do/ respond to users, are not aware of those stale issues. Regards Antoine.

4 4

Why a merge for 3.8.0a1?
by Stephane Wirtel 04 Feb '19

04 Feb '19

Hi all, After a git pull, I have seen there is a merge for v3.8.0a1 by Łukasz Langa, why? I think the code should keep a linear commit and in this case, it's against the "commit&squash" of CPython and Github :/ Thank you for your response. Stéphane -- Stéphane Wirtel - https://wirtel.be - @matrixise

2 2

Difference between Include/internal and Include/cpython ?
by Antoine Pitrou 03 Feb '19

03 Feb '19

Hello, Can someone explain why we have two separate directories Include/internal and Include/cpython? What is the rule for declaring an API inside one or another? At first sight, it seems to me we're having gratuitous complication here. For example, I notice that PyFloat_Fini() is declared in Include/cpython/pylifecycle.h but PyLong_Fini() is declared in Include/internal/pycore_pylifecycle.h? (and why the additional "pycore_XXX.h" naming convention for some of those files?) Regards Antoine.

3 4

Summary of Python tracker Issues
by Python tracker 01 Feb '19

01 Feb '19

ACTIVITY SUMMARY (2019-01-25 - 2019-02-01) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6985 (+32) closed 40649 (+23) total 47634 (+55) Open issues with patches: 2775 Issues opened (45) ================== #25592: distutils docs: data_files always uses sys.prefix https://bugs.python.org/issue25592 reopened by pitrou #30670: pprint for dict in sorted order or insert order? https://bugs.python.org/issue30670 reopened by josephsmeng #35829: datetime: parse "Z" timezone suffix in fromisoformat() https://bugs.python.org/issue35829 opened by rdb #35830: building multiple (binary) packages from a single project https://bugs.python.org/issue35830 opened by stefan #35832: Installation error https://bugs.python.org/issue35832 opened by Stefano Bonalumi #35833: IDLE: revise doc for control chars sent to Shell https://bugs.python.org/issue35833 opened by Dude Roast #35834: get_type_hints exposes an instance of ForwardRef (internal cla https://bugs.python.org/issue35834 opened by Lincoln Quirk #35838: ConfigParser calls optionxform twice when assigning dict https://bugs.python.org/issue35838 opened by Phil Kang #35839: Suggestion: Ignore sys.modules entries with no __spec__ attrib https://bugs.python.org/issue35839 opened by ncoghlan #35840: Control flow inconsistency on closed asyncio stream https://bugs.python.org/issue35840 opened by schlamar #35841: Datetime strftime() does not return correct week numbers for 2 https://bugs.python.org/issue35841 opened by tr12 #35843: importlib.util docs for namespace packages innaccurate https://bugs.python.org/issue35843 opened by Anthony Sottile #35844: Calling `Multiprocessing.Queue.close()` too quickly causes int https://bugs.python.org/issue35844 opened by charmonium #35845: Can't read a F-contiguous memoryview in physical order https://bugs.python.org/issue35845 opened by pitrou #35846: Incomplete documentation for re.sub https://bugs.python.org/issue35846 opened by pbugnion #35847: RISC-V needs CTYPES_PASS_BY_REF_HACK https://bugs.python.org/issue35847 opened by schwab #35848: readinto is not a method on io.TextIOBase https://bugs.python.org/issue35848 opened by steverpalmer #35849: Added thousands separators to Lib/pstats.py final report https://bugs.python.org/issue35849 opened by addons_zz #35851: Make search result in online docs keep their position when sea https://bugs.python.org/issue35851 opened by roelschroeven #35852: Fixed tests regenerating using CRLF when running it on Windows https://bugs.python.org/issue35852 opened by addons_zz #35854: EnvBuilder and venv symlinks do not work on Windows on 3.7.2 https://bugs.python.org/issue35854 opened by steve.dower #35855: IDLE squeezer: improve unsqueezing and autosqueeze default https://bugs.python.org/issue35855 opened by terry.reedy #35856: bundled pip syntaxwarning https://bugs.python.org/issue35856 opened by Dima.Tisnek #35857: Stacktrace shows lines from updated file on disk, not code act https://bugs.python.org/issue35857 opened by Steve Pryde #35859: Capture behavior depends on the order of an alternation https://bugs.python.org/issue35859 opened by davisjam #35860: ProcessPoolExecutor subprocesses crash & break pool when raisi https://bugs.python.org/issue35860 opened by underyx #35861: test_named_expressions raises SyntaxWarning https://bugs.python.org/issue35861 opened by xtreak #35862: Change the environment for a new process https://bugs.python.org/issue35862 opened by r-or #35866: concurrent.futures deadlock https://bugs.python.org/issue35866 opened by jwilk #35867: NameError is not caught at Task execution https://bugs.python.org/issue35867 opened by Sampsa Riikonen #35868: Support ALL_PROXY environment variable in urllib https://bugs.python.org/issue35868 opened by Oleh Khoma #35869: io.BufferReader.read() returns None https://bugs.python.org/issue35869 opened by steverpalmer #35870: readline() specification is unclear https://bugs.python.org/issue35870 opened by porton #35871: Pdb NameError in generator param and list comprehension https://bugs.python.org/issue35871 opened by jayanth #35872: Creating venv from venv no longer works in 3.7.2 https://bugs.python.org/issue35872 opened by schlamar #35873: Controlling venv from venv no longer works in 3.7.2 https://bugs.python.org/issue35873 opened by schlamar #35874: Clarify that the (...) convertor to PyArg_ParseTuple... accept https://bugs.python.org/issue35874 opened by Antony.Lee #35875: Crash - algos.cp36-win_amd64.pyd join.cp36-win_amd64.pyd https://bugs.python.org/issue35875 opened by AxelArnoldBangert #35876: test_start_new_session for posix_spawnp fails https://bugs.python.org/issue35876 opened by pablogsal #35877: parenthesis is mandatory for named expressions in while statem https://bugs.python.org/issue35877 opened by xtreak #35878: ast.c: end_col_offset may be used uninitialized in this functi https://bugs.python.org/issue35878 opened by vstinner #35879: test_type_comments leaks references https://bugs.python.org/issue35879 opened by vstinner #35880: math.sin has no backward error; this isn't documented https://bugs.python.org/issue35880 opened by jneb #35882: distutils fails with UnicodeEncodeError with strange filename https://bugs.python.org/issue35882 opened by scjody #35883: Change invalid unicode characters to replacement characters in https://bugs.python.org/issue35883 opened by Neui Most recent 15 issues with no replies (15) ========================================== #35882: distutils fails with UnicodeEncodeError with strange filename https://bugs.python.org/issue35882 #35878: ast.c: end_col_offset may be used uninitialized in this functi https://bugs.python.org/issue35878 #35874: Clarify that the (...) convertor to PyArg_ParseTuple... accept https://bugs.python.org/issue35874 #35873: Controlling venv from venv no longer works in 3.7.2 https://bugs.python.org/issue35873 #35867: NameError is not caught at Task execution https://bugs.python.org/issue35867 #35860: ProcessPoolExecutor subprocesses crash & break pool when raisi https://bugs.python.org/issue35860 #35852: Fixed tests regenerating using CRLF when running it on Windows https://bugs.python.org/issue35852 #35844: Calling `Multiprocessing.Queue.close()` too quickly causes int https://bugs.python.org/issue35844 #35827: C API dictionary views type checkers are not documented https://bugs.python.org/issue35827 #35813: shared memory construct to avoid need for serialization betwee https://bugs.python.org/issue35813 #35812: Don't log an exception from the main coroutine in asyncio.run( https://bugs.python.org/issue35812 #35807: Update bundled pip to 19.0 https://bugs.python.org/issue35807 #35803: Test and document that `dir=...` in tempfile may be PathLike https://bugs.python.org/issue35803 #35801: venv in 3.7 references python3 executable https://bugs.python.org/issue35801 #35792: Specifying AbstractEventLoop.run_in_executor as a coroutine co https://bugs.python.org/issue35792 Most recent 15 issues waiting for review (15) ============================================= #35877: parenthesis is mandatory for named expressions in while statem https://bugs.python.org/issue35877 #35876: test_start_new_session for posix_spawnp fails https://bugs.python.org/issue35876 #35862: Change the environment for a new process https://bugs.python.org/issue35862 #35861: test_named_expressions raises SyntaxWarning https://bugs.python.org/issue35861 #35854: EnvBuilder and venv symlinks do not work on Windows on 3.7.2 https://bugs.python.org/issue35854 #35852: Fixed tests regenerating using CRLF when running it on Windows https://bugs.python.org/issue35852 #35849: Added thousands separators to Lib/pstats.py final report https://bugs.python.org/issue35849 #35847: RISC-V needs CTYPES_PASS_BY_REF_HACK https://bugs.python.org/issue35847 #35843: importlib.util docs for namespace packages innaccurate https://bugs.python.org/issue35843 #35826: Typo in example for async with statement with condition https://bugs.python.org/issue35826 #35824: http.cookies._CookiePattern modifying regular expressions https://bugs.python.org/issue35824 #35823: Use vfork() in subprocess on Linux https://bugs.python.org/issue35823 #35813: shared memory construct to avoid need for serialization betwee https://bugs.python.org/issue35813 #35810: Object Initialization Bug with Heap-allocated Types https://bugs.python.org/issue35810 #35803: Test and document that `dir=...` in tempfile may be PathLike https://bugs.python.org/issue35803 Top 10 most discussed issues (10) ================================= #35431: Add a function for computing binomial coefficients to the math https://bugs.python.org/issue35431 12 msgs #25592: distutils docs: data_files always uses sys.prefix https://bugs.python.org/issue25592 10 msgs #35857: Stacktrace shows lines from updated file on disk, not code act https://bugs.python.org/issue35857 10 msgs #35854: EnvBuilder and venv symlinks do not work on Windows on 3.7.2 https://bugs.python.org/issue35854 7 msgs #35848: readinto is not a method on io.TextIOBase https://bugs.python.org/issue35848 6 msgs #35859: Capture behavior depends on the order of an alternation https://bugs.python.org/issue35859 6 msgs #35823: Use vfork() in subprocess on Linux https://bugs.python.org/issue35823 5 msgs #35829: datetime: parse "Z" timezone suffix in fromisoformat() https://bugs.python.org/issue35829 5 msgs #30670: pprint for dict in sorted order or insert order? https://bugs.python.org/issue30670 4 msgs #32834: test_gdb fails with Posix locale in 3.7 https://bugs.python.org/issue32834 4 msgs Issues closed (22) ================== #2212: Cookie.BaseCookie has ambiguous unicode handling https://bugs.python.org/issue2212 closed by martin.panter #29235: Allow profile/cProfile to be used as context managers https://bugs.python.org/issue29235 closed by cheryl.sabella #34003: csv.DictReader can return basic dict instead of OrderedDict https://bugs.python.org/issue34003 closed by rhettinger #35196: IDLE text squeezer is too aggressive and is slow https://bugs.python.org/issue35196 closed by terry.reedy #35717: enum.Enum error on sys._getframe(2) https://bugs.python.org/issue35717 closed by vstinner #35769: IDLE: change new file name from ''Untitled" to "untitled" https://bugs.python.org/issue35769 closed by cheryl.sabella #35780: Recheck logic in the C version of the lru_cache() https://bugs.python.org/issue35780 closed by rhettinger #35797: concurrent.futures.ProcessPoolExecutor does not work in venv o https://bugs.python.org/issue35797 closed by steve.dower #35811: py.exe should unset the __PYVENV_LAUNCHER__ environment variab https://bugs.python.org/issue35811 closed by steve.dower #35825: Py_UNICODE_SIZE=4 fails to link on Windows https://bugs.python.org/issue35825 closed by inada.naoki #35831: Format Spec example says limited to 3.1+ but works in 2.7 https://bugs.python.org/issue35831 closed by fdrake #35835: There is no mention of breakpoint() in the pdb documentation https://bugs.python.org/issue35835 closed by Mariatta #35836: ZeroDivisionError class should have a __name__ attr https://bugs.python.org/issue35836 closed by steven.daprano #35837: smtpd PureProxy breaks on mail_options keyword argument https://bugs.python.org/issue35837 closed by r.david.murray #35842: A potential bug about use of uninitialised variable https://bugs.python.org/issue35842 closed by josh.r #35850: CKAN installation went on script error https://bugs.python.org/issue35850 closed by christian.heimes #35853: Extend the functools module with more higher order function co https://bugs.python.org/issue35853 closed by rhettinger #35858: Consider adding the option of running shell/console commands i https://bugs.python.org/issue35858 closed by jcrmatos #35863: email.headers wraps headers badly https://bugs.python.org/issue35863 closed by r.david.murray #35864: Replace OrderedDict with regular dict in namedtuple's _asdict( https://bugs.python.org/issue35864 closed by rhettinger #35865: configparser document refers about random dict order https://bugs.python.org/issue35865 closed by inada.naoki #35881: test_type_comments leaks references and memory blocks https://bugs.python.org/issue35881 closed by vstinner

1 0

Re: [Python-Dev] How to update namedtuple asdict() to use dict instead of OrderedDict
by eamanu15 01 Feb '19

01 Feb '19

Hi! > Option 4) Just make the change directly in 3.8, s/OrderedDict/dict/, and > be done will it. This gives users the benefits right away and doesn't > annoy them with warnings that they likely don't care about. There is some > precedent for this. To make namedtuple class creation faster, the > *verbose* option was dropped without any deprecation period. It looks like > no one missed that feature at all, but they did get the immediate benefit > of faster import times. In the case of using regular dicts in named > tuples, people will get immediate and significant space savings as well as > a speed benefit. > +1 for option 4 Regards! eamanu

1 0