Mailman 3 April 2019 - Python-Dev

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

Boundaries between numbers and identifiers
by Serhiy Storchaka 15 Apr '21

15 Apr '21

In Python 2.5 `0or[]` was accepted by the Python parser. It became an error in 2.6 because "0o" became recognizing as an incomplete octal number. `1or[]` still is accepted. On other hand, `1if 2else 3` is accepted despites the fact that "2e" can be recognized as an incomplete floating point number. In this case the tokenizer pushes "e" back and returns "2". Shouldn't it do the same with "0o"? It is possible to make `0or[]` be parseable again. Python implementation is able to tokenize this example: $ echo '0or[]' | ./python -m tokenize 1,0-1,1: NUMBER '0' 1,1-1,3: NAME 'or' 1,3-1,4: OP '[' 1,4-1,5: OP ']' 1,5-1,6: NEWLINE '\n' 2,0-2,0: ENDMARKER '' On other hand, all these examples look weird. There is an assymmetry: `1or 2` is a valid syntax, but `1 or2` is not. It is hard to recognize visually the boundary between a number and the following identifier or keyword, especially if numbers can contain letters ("b", "e", "j", "o", "x") and underscores, and identifiers can contain digits. On both sides of the boundary can be letters, digits, and underscores. I propose to change the Python syntax by adding a requirement that there should be a whitespace or delimiter between a numeric literal and the following keyword.

8 10

About "python-porting" mail list
by Facundo Batista 20 Jan '20

20 Jan '20

Hello! This list (which I co-admin, with Georg) is getting less and less traffic as months pass by. See: https://mail.python.org/pipermail/python-porting/ The interwebs has been collecting ton of resources about porting py2 to 3 during these years. Any not-yet-answered question surely can be done in a list with more participants. Can we kill this list? Thanks! Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org.ar/ Twitter: @facundobatista

4 7

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton 15 Sep '19

15 Sep '19

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

why is not 64-bit installer the default download link for Windows?
by Cosimo Lupo 19 Jun '19

19 Jun '19

If one goes to httWhps://www.python.org/downloads <https://www.python.org/downloads> from a Windows browser, the default download URL is for the 32-bit installer instead of the 64-bit one. I wonder why is this still the case? Shouldn't we encourage new Windows users (who may not even know the distinction between the two architectures) to use the 64-bit version of Python, since most likely they can? If this is not the correct forum for this, please let me know where I can direct my question/feature request, thanks. Cosimo -- Cosimo Lupo

14 22

BDFL-Delegate appointments for several PEPs
by Nick Coghlan 22 May '19

22 May '19

Hi folks, With the revised PEP 1 published, the Steering Council members have been working through the backlog of open PEPs, figuring out which ones are at a stage of maturity where we think it makes sense to appoint a BDFL-Delegate to continue moving the PEP through the review process, and eventually make the final decision on whether or not to accept or reject the change. We'll be announcing those appointments as we go, so I'm happy to report that I will be handling the BDFL-Delegate responsibilities for the following PEPs: * PEP 499: Binding "-m" executed modules under their module name as well as `__main__` * PEP 574: Pickle protocol 5 with out of band data I'm also pleased to report that Petr Viktorin has agreed to take on the responsibility of reviewing the competing proposals to improve the way CPython's C API exposes callables for direct invocation by third party low level code: * PEP 576: Exposing the internal FastCallKeywords convention to 3rd party modules * PEP 580: Revising the callable struct hierarchy internally and in the public C API * PEP 579: Background information for the problems the other two PEPs are attempting to address Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

9 61

PEP 574 ready for review
by Antoine Pitrou 07 May '19

07 May '19

Hello, I've put the final touches to PEP 574 - Pickle protocol 5 with out-of-band data (*). It is now ready for review. The implementation is fully functional, as well as its PyPI backport (**), and has regression tests against Numpy. Numpy and PyArrow have their own tests against the pickle5 backport. (*) https://www.python.org/dev/peps/pep-0574/ (**) https://pypi.org/project/pickle5/ Regards Antoine.

4 5

"if __name__ == '__main__'" at the bottom of python unittest files
by Chris Withers 04 May '19

04 May '19

Hi All, I have a crazy idea of getting unittest.mock up to 100% code coverage. I noticed at the bottom of all of the test files in testmock/, there's a: if __name__ == '__main__': unittest.main() ...block. How would people feel about these going away? I don't *think* they're needed now that we have unittest discover, but thought I'd ask. Chris

10 25

drop jython support in mock backport?
by Chris Withers 30 Apr '19

30 Apr '19

[resending to python-dev in case there are Jython users here...] Hi All, If you need Jython support in the mock backport, please shout now: https://github.com/testing-cabal/mock/issues/453 cheers, Chris

2 1

Use C extensions compiled in release mode on a Python compiled in debug mode
by Victor Stinner 29 Apr '19

29 Apr '19

Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on a Python compiled in debug mode ("pydebug"). The use case is to debug bugs in C extensions thanks to additional runtime checks of a Python debug build, and more generally get a better debugging experiences on Python. Even for pure Python, a debug build is useful (to get the Pyhon traceback in gdb using "py-bt" command). Currently, using a Python compiled in debug mode means to have to recompile C extensions in debug mode. Compile a C extension requires a C compiler, header files, pull dependencies, etc. It can be very complicated in practical (and pollute your system with all these additional dependencies). On Linux, it's already hard, but on Windows it can be even harder. Just one concrete example: no debug build of numpy is provided at https://pypi.org/project/numpy/ Good luck to build numpy in debug mode manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.) :-) -- The first requirement for the use case is that a Python debug build supports the ABI of a release build. The current blocker issue is that the Py_DEBUG define imply the Py_TRACE_REFS define: PyObject gets 2 extra fields (_ob_prev and _ob_next) which change the offset of all attributes of all objects and makes the ABI completely incompatible. I propose to no longer imply Py_TRACE_REFS *by default* (but keep the code): https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 (Py_TRACE_REFS would be a different ABI.) The second issue is that library filenames are different for a debug build: SOABI gets an additional "d" flag for Py_DEBUG. A debug build should first look for "NAME.cpython-38dm.so" (flags: "dm"), but then also look for "NAME.cpython-38m.so" (flags: "m"). The opposite is not possible: a debug build contains many additional functions missing from a release build. For Windows, maybe we should provide a Python compiled in debug mode with the same C Runtime than a Python compiled in release mode. Otherwise, the debug C Runtime is causing another ABI issue. Maybe pip could be enhanced to support installing C extensions compiled in release mode when using a debug mode. But that's more for convenience, it's not really required, since it is easy to switch the Python runtime between release and debug build. Apart of Py_TRACE_REFS, I'm not aware of other ABI differences in structures. I know that the COUNT_ALLOCS define changes the ABI, but it's not implied by Py_DEBUG: you have to opt-in for COUNT_ALLOCS. (I propose to do the same for Py_TRACE_REFS ;-)) Note: Refleaks buildbots don't use Py_TRACE_REFS to track memory leaks, only sys.gettotalrefcount(). -- Python debug build has many benefit. If you ignore C extensions, the debug build is usually compiled with compiler optimization disabled which makes debugging in gdb a much better experience. If you never tried: on a release build, most (if not all) variables are "<optimized out>" and it's really painful to basic debug functions like displaying the current Python frame. Assertions are removed in release modes, whereas they can detect a wide range of bugs way earlier: integer overflow, buffer under- and overflow, exceptions ignored silently, etc. Nobody likes to see a bug for the first time in production. For example, I modified Python 3.8 to now logs I/O errors when a file is closed implicitly, but only in debug or development mode. In release Python silently ignored EBADF error on such case, whereas it can lead to very nasty bugs causing Python to call abort() (which creates a coredump on Linux): see https://bugs.python.org/issue18748 ... DeprecationWarning and ResourceWarning are shown by default in debug mode :-) There are too many different additional checks done at runtime: I cannot list them all here. -- Being able to switch between Python in release mode and Python in debug mode is a first step. My long term plan would be to better separate "Python" from its "runtime". CPython in release mode would be one runtime, CPython in debug mode would be another runtime, PyPy can seeen as another runtime, etc. The more general idea is: "compile your C extension once and use any Python runtime". https://pythoncapi.readthedocs.io/runtimes.html#runtimes If you opt-in for the stable ABI, you can already switch between runtimes of different Python versions (ex: Python 3.6 or Python 3.8). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

9 28

datetime.fromisocalendar
by Paul Ganssle 29 Apr '19

29 Apr '19

Greetings, Some time ago, I proposed adding a `.fromisocalendar` alternate constructor to `datetime` (bpo-36004 <https://bugs.python.org/issue36004>), with a corresponding implementation (PR #11888 <https://github.com/python/cpython/pull/11888>). I advertised it on datetime-SIG some time ago but haven't seen much discussion there, so I'd like to bring it to python-dev's attention as we near the cut-off for new Python 3.8 features. Other than the fact that I've needed this functionality in the past, I also think a good general principle for the datetime module is that when a class (time, date, datetime) has a "serialization" method (.strftime, .timestamp, .isoformat, .isocalendar, etc), there should be a corresponding /deserialization/ method (.strptime, .fromtimestamp, .fromisoformat) that constructs a datetime from the output. Now that `fromisoformat` was introduced in Python 3.7, I think `isocalendar` is the only remaining method without an inverse. Do people agree with this principle? Should we add the `fromisocalendar` method? Thanks, Paul

4 3

git history conundrum
by Chris Withers 28 Apr '19

28 Apr '19

Hi All, I'm in the process of bringing the mock backport up to date, but this has got me stumped: $ git log --oneline --no-merges 5943ea76d529f9ea18c73a61e10c6f53bdcc864f.. -- Lib/unittest/mock.py Lib/unittest/test/testmock/ | tail 362f058a89 Issue #28735: Fixed the comparison of mock.MagickMock with mock.ANY. d9c956fb23 Issue #20804: The unittest.mock.sentinel attributes now preserve their identity when they are copied or pickled. 84b6fb0eea Fix unittest.mock._Call: don't ignore name 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now properly support assert_called, assert_not_called, and assert_called_once. 0be894b2f6 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). d4583d7fea Issue #26750: use inspect.isdatadescriptor instead of our own _is_data_descriptor(). 9854789efe Issue #26750: unittest.mock.create_autospec() now works properly for subclasses of property() and other data descriptors. 204bf0b9ae English spelling and grammar fixes Right, so I've merged up to 15f44ab043, what comes next? $ git log --oneline --no-merges 15f44ab043.. -- Lib/unittest/mock.py Lib/unittest/test/testmock/ | tail -n 3 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now properly support assert_called, assert_not_called, and assert_called_once. 0be894b2f6 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). Okay, no idea why 0be894b2f6 is there, appears to be a totally identical commit to 15f44ab043, so let's skip it: $ git log --oneline --no-merges 0be894b2f6.. -- Lib/unittest/mock.py Lib/unittest/test/testmock/ | tail -n 3 161a4dd495 Issue #28919: Simplify _copy_func_details() in unittest.mock ac5084b6c7 Fixes issue28380: unittest.mock Mock autospec functions now properly support assert_called, assert_not_called, and assert_called_once. 15f44ab043 Issue #27895: Spelling fixes (Contributed by Ville Skyttä). Wat?! Why is 15f44ab043 showing up again?! What's the git subtlety I'm missing here? Chris

5 8

PEP 580 and PEP 590 comparison.
by Mark Shannon 27 Apr '19

27 Apr '19

Hi Petr, Thanks for spending time on this. I think the comparison of the two PEPs falls into two broad categories, performance and capability. I'll address capability first. Let's try a thought experiment. Consider PEP 580. It uses the old `tp_print` slot as an offset to mark the location of the CCall structure within the callable. Now suppose instead that it uses a `tp_flag` to mark the presence of an offset field and that the offset field is moved to the end of the TypeObject. This would not impact the capabilities of PEP 580. Now add a single line nargs ~= PY_VECTORCALL_ARGUMENTS_OFFSET here https://github.com/python/cpython/compare/master...jdemeyer:pep580#diff-116… which would make PyCCall_FastCall compatible with the PEP 590 vectorcall protocol. Now rebase the PEP 580 reference code on top of PEP 590 minimal implementation and make the vectorcall field of CFunction point to PyCCall_FastCall. The resulting hybrid is both a PEP 590 conformant implementation, and is at least as capable as the reference PEP 580 implementation. Therefore PEP 590, must be at least as capable at PEP 580. Now performance. Currently the PEP 590 implementation is intentionally minimal. It does nothing for performance. The benchmark Jeroen provides is a micro-benchmark that calls the same functions repeatedly. This is trivial and unrealistic. So, there is no real evidence either way. I will try to provide some. The point of PEP 590 is that it allows performance improvements by allowing callables more freedom of implementation. To repeat an example from an earlier email, which may have been overlooked, this code reduces the time to create ranges and small lists by about 30% https://github.com/markshannon/cpython/compare/vectorcall-minimal...marksha… https://gist.github.com/markshannon/5cef3a74369391f6ef937d52cca9bfc8 To speed up calls to builtin functions by a measurable amount will need some work on argument clinic. I plan to have that done before PyCon in May. Cheers, Mark.

2 4

Actor Model in Python
by Aratz Manterola Lasa 27 Apr '19

27 Apr '19

Hello, I was wondering if there was any Python project aiming to implement the actor model for python concurrency. ¿Does anyone know it? Aratz.

4 4

Increasing the C-optimized pickle extensibility
by Pierre Glaser 26 Apr '19

26 Apr '19

Hi All, We (Antoine Pitrou, Olivier Grisel and myself) spent some efforts recently on enabling pickle extensions to extend the C-optimized Pickler instead of the pure Python one. Pickle extensions have a crucial role in many distributed computing libraries: cloudpickle (https://github.com/cloudpipe/cloudpickle) for example is vendored in dask, pyspark, ray, and joblib. Early benchmarks show that relying on the C-optimized pickle yields significant serialization speed improvements (up to 30x faster). (draft PR of the CPickler-backed version of cloudpickle: https://github.com/cloudpipe/cloudpickle/pull/253) To make extending the C Pickler possible, we are currently moving forward with a few enhancements to the public pickle API. * First, we are enabling Pickler subclasses to implement a reducer_override method, that will be have priority over the registered reducers in the dispatch_table and over the default handling of classes and functions. (PR link: https://github.com/python/cpython/pull/12499) * Then, we are adding a new keyword argument to save_reduce called state_setter. (consequently we allow a reducer's return value to have a new, 6th item). This state setter callable is useful to override programmatically the state updating behavior of an object, that would otherwise be restricted to its static ``__setstate__`` method. (PR link: https://github.com/python/cpython/pull/12588) The PR review process of these changes is in progress, and anyone is welcomed to chime in and share some thoughts. The first addition is very non-invasive. We estimated that the second point did not require introducing a new opcode, as this change could be implemented as simple sequence of standard pickle instructions. We therefore think that it is not necessary to make this change dependent on the new protocol 5 proposed in PEP 574. The key advantage in not creating a new opcode that this makes our change backward-compatible, meaning that 3.8-written pickles will not break because of our change if read using earlier Python versions. OTOH, one might argue that a new OPCODE might * make the code a little bit cleaner * make it easier to interpret disassembled pickle strings. If you are interested, here is an example of a disassembled pickle string using our currently proposed solution: https://github.com/pierreglaser/cpython/pull/2#issuecomment-486243350 Does anyone have an opinion on this? Thanks, Pierre

3 2

Summary of Python tracker Issues
by Python tracker 26 Apr '19

26 Apr '19

ACTIVITY SUMMARY (2019-04-19 - 2019-04-26) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7083 (+25) closed 41402 (+39) total 48485 (+64) Open issues with patches: 2821 Issues opened (46) ================== #11871: test_default_timeout() of test_threading.BarrierTests failure: https://bugs.python.org/issue11871 reopened by vstinner #36672: A compiler warning in winreg.SetValue() https://bugs.python.org/issue36672 opened by ZackerySpytz #36673: Comment/PI parsing support for ElementTree https://bugs.python.org/issue36673 opened by scoder #36674: "unittest.TestCase.debug" should honour "skip" (and other test https://bugs.python.org/issue36674 opened by dmaurer #36675: Doctest directives and comments not visible or missing from co https://bugs.python.org/issue36675 opened by steven.daprano #36676: Make ET.XMLParser target aware of namespace prefixes https://bugs.python.org/issue36676 opened by scoder #36678: duplicate method definitions in Lib/test/test_dataclasses.py https://bugs.python.org/issue36678 opened by xdegaye #36679: duplicate method definition in Lib/test/test_genericclass.py https://bugs.python.org/issue36679 opened by xdegaye #36680: duplicate method definition in Lib/test/test_importlib/test_u https://bugs.python.org/issue36680 opened by xdegaye #36681: duplicate method definition in Lib/test/test_logging.py https://bugs.python.org/issue36681 opened by xdegaye #36683: duplicate method definition in Lib/test/test_utf8_mode.py https://bugs.python.org/issue36683 opened by xdegaye #36684: codecov.io code coverage has not updated since 2019-04-13 https://bugs.python.org/issue36684 opened by gphemsley #36685: C implementation of xml.etree.ElementTree does not make a copy https://bugs.python.org/issue36685 opened by gphemsley #36686: Docs: asyncio.loop.subprocess_exec documentation is confusing, https://bugs.python.org/issue36686 opened by sbstp #36688: _dummy_thread lacks an RLock implementaiton https://bugs.python.org/issue36688 opened by xtreak #36689: docs: os.path.commonpath raises ValueError for different drive https://bugs.python.org/issue36689 opened by lazka #36691: SystemExit & sys.exit : Allow both exit status and message https://bugs.python.org/issue36691 opened by takluyver #36692: Unexpected stderr output from test_sys_settrace https://bugs.python.org/issue36692 opened by ncoghlan #36694: Excessive memory use or memory fragmentation when unpickling m https://bugs.python.org/issue36694 opened by Ellenbogen #36697: inspect.getclosurevars returns wrong globals dict https://bugs.python.org/issue36697 opened by Noitul #36698: Shell restart when error message contains non-BMP characters https://bugs.python.org/issue36698 opened by TheMathsGod #36699: building for riscv multilib (patch attached) https://bugs.python.org/issue36699 opened by Andreas K. Hüttel #36700: base64 has old references that should be updated https://bugs.python.org/issue36700 opened by paulehoffman #36702: test_dtrace failed https://bugs.python.org/issue36702 opened by sayno996 #36703: [Easy][Windows] test_subprocess: test_close_fds_with_stdio() h https://bugs.python.org/issue36703 opened by vstinner #36704: logging.FileHandler currently hardcodes errors='strict' https://bugs.python.org/issue36704 opened by sourcejedi #36706: Python script on startup stucks at import https://bugs.python.org/issue36706 opened by serge g #36709: Asyncio SSL keep-alive connections raise errors after loop clo https://bugs.python.org/issue36709 opened by tomchristie #36710: Pass _PyRuntimeState as an argument rather than using the _PyR https://bugs.python.org/issue36710 opened by vstinner #36711: duplicate method definition in Lib/email/feedparser.py https://bugs.python.org/issue36711 opened by xdegaye #36712: duplicate method definition in Lib/email/test/test_email_renam https://bugs.python.org/issue36712 opened by xdegaye #36713: uplicate method definition in Lib/ctypes/test/test_unicode.py https://bugs.python.org/issue36713 opened by xdegaye #36714: Tweak doctest 'example' regex to allow a leading ellipsis in ' https://bugs.python.org/issue36714 opened by bskinn #36715: Dictionary initialization https://bugs.python.org/issue36715 opened by Aditya Sane #36716: Embedded Python fails to import module files with version_plat https://bugs.python.org/issue36716 opened by ecosatto #36717: Allow retrieval of return value from the target of a threading https://bugs.python.org/issue36717 opened by Joel Croteau2 #36719: regrtest --findleaks should fail if an uncollectable object is https://bugs.python.org/issue36719 opened by vstinner #36721: Add pkg-config python-3.8-embed https://bugs.python.org/issue36721 opened by vstinner #36723: Unittest Discovery for namespace subpackages dot notation fail https://bugs.python.org/issue36723 opened by mrwaffles #36724: Clear _PyRuntime at exit https://bugs.python.org/issue36724 opened by vstinner #36725: Reference leak regression with Python3.8a3 https://bugs.python.org/issue36725 opened by kayhayen #36728: Remove PyEval_ReInitThreads() from the public C API https://bugs.python.org/issue36728 opened by vstinner #36729: Delete unused text variable on tests https://bugs.python.org/issue36729 opened by eamanu #36730: Change outdated references to macOS https://bugs.python.org/issue36730 opened by Sebastian Bassi #36732: test_asyncio: test_huge_content_recvinto() fails randomly https://bugs.python.org/issue36732 opened by vstinner #36734: Modules/faulthandler.c does not compile on HP-UX due to bpo-35 https://bugs.python.org/issue36734 opened by michael-o Most recent 15 issues with no replies (15) ========================================== #36732: test_asyncio: test_huge_content_recvinto() fails randomly https://bugs.python.org/issue36732 #36730: Change outdated references to macOS https://bugs.python.org/issue36730 #36729: Delete unused text variable on tests https://bugs.python.org/issue36729 #36728: Remove PyEval_ReInitThreads() from the public C API https://bugs.python.org/issue36728 #36723: Unittest Discovery for namespace subpackages dot notation fail https://bugs.python.org/issue36723 #36721: Add pkg-config python-3.8-embed https://bugs.python.org/issue36721 #36717: Allow retrieval of return value from the target of a threading https://bugs.python.org/issue36717 #36713: uplicate method definition in Lib/ctypes/test/test_unicode.py https://bugs.python.org/issue36713 #36712: duplicate method definition in Lib/email/test/test_email_renam https://bugs.python.org/issue36712 #36703: [Easy][Windows] test_subprocess: test_close_fds_with_stdio() h https://bugs.python.org/issue36703 #36702: test_dtrace failed https://bugs.python.org/issue36702 #36699: building for riscv multilib (patch attached) https://bugs.python.org/issue36699 #36692: Unexpected stderr output from test_sys_settrace https://bugs.python.org/issue36692 #36676: Make ET.XMLParser target aware of namespace prefixes https://bugs.python.org/issue36676 #36675: Doctest directives and comments not visible or missing from co https://bugs.python.org/issue36675 Most recent 15 issues waiting for review (15) ============================================= #36734: Modules/faulthandler.c does not compile on HP-UX due to bpo-35 https://bugs.python.org/issue36734 #36729: Delete unused text variable on tests https://bugs.python.org/issue36729 #36725: Reference leak regression with Python3.8a3 https://bugs.python.org/issue36725 #36724: Clear _PyRuntime at exit https://bugs.python.org/issue36724 #36719: regrtest --findleaks should fail if an uncollectable object is https://bugs.python.org/issue36719 #36715: Dictionary initialization https://bugs.python.org/issue36715 #36710: Pass _PyRuntimeState as an argument rather than using the _PyR https://bugs.python.org/issue36710 #36688: _dummy_thread lacks an RLock implementaiton https://bugs.python.org/issue36688 #36685: C implementation of xml.etree.ElementTree does not make a copy https://bugs.python.org/issue36685 #36683: duplicate method definition in Lib/test/test_utf8_mode.py https://bugs.python.org/issue36683 #36681: duplicate method definition in Lib/test/test_logging.py https://bugs.python.org/issue36681 #36680: duplicate method definition in Lib/test/test_importlib/test_u https://bugs.python.org/issue36680 #36679: duplicate method definition in Lib/test/test_genericclass.py https://bugs.python.org/issue36679 #36678: duplicate method definitions in Lib/test/test_dataclasses.py https://bugs.python.org/issue36678 #36676: Make ET.XMLParser target aware of namespace prefixes https://bugs.python.org/issue36676 Top 10 most discussed issues (10) ================================= #36710: Pass _PyRuntimeState as an argument rather than using the _PyR https://bugs.python.org/issue36710 12 msgs #36725: Reference leak regression with Python3.8a3 https://bugs.python.org/issue36725 8 msgs #35224: PEP 572: Assignment Expressions https://bugs.python.org/issue35224 6 msgs #36670: test suite broken due to cpu usage feature on win 10/ german https://bugs.python.org/issue36670 6 msgs #16079: list duplicate test names with patchcheck https://bugs.python.org/issue16079 5 msgs #35824: http.cookies._CookiePattern modifying regular expressions https://bugs.python.org/issue35824 5 msgs #36661: Missing dataclass decorator import in dataclasses module docs https://bugs.python.org/issue36661 5 msgs #36719: regrtest --findleaks should fail if an uncollectable object is https://bugs.python.org/issue36719 5 msgs #32424: Synchronize copy methods between Python and C implementations https://bugs.python.org/issue32424 4 msgs #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 4 msgs Issues closed (40) ================== #9194: winreg:fixupMultiSZ should check that P < Q in the inner loop https://bugs.python.org/issue9194 closed by steve.dower #17349: wsgiref.simple_server.demo_app is not PEP-3333 compatible https://bugs.python.org/issue17349 closed by berker.peksag #18372: _Pickler_New() doesn't call PyObject_GC_Track(self) https://bugs.python.org/issue18372 closed by inada.naoki #21536: extension built with a shared python cannot be loaded with a s https://bugs.python.org/issue21536 closed by vstinner #23078: unittest.mock patch autospec doesn't work on staticmethods https://bugs.python.org/issue23078 closed by berker.peksag #24011: Add error checks to PyInit_signal() https://bugs.python.org/issue24011 closed by berker.peksag #28552: Distutils fail if sys.executable is None https://bugs.python.org/issue28552 closed by vstinner #30840: Contrary to documentation, relative imports cannot pass throug https://bugs.python.org/issue30840 closed by ncoghlan #35149: pip3 show causing Error for ConfigParaser https://bugs.python.org/issue35149 closed by eryksun #36454: test_time: test_monotonic() failed on AMD64 FreeBSD 10-STABLE https://bugs.python.org/issue36454 closed by vstinner #36465: Make release and debug ABI compatible https://bugs.python.org/issue36465 closed by vstinner #36469: Stuck during interpreter exit, attempting to take the GIL https://bugs.python.org/issue36469 closed by mocramis #36523: Add docstring to io.IOBase.writelines https://bugs.python.org/issue36523 closed by inada.naoki #36546: Add quantiles() to the statistics module https://bugs.python.org/issue36546 closed by rhettinger #36635: Add _testinternalcapi module https://bugs.python.org/issue36635 closed by vstinner #36645: re.sub() library entry does not adequately document surprising https://bugs.python.org/issue36645 closed by berker.peksag #36650: Cached method implementation no longer works on Python 3.7.3 https://bugs.python.org/issue36650 closed by rhettinger #36658: Py_Initialze() throws error 'unable to load the file system en https://bugs.python.org/issue36658 closed by steve.dower #36659: distutils UnixCCompiler: Remove standard library path from rpa https://bugs.python.org/issue36659 closed by vstinner #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 closed by pitrou #36669: weakref proxy doesn't support the matrix multiplication operat https://bugs.python.org/issue36669 closed by SilentGhost #36671: str.lower() looses character information when working with UTF https://bugs.python.org/issue36671 closed by SilentGhost #36677: support visual studio multiprocess compile https://bugs.python.org/issue36677 closed by Manjusaka #36682: duplicate method definitions in Lib/test/test_sys_setprofile.p https://bugs.python.org/issue36682 closed by steve.dower #36687: subprocess encoding https://bugs.python.org/issue36687 closed by sbstp #36690: A typing error in demo rpython.py https://bugs.python.org/issue36690 closed by berker.peksag #36693: Reversing large ranges results in a minor type inconsistency https://bugs.python.org/issue36693 closed by rhettinger #36695: Change (regression?) in v3.8.0a3 doctest output after capturin https://bugs.python.org/issue36695 closed by bskinn #36696: possible multiple regressions on AIX https://bugs.python.org/issue36696 closed by vstinner #36701: module 'urllib' has no attribute 'request' https://bugs.python.org/issue36701 closed by brett.cannon #36705: Unexpected Behaviour of pprint.pprint https://bugs.python.org/issue36705 closed by fdrake #36707: The "m" ABI flag of SOABI for pymalloc is no longer needed https://bugs.python.org/issue36707 closed by vstinner #36708: can not execute the python + version, to launch python under w https://bugs.python.org/issue36708 closed by brett.cannon #36718: Python 2.7 compilation fails on AMD64 Ubuntu Shared 2.7 buildb https://bugs.python.org/issue36718 closed by vstinner #36720: Correct Should to Must in Definition of object.__len__ https://bugs.python.org/issue36720 closed by brett.cannon #36722: In debug build, load also C extensions compiled in release mod https://bugs.python.org/issue36722 closed by vstinner #36726: Empty select() on windows gives error. https://bugs.python.org/issue36726 closed by martin.panter #36727: python 3.6+ docs use ul tags instead of ol tags https://bugs.python.org/issue36727 closed by eric.smith #36731: Add example to priority queue https://bugs.python.org/issue36731 closed by rhettinger #36733: make regen-all doesn't work in subfolder: No module named Pars https://bugs.python.org/issue36733 closed by vstinner

1 0

Steering Council Update for April 2019
by Guido van Rossum 26 Apr '19

26 Apr '19

I've posted an update from the Steering Council to our repo: https://github.com/python/steering-council/blob/master/updates/2019-04-26_s… I will also link to this on Discourse (discuss.python.org). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

2 1

Any core dev event plans for EP19?
by Stefan Behnel 26 Apr '19

26 Apr '19

Hi core devs, there are several core dev events happening at the US PyCon this year, so I was wondering if we could organise something similar at EuroPython. Does anyone have any plans or ideas already? And, how many of us are planning to attend EP19 in Basel this year? Unless there's something already going on that I missed, I can (try to) set up a poll on dpo to count the interest and collect ideas. Sprints would probably be a straight-forward option, a mentoring session could be another, a language summit or PEP discussion/mentoring round would also be a possibility. More ideas welcome. Stefan

4 4

Invitation to the PyCon typing summit (Thu, May 2nd, 1-5pm, at PyCon in Cleveland)
by Guido van Rossum 24 Apr '19

24 Apr '19

The typing summit is primarily a place for developers of type checkers to collaborate, but we are also inviting (potential) users of type checkers. For example, there are plans to extend the standard Python type system with features intended to support numpy, Pandas, tensorflow and similar libraries, and we will discuss these at the summit. Therefore developers and power-users of such frameworks are especially welcome at the summit. With Ewa's and Dropbox's help I've arranged a room at PyCon. *When: Thursday May 2nd, 1-5 pm (i.e. the day between the Language Summit and the conference proper)* *Where: Room 6 at PyCon in Cleveland* If you're planning to attend, please fill out this form: https://goo.gl/forms/rG9dVTBbgyBgDK8H2 *If you already emailed me about attending, please still fill out the form!* (Also if you already registered in response to my invitation on typing-sig, I've got your registration info.) -- --Guido van Rossum (python.org/~guido)

1 1

Proposal: dict.with_values(iterable)
by Inada Naoki 23 Apr '19

23 Apr '19

Hi, all. I propose adding new method: dict.with_values(iterable) # Motivation Python is used to handle data. While dict is not efficient way to handle may records, it is still convenient way. When creating many dicts with same keys, dict need to lookup internal hash table while inserting each keys. It is costful operation. If we can reuse existing keys of dict, we can skip this inserting cost. Additionally, we have "Key-Sharing Dictionary (PEP 412)". When all keys are string, many dict can share one key. It reduces memory consumption. This might be usable for: * csv.DictReader * namedtuple._asdict() * DB-API 2.0 implementations: (e.g. DictCursor of mysqlclient-python) # Draft implementation pull request: https://github.com/python/cpython/pull/12802 with_values(self, iterable, /) Create a new dictionary with keys from this dict and values from iterable. When length of iterable is different from len(self), ValueError is raised. This method does not support dict subclass. ## Memory usage (Key-Sharing dict) >>> import sys >>> keys = tuple("abcdefg") >>> keys ('a', 'b', 'c', 'd', 'e', 'f', 'g') >>> d = dict(zip(keys, range(7))) >>> d {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6} >>> sys.getsizeof(d) 360 >>> keys = dict.fromkeys("abcdefg") >>> d = keys.with_values(range(7)) >>> d {'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4, 'f': 5, 'g': 6} >>> sys.getsizeof(d) 144 ## Speed $ ./python -m perf timeit -o zip_dict.json -s 'keys = tuple("abcdefg"); values=[*range(7)]' 'dict(zip(keys, values))' $ ./python -m perf timeit -o with_values.json -s 'keys = dict.fromkeys("abcdefg"); values=[*range(7)]' 'keys.with_values(values)' $ ./python -m perf compare_to zip_dict.json with_values.json Mean +- std dev: [zip_dict] 935 ns +- 9 ns -> [with_values] 109 ns +- 2 ns: 8.59x faster (-88%) How do you think? Any comments are appreciated. Regards, -- Inada Naoki <songofacandy(a)gmail.com>

9 26

Concurrent.futures: no type discovery for PyCharm
by Ilya Kamenshchikov 23 Apr '19

23 Apr '19

I am using concurrent.futures to parallelize independent tasks on multiple cores once in a while. Each time I have a difficulty remembering the specific syntax and have to look it up in old code or google. I would much prefer to be able to find the source through the PyCharm and have autocompletion. It takes adding two lines to the __init__.py of concurrent.futures: (insert on line 19) from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor I would also guess that it would make the __getattr__ redundant? Am I missing something or can this change be done this way and would indeed be an improvement? Best Regards, -- Ilya Kamenshchikov

9 16

Python Documentation Translation in italian language
by Alessandro Cucci 20 Apr '19

20 Apr '19

Hello folks, I want to start a project for translating the Python Documentation in Italian. I'm reading the PEP545, trying to understand how it works. I founded a Python User Group in my city and I can work with them on the translations, plus next month I will be speaker at Pycon Italy, so I can easily sponsor this project during the talk and reclute more people to work on that. Is there anybody who can help me to start? Thanks, have a nice day. *Alessandro Cucci*

2 1

Summary of Python tracker Issues
by Python tracker 19 Apr '19

19 Apr '19

ACTIVITY SUMMARY (2019-04-12 - 2019-04-19) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7058 ( -5) closed 41363 (+56) total 48421 (+51) Open issues with patches: 2807 Issues opened (33) ================== #30485: Element.findall(path, dict) doesn't insert null namespace https://bugs.python.org/issue30485 reopened by scoder #36345: Deprecate Tools/scripts/serve.py in favour of python -m http.s https://bugs.python.org/issue36345 reopened by matrixise #36620: Documentation missing parameter for Itertools.zip_longest https://bugs.python.org/issue36620 opened by CharlesMerriam #36621: shutil.rmtree follows junctions on windows https://bugs.python.org/issue36621 opened by Jordan Hueckstaedt #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 opened by Michael.Felt #36630: failure of test_colors_funcs in test_curses with ncurses 6.1 https://bugs.python.org/issue36630 opened by xdegaye #36631: test_urllib2net: test_ftp_no_timeout() killed after a timeout https://bugs.python.org/issue36631 opened by vstinner #36632: test_multiprocessing_forkserver: test_rapid_restart() leaked a https://bugs.python.org/issue36632 opened by vstinner #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 opened by BWenzel #36635: Add _testinternalcapi module https://bugs.python.org/issue36635 opened by vstinner #36636: Inner exception is not being raised using asyncio.gather https://bugs.python.org/issue36636 opened by Drew Budwin #36640: python ibm_db setup.py post install script does not seem to wo https://bugs.python.org/issue36640 opened by sabakauser #36643: Forward reference is not resolved by dataclasses.fields() https://bugs.python.org/issue36643 opened by mdrachuk #36644: Improve documentation of slice.indices() https://bugs.python.org/issue36644 opened by pewscorner #36645: re.sub() library entry does not adequately document surprising https://bugs.python.org/issue36645 opened by mollison #36646: os.listdir() got permission error in Python3.6 but it's fine i https://bugs.python.org/issue36646 opened by Ryan_D(a)163.com #36647: TextTestRunner doesn't honour "buffer" argument https://bugs.python.org/issue36647 opened by José Luis Segura Lucas #36648: MAP_SHARED isn't proper for anonymous mappings for VxWorks https://bugs.python.org/issue36648 opened by lzhao #36650: Cached method implementation no longer works on Python 3.7.3 https://bugs.python.org/issue36650 opened by jaraco #36654: Add example to tokenize.tokenize https://bugs.python.org/issue36654 opened by Windson Yang #36656: Allow os.symlink(src, target, force=True) to prevent race cond https://bugs.python.org/issue36656 opened by Tom Hale #36658: Py_Initialze() throws error 'unable to load the file system en https://bugs.python.org/issue36658 opened by rvq #36659: distutils UnixCCompiler: Remove standard library path from rpa https://bugs.python.org/issue36659 opened by vstinner #36661: Missing dataclass decorator import in dataclasses module docs https://bugs.python.org/issue36661 opened by mfisherlevine #36662: asdict/astuple Dataclass methods https://bugs.python.org/issue36662 opened by gsakkis #36663: pdb: store whole exception information in locals (via user_exc https://bugs.python.org/issue36663 opened by blueyed #36664: argparse: parser aliases in subparsers stores alias in dest va https://bugs.python.org/issue36664 opened by Peter McEldowney #36665: REPL doesn't ensure builtins are available when implicitly rec https://bugs.python.org/issue36665 opened by ncoghlan #36666: threading.Thread should have way to catch an exception thrown https://bugs.python.org/issue36666 opened by Joel Croteau #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 opened by blueyed #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 opened by tomMoral #36669: weakref proxy doesn't support the matrix multiplication operat https://bugs.python.org/issue36669 opened by bup #36670: test suite broken due to cpu usage feature on win 10/ german https://bugs.python.org/issue36670 opened by LorenzMende Most recent 15 issues with no replies (15) ========================================== #36669: weakref proxy doesn't support the matrix multiplication operat https://bugs.python.org/issue36669 #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 #36663: pdb: store whole exception information in locals (via user_exc https://bugs.python.org/issue36663 #36654: Add example to tokenize.tokenize https://bugs.python.org/issue36654 #36647: TextTestRunner doesn't honour "buffer" argument https://bugs.python.org/issue36647 #36644: Improve documentation of slice.indices() https://bugs.python.org/issue36644 #36643: Forward reference is not resolved by dataclasses.fields() https://bugs.python.org/issue36643 #36621: shutil.rmtree follows junctions on windows https://bugs.python.org/issue36621 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 Most recent 15 issues waiting for review (15) ============================================= #36668: semaphore_tracker is not reused by child processes https://bugs.python.org/issue36668 #36667: pdb: restore SIGINT handler in sigint_handler already https://bugs.python.org/issue36667 #36659: distutils UnixCCompiler: Remove standard library path from rpa https://bugs.python.org/issue36659 #36648: MAP_SHARED isn't proper for anonymous mappings for VxWorks https://bugs.python.org/issue36648 #36645: re.sub() library entry does not adequately document surprising https://bugs.python.org/issue36645 #36635: Add _testinternalcapi module https://bugs.python.org/issue36635 #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 #36601: signals can be caught by any thread https://bugs.python.org/issue36601 Top 10 most discussed issues (10) ================================= #35866: concurrent.futures deadlock https://bugs.python.org/issue35866 17 msgs #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 14 msgs #30485: Element.findall(path, dict) doesn't insert null namespace https://bugs.python.org/issue30485 10 msgs #36646: os.listdir() got permission error in Python3.6 but it's fine i https://bugs.python.org/issue36646 10 msgs #27987: obmalloc's 8-byte alignment causes undefined behavior https://bugs.python.org/issue27987 8 msgs #36624: cleanup the stdlib and tests with regard to sys.platform usage https://bugs.python.org/issue36624 8 msgs #16079: list duplicate test names with patchcheck https://bugs.python.org/issue16079 7 msgs #31904: Python should support VxWorks RTOS https://bugs.python.org/issue31904 6 msgs #36634: venv: activate.bat fails for venv with parentheses in PATH https://bugs.python.org/issue36634 6 msgs #32782: memoryview & ctypes: incorrect itemsize for empty array https://bugs.python.org/issue32782 5 msgs Issues closed (53) ================== #2007: cookielib lacks FileCookieJar class for Internet Explorer https://bugs.python.org/issue2007 closed by inada.naoki #15917: hg hook to detect unmerged changesets https://bugs.python.org/issue15917 closed by inada.naoki #16254: Make PyUnicode_AsWideCharString() increase temporary https://bugs.python.org/issue16254 closed by inada.naoki #18610: wsgiref.validate expects wsgi.input read to give exactly one a https://bugs.python.org/issue18610 closed by cheryl.sabella #22991: test_gdb leaves the terminal in raw mode with gdb 7.8.1 https://bugs.python.org/issue22991 closed by xdegaye #23768: assert on getting the representation of a thread in atexit fun https://bugs.python.org/issue23768 closed by xdegaye #27326: SIGSEV in test_window_funcs of test_curses https://bugs.python.org/issue27326 closed by xdegaye #28055: pyhash's siphash24 assumes alignment of the data pointer https://bugs.python.org/issue28055 closed by vstinner #28809: mention asyncio.gather non-deterministic task starting order https://bugs.python.org/issue28809 closed by cheryl.sabella #31658: xml.sax.parse won't accept path objects https://bugs.python.org/issue31658 closed by scoder #32849: Fatal Python error: Py_Initialize: can't initialize sys standa https://bugs.python.org/issue32849 closed by vstinner #32913: Improve regular expression HOWTO https://bugs.python.org/issue32913 closed by brett.cannon #33783: Use proper class markup for random.Random docs https://bugs.python.org/issue33783 closed by vstinner #34814: makesetup: must link C extensions to libpython when compiled i https://bugs.python.org/issue34814 closed by vstinner #35581: Document @typing.type_check_only https://bugs.python.org/issue35581 closed by gvanrossum #35697: _decimal: Implement the previously rejected changes from #7442 https://bugs.python.org/issue35697 closed by vstinner #35755: On Unix, shutil.which() and subprocess no longer look for the https://bugs.python.org/issue35755 closed by vstinner #36071: Add support for Windows ARM32 in ctypes/libffi https://bugs.python.org/issue36071 closed by steve.dower #36227: Add default_namespace argument to xml.etree.ElementTree.tostri https://bugs.python.org/issue36227 closed by scoder #36263: test_hashlib.test_scrypt() fails on Fedora 29 https://bugs.python.org/issue36263 closed by vstinner #36348: test_imaplib.RemoteIMAP_STARTTLSTest.test_logout() fails rando https://bugs.python.org/issue36348 closed by vstinner #36427: Document that PyEval_RestoreThread and PyGILState_Ensure can t https://bugs.python.org/issue36427 closed by pablogsal #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 closed by cary #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 closed by vstinner #36558: Change time.mktime() return type from float to int? https://bugs.python.org/issue36558 closed by vstinner #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 closed by SilentGhost #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 closed by pablogsal #36593: isinstance check fails for Mock objects with spec executed und https://bugs.python.org/issue36593 closed by pablogsal #36600: re-enable test in nntplib https://bugs.python.org/issue36600 closed by Marcin Niemira #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 closed by vstinner #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 closed by vstinner #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 closed by vstinner #36622: Inconsistent exponent notation formatting https://bugs.python.org/issue36622 closed by mark.dickinson #36623: Clean unused parser headers https://bugs.python.org/issue36623 closed by pablogsal #36625: Obsolete comments in docstrings in fractions module https://bugs.python.org/issue36625 closed by mark.dickinson #36626: asyncio run_forever blocks indefinitely https://bugs.python.org/issue36626 closed by asvetlov #36627: composing generator expression doesn't work as expected https://bugs.python.org/issue36627 closed by SilentGhost #36628: Enhancement: i-Strings https://bugs.python.org/issue36628 closed by SilentGhost #36629: imaplib test fails with errno 101 https://bugs.python.org/issue36629 closed by vstinner #36633: py_compile.compile: AttributeError on importlib.utils https://bugs.python.org/issue36633 closed by xtreak #36637: Restrict syntax for tuple literals with one element https://bugs.python.org/issue36637 closed by brett.cannon #36638: typeperf.exe is not in all skus of Windows SKUs https://bugs.python.org/issue36638 closed by steve.dower #36639: Provide list.rindex() https://bugs.python.org/issue36639 closed by rhettinger #36641: make docstring in C const https://bugs.python.org/issue36641 closed by inada.naoki #36642: make unicodedata "const" https://bugs.python.org/issue36642 closed by inada.naoki #36649: Windows Store app install registry keys have incorrect paths https://bugs.python.org/issue36649 closed by steve.dower #36651: Asyncio Event Loop documentation inconsistency (call_later and https://bugs.python.org/issue36651 closed by asvetlov #36652: Non-embedded zip distribution https://bugs.python.org/issue36652 closed by steve.dower #36653: Dictionary Key is without ' ' quotes https://bugs.python.org/issue36653 closed by mrabarnett #36655: Division Precision Problem https://bugs.python.org/issue36655 closed by christian.heimes #36657: AttributeError https://bugs.python.org/issue36657 closed by xtreak #36660: TypeError https://bugs.python.org/issue36660 closed by vstinner #1402289: Allow mappings as globals (was: Fix dictionary subclass ...) https://bugs.python.org/issue1402289 closed by rhettinger

1 0

PEP 591 discussion (final qualifier) happening at typing-sig@
by Michael Sullivan 18 Apr '19

18 Apr '19

I've submitted PEP 591 (Adding a final qualifier to typing) for discussion to typing-sig [1]. Here's the abstract: This PEP proposes a "final" qualifier to be added to the ``typing`` module---in the form of a ``final`` decorator and a ``Final`` type annotation---to serve three related purposes: * Declaring that a method should not be overridden * Declaring that a class should not be subclassed * Declaring that a variable or attribute should not be reassigned Full text at https://www.python.org/dev/peps/pep-0591/ -sully [1] https://mail.python.org/mailman3/lists/typing-sig.python.org/

3 3

Adding shlex.join?
by Paul Ganssle 17 Apr '19

17 Apr '19

Hey all, I've been reviewing old "awaiting review" PRs recently, and about a week ago I found PR #7605 <https://github.com/python/cpython/pull/7605>, adding shlex.join(), with a corresponding bug at bpo-22454 <https://bugs.python.org/issue22454>. The PR's implementation is simple and seems reasonable and decently well-tested, but it has been unreviewed for ~10 months. The reason I'm bringing it up here is that I believe the major blocker here is getting agreement to actually add the function. There doesn't seem to be much /opposition/ in the BPO issue, but given how infrequently the shlex module is changed I'm worried that there may be no one around who feels confident to judge how the interface should evolve. Does anyone feel strongly about this issue? Is there anyone who wants to make a yes/no decision on this feature? Best, Paul P.S. The PR's submitter seems responsive. I made a comment on the documentation and it was addressed in something like 5 minutes.

4 3

PEP 7: Adding anonymous union / struct
by Inada Naoki 17 Apr '19

17 Apr '19

Hi, all. PEP 7 includes some C99 features. I propose to add include anonymous union and struct to the list. https://www.geeksforgeeks.org/g-fact-38-anonymous-union-and-structure/ Anonymous union and struct are C11 feature, not C99. But gcc and MSVC supported it as language extension from before C11. Anonymous union is useful when all union members have different names. Especially, when we need to add dummy member only for padding / alignment, union name looks too verbose: ... # in some struct union { struct { int member1; int member2; } s; long double _dummy; // for largest alignment. } u; ... x.u.s.member1 = 42; vs ... union { struct { int member1; int member2; }; long double _dummy; // for largest alignment. }; ... x.member1 = 42; Does anyone know compiler which can be use to compile Python but doesn't support anonymous union / struct? Regards, -- Inada Naoki <songofacandy(a)gmail.com>

2 2

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build
by Michael Sullivan 16 Apr '19

16 Apr '19

> The main question is if anyone ever used Py_TRACE_REFS? Does someone > use sys.getobjects() or PYTHONDUMPREFS environment variable? I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy. We were leaking memory badly but no sign of the leak was showing up in mypy's gc.get_objects() based profiler. Using a debug build and switching to sys.getobjects() showed that we were badly leaking int objects. A quick inspection of the values in question (large and random looking) suggested we were leaking hash values, and that quickly pointed me to https://github.com/mypyc/mypyc/pull/562. I don't have any strong feelings about whether to keep it in the "default" debug build, though. I was using a debug build that I built myself with every debug feature that seemed potentially useful. -sully

3 6

PEP 578: Python Runtime Audit Hooks
by Steve Dower 16 Apr '19

16 Apr '19

Hi all Time is short, but I'm hoping to get PEP 578 (formerly PEP 551) into Python 3.8. Here's the current text for review and comment before I submit to the Steering Council. The formatted text is at https://www.python.org/dev/peps/pep-0578/ (update just pushed, so give it an hour or so, but it's fundamentally the same as what's there) No Discourse post, because we don't have a python-dev equivalent there yet, so please reply here for this one. Implementation is at https://github.com/zooba/cpython/tree/pep-578/ and my backport to 3.7 (https://github.com/zooba/cpython/tree/pep-578-3.7/) is already getting some real use (though this will not be added to 3.7, unless people *really* want it, so the backport is just for reference). Cheers, Steve ===== PEP: 578 Title: Python Runtime Audit Hooks Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-Jun-2018 Python-Version: 3.8 Post-History: Abstract ======== This PEP describes additions to the Python API and specific behaviors for the CPython implementation that make actions taken by the Python runtime visible to auditing tools. Visibility into these actions provides opportunities for test frameworks, logging frameworks, and security tools to monitor and optionally limit actions taken by the runtime. This PEP proposes adding two APIs to provide insights into a running Python application: one for arbitrary events, and another specific to the module import system. The APIs are intended to be available in all Python implementations, though the specific messages and values used are unspecified here to allow implementations the freedom to determine how best to provide information to their users. Some examples likely to be used in CPython are provided for explanatory purposes. See PEP 551 for discussion and recommendations on enhancing the security of a Python runtime making use of these auditing APIs. Background ========== Python provides access to a wide range of low-level functionality on many common operating systems. While this is incredibly useful for "write-once, run-anywhere" scripting, it also makes monitoring of software written in Python difficult. Because Python uses native system APIs directly, existing monitoring tools either suffer from limited context or auditing bypass. Limited context occurs when system monitoring can report that an action occurred, but cannot explain the sequence of events leading to it. For example, network monitoring at the OS level may be able to report "listening started on port 5678", but may not be able to provide the process ID, command line, parent process, or the local state in the program at the point that triggered the action. Firewall controls to prevent such an action are similarly limited, typically to process names or some global state such as the current user, and in any case rarely provide a useful log file correlated with other application messages. Auditing bypass can occur when the typical system tool used for an action would ordinarily report its use, but accessing the APIs via Python do not trigger this. For example, invoking "curl" to make HTTP requests may be specifically monitored in an audited system, but Python's "urlretrieve" function is not. Within a long-running Python application, particularly one that processes user-provided information such as a web app, there is a risk of unexpected behavior. This may be due to bugs in the code, or deliberately induced by a malicious user. In both cases, normal application logging may be bypassed resulting in no indication that anything out of the ordinary has occurred. Additionally, and somewhat unique to Python, it is very easy to affect the code that is run in an application by manipulating either the import system's search path or placing files earlier on the path than intended. This is often seen when developers create a script with the same name as the module they intend to use - for example, a ``random.py`` file that attempts to import the standard library ``random`` module. This is not sandboxing, as this proposal does not attempt to prevent malicious behavior (though it enables some new options to do so). See the `Why Not A Sandbox`_ section below for further discussion. Overview of Changes =================== The aim of these changes is to enable both application developers and system administrators to integrate Python into their existing monitoring systems without dictating how those systems look or behave. We propose two API changes to enable this: an Audit Hook and Verified Open Hook. Both are available from Python and native code, allowing applications and frameworks written in pure Python code to take advantage of the extra messages, while also allowing embedders or system administrators to deploy builds of Python where auditing is always enabled. Only CPython is bound to provide the native APIs as described here. Other implementations should provide the pure Python APIs, and may provide native versions as appropriate for their underlying runtimes. Auditing events are likewise considered implementation specific, but are bound by normal feature compatibility guarantees. Audit Hook ---------- In order to observe actions taken by the runtime (on behalf of the caller), an API is required to raise messages from within certain operations. These operations are typically deep within the Python runtime or standard library, such as dynamic code compilation, module imports, DNS resolution, or use of certain modules such as ``ctypes``. The following new C APIs allow embedders and CPython implementors to send and receive audit hook messages:: # Add an auditing hook typedef int (*hook_func)(const char *event, PyObject *args, void *userData); int PySys_AddAuditHook(hook_func hook, void *userData); # Raise an event with all auditing hooks int PySys_Audit(const char *event, PyObject *args); # Internal API used during Py_Finalize() - not publicly accessible void _Py_ClearAuditHooks(void); The new Python APIs for receiving and raising audit hooks are:: # Add an auditing hook sys.addaudithook(hook: Callable[[str, tuple]]) # Raise an event with all auditing hooks sys.audit(str, *args) Hooks are added by calling ``PySys_AddAuditHook()`` from C at any time, including before ``Py_Initialize()``, or by calling ``sys.addaudithook()`` from Python code. Hooks cannot be removed or replaced. When events of interest are occurring, code can either call ``PySys_Audit()`` from C (while the GIL is held) or ``sys.audit()``. The string argument is the name of the event, and the tuple contains arguments. A given event name should have a fixed schema for arguments, which should be considered a public API (for each x.y version release), and thus should only change between feature releases with updated documentation. For maximum compatibility, events using the same name as an event in the reference interpreter CPython should make every attempt to use compatible arguments. Including the name or an abbreviation of the implementation in implementation-specific event names will also help prevent collisions. For example, a ``pypy.jit_invoked`` event is clearly distinguised from an ``ipy.jit_invoked`` event. When an event is audited, each hook is called in the order it was added with the event name and tuple. If any hook returns with an exception set, later hooks are ignored and *in general* the Python runtime should terminate. This is intentional to allow hook implementations to decide how to respond to any particular event. The typical responses will be to log the event, abort the operation with an exception, or to immediately terminate the process with an operating system exit call. When an event is audited but no hooks have been set, the ``audit()`` function should impose minimal overhead. Ideally, each argument is a reference to existing data rather than a value calculated just for the auditing call. As hooks may be Python objects, they need to be freed during ``Py_Finalize()``. To do this, we add an internal API ``_Py_ClearAuditHooks()`` that releases any Python hooks and any memory held. This is an internal function with no public export, and we recommend it raise its own audit event for all current hooks to ensure that unexpected calls are observed. Below in `Suggested Audit Hook Locations`_, we recommend some important operations that should raise audit events. Python implementations should document which operations will raise audit events, along with the event schema. It is intentional that ``sys.addaudithook(print)`` be a trivial way to display all messages. Verified Open Hook ------------------ Most operating systems have a mechanism to distinguish between files that can be executed and those that can not. For example, this may be an execute bit in the permissions field, a verified hash of the file contents to detect potential code tampering, or file system path restrictions. These are an important security mechanism for preventing execution of data or code that is not approved for a given environment. Currently, Python has no way to integrate with these when launching scripts or importing modules. The new public C API for the verified open hook is:: # Set the handler typedef PyObject *(*hook_func)(PyObject *path, void *userData) int PyImport_SetOpenForImportHook(hook_func handler, void *userData) # Open a file using the handler PyObject *PyImport_OpenForImport(const char *path) The new public Python API for the verified open hook is:: # Open a file using the handler importlib.util.open_for_import(path : str) -> io.IOBase The ``importlib.util.open_for_import()`` function is a drop-in replacement for ``open(str(pathlike), 'rb')``. Its default behaviour is to open a file for raw, binary access. To change the behaviour a new handler should be set. Handler functions only accept ``str`` arguments. The C API ``PyImport_OpenForImport`` function assumes UTF-8 encoding. A custom handler may be set by calling ``PyImport_SetOpenForImportHook()`` from C at any time, including before ``Py_Initialize()``. However, if a hook has already been set then the call will fail. When ``open_for_import()`` is called with a hook set, the hook will be passed the path and its return value will be returned directly. The returned object should be an open file-like object that supports reading raw bytes. This is explicitly intended to allow a ``BytesIO`` instance if the open handler has already read the entire file into memory. Note that these hooks can import and call the ``_io.open()`` function on CPython without triggering themselves. They can also use ``_io.BytesIO`` to return a compatible result using an in-memory buffer. If the hook determines that the file should not be loaded, it should raise an exception of its choice, as well as performing any other logging. All import and execution functionality involving code from a file will be changed to use ``open_for_import()`` unconditionally. It is important to note that calls to ``compile()``, ``exec()`` and ``eval()`` do not go through this function - an audit hook that includes the code from these calls is the best opportunity to validate code that is read from the file. Given the current decoupling between import and execution in Python, most imported code will go through both ``open_for_import()`` and the log hook for ``compile``, and so care should be taken to avoid repeating verification steps. There is no Python API provided for changing the open hook. To modify import behavior from Python code, use the existing functionality provided by ``importlib``. API Availability ---------------- While all the functions added here are considered public and stable API, the behavior of the functions is implementation specific. Most descriptions here refer to the CPython implementation, and while other implementations should provide the functions, there is no requirement that they behave the same. For example, ``sys.addaudithook()`` and ``sys.audit()`` should exist but may do nothing. This allows code to make calls to ``sys.audit()`` without having to test for existence, but it should not assume that its call will have any effect. (Including existence tests in security-critical code allows another vector to bypass auditing, so it is preferable that the function always exist.) ``importlib.util.open_for_import(path)`` should at a minimum always return ``_io.open(path, 'rb')``. Code using the function should make no further assumptions about what may occur, and implementations other than CPython are not required to let developers override the behavior of this function with a hook. Suggested Audit Hook Locations ============================== The locations and parameters in calls to ``sys.audit()`` or ``PySys_Audit()`` are to be determined by individual Python implementations. This is to allow maximum freedom for implementations to expose the operations that are most relevant to their platform, and to avoid or ignore potentially expensive or noisy events. Table 1 acts as both suggestions of operations that should trigger audit events on all implementations, and examples of event schemas. Table 2 provides further examples that are not required, but are likely to be available in CPython. Refer to the documentation associated with your version of Python to see which operations provide audit events. .. csv-table:: Table 1: Suggested Audit Hooks :header: "API Function", "Event Name", "Arguments", "Rationale" :widths: 2, 2, 3, 6 ``PySys_AddAuditHook``, ``sys.addaudithook``, "", "Detect when new audit hooks are being added. " ``PyImport_SetOpenForImportHook``, ``setopenforimporthook``, "", " Detects any attempt to set the ``open_for_import`` hook. " "``compile``, ``exec``, ``eval``, ``PyAst_CompileString``, ``PyAST_obj2mod``", ``compile``, "``(code, filename_or_none)``", " Detect dynamic code compilation, where ``code`` could be a string or AST. Note that this will be called for regular imports of source code, including those that were opened with ``open_for_import``. " "``exec``, ``eval``, ``run_mod``", ``exec``, "``(code_object,)``", " Detect dynamic execution of code objects. This only occurs for explicit calls, and is not raised for normal function invocation. " ``import``, ``import``, "``(module, filename, sys.path, sys.meta_path, sys.path_hooks)``", "Detect when modules are imported. This is raised before the module name is resolved to a file. All arguments other than the module name may be ``None`` if they are not used or available. " "``open``", ``open``, "``(path, mode, flags)``", "Detect when a file is about to be opened. *path* and *mode* are the usual parameters to ``open`` if available, while *flags* is provided instead of *mode* in some cases. " ``PyEval_SetProfile``, ``sys.setprofile``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.setprofile`` eventually calls this function, so the event will be audited for each thread. " ``PyEval_SetTrace``, ``sys.settrace``, "", "Detect when code is injecting trace functions. Because of the implementation, exceptions raised from the hook will abort the operation, but will not be raised in Python code. Note that ``threading.settrace`` eventually calls this function, so the event will be audited for each thread. " "``_PyObject_GenericSetAttr``, ``check_set_special_type_attr``, ``object_set_class``, ``func_set_code``, ``func_set_[kw]defaults``"," ``object.__setattr__``","``(object, attr, value)``","Detect monkey patching of types and objects. This event is raised for the ``__class__`` attribute and any attribute on ``type`` objects. " "``_PyObject_GenericSetAttr``",``object.__delattr__``,"``(object, attr)``","Detect deletion of object attributes. This event is raised for any attribute on ``type`` objects. " "``Unpickler.find_class``",``pickle.find_class``,"``(module_name, global_name)``","Detect imports and global name lookup when unpickling. " .. csv-table:: Table 2: Potential CPython Audit Hooks :header: "API Function", "Event Name", "Arguments", "Rationale" :widths: 2, 2, 3, 6 ``_PySys_ClearAuditHooks``, ``sys._clearaudithooks``, "", "Notifies hooks they are being cleaned up, mainly in case the event is triggered unexpectedly. This event cannot be aborted. " ``code_new``, ``code.__new__``, "``(bytecode, filename, name)``", " Detect dynamic creation of code objects. This only occurs for direct instantiation, and is not raised for normal compilation. " ``func_new_impl``, ``function.__new__``, "``(code,)``", "Detect dynamic creation of function objects. This only occurs for direct instantiation, and is not raised for normal compilation. " "``_ctypes.dlopen``, ``_ctypes.LoadLibrary``", ``ctypes.dlopen``, " ``(module_or_path,)``", "Detect when native modules are used. " ``_ctypes._FuncPtr``, ``ctypes.dlsym``, "``(lib_object, name)``", " Collect information about specific symbols retrieved from native modules. " ``_ctypes._CData``, ``ctypes.cdata``, "``(ptr_as_int,)``", "Detect when code is accessing arbitrary memory using ``ctypes``. " "``new_mmap_object``",``mmap.__new__``,"``(fileno, map_size, access, offset)``", "Detects creation of mmap objects. On POSIX, access may have been calculated from the ``prot`` and ``flags`` arguments. " ``sys._getframe``, ``sys._getframe``, "``(frame_object,)``", "Detect when code is accessing frames directly. " ``sys._current_frames``, ``sys._current_frames``, "", "Detect when code is accessing frames directly. " "``socket.bind``, ``socket.connect``, ``socket.connect_ex``, ``socket.getaddrinfo``, ``socket.getnameinfo``, ``socket.sendmsg``, ``socket.sendto``", ``socket.address``, "``(address,)``", "Detect access to network resources. The address is unmodified from the original call. " "``member_get``, ``func_get_code``, ``func_get_[kw]defaults`` ",``object.__getattr__``,"``(object, attr)``","Detect access to restricted attributes. This event is raised for any built-in members that are marked as restricted, and members that may allow bypassing imports. " "``urllib.urlopen``",``urllib.Request``,"``(url, data, headers, method)``", "Detects URL requests. " Performance Impact ================== The important performance impact is the case where events are being raised but there are no hooks attached. This is the unavoidable case - once a developer has added audit hooks they have explicitly chosen to trade performance for functionality. Performance impact with hooks added are not of interest here, since this is opt-in functionality. Analysis using the Python Performance Benchmark Suite [1]_ shows no significant impact, with the vast majority of benchmarks showing between 1.05x faster to 1.05x slower. In our opinion, the performance impact of the set of auditing points described in this PEP is negligible. Rejected Ideas ============== Separate module for audit hooks ------------------------------- The proposal is to add a new module for audit hooks, hypothetically ``audit``. This would separate the API and implementation from the ``sys`` module, and allow naming the C functions ``PyAudit_AddHook`` and ``PyAudit_Audit`` rather than the current variations. Any such module would need to be a built-in module that is guaranteed to always be present. The nature of these hooks is that they must be callable without condition, as any conditional imports or calls provide opportunities to intercept and suppress or modify events. Given it is one of the most core modules, the ``sys`` module is somewhat protected against module shadowing attacks. Replacing ``sys`` with a sufficiently functional module that the application can still run is a much more complicated task than replacing a module with only one function of interest. An attacker that has the ability to shadow the ``sys`` module is already capable of running arbitrary code from files, whereas an ``audit`` module could be replaced with a single line in a ``.pth`` file anywhere on the search path:: import sys; sys.modules['audit'] = type('audit', (object,), {'audit': lambda *a: None, 'addhook': lambda *a: None}) Multiple layers of protection already exist for monkey patching attacks against either ``sys`` or ``audit``, but assignments or insertions to ``sys.modules`` are not audited. This idea is rejected because it makes it trivial to suppress all calls to ``audit``. Flag in sys.flags to indicate "audited" mode -------------------------------------------- The proposal is to add a value in ``sys.flags`` to indicate when Python is running in a "secure" or "audited" mode. This would allow applications to detect when some features are enabled or when hooks have been added and modify their behaviour appropriately. Currently, we are not aware of any legitimate reasons for a program to behave differently in the presence of audit hooks. Both application-level APIs ``sys.audit`` and ``importlib.util.open_for_import`` are always present and functional, regardless of whether the regular ``python`` entry point or some alternative entry point is used. Callers cannot determine whether any hooks have been added (except by performing side-channel analysis), nor do they need to. The calls should be fast enough that callers do not need to avoid them, and the program is responsible for ensuring that any added hooks are fast enough to not affect application performance. The argument that this is "security by obscurity" is valid, but irrelevant. Security by obscurity is only an issue when there are no other protective mechanisms; obscurity as the first step in avoiding attack is strongly recommended (see `this article <https://danielmiessler.com/study/security-by-obscurity/>`_ for discussion). This idea is rejected because there are no appropriate reasons for an application to change its behaviour based on whether these APIs are in use. Why Not A Sandbox ================= Sandboxing CPython has been attempted many times in the past, and each past attempt has failed. Fundamentally, the problem is that certain functionality has to be restricted when executing the sandboxed code, but otherwise needs to be available for normal operation of Python. For example, completely removing the ability to compile strings into bytecode also breaks the ability to import modules from source code, and if it is not completely removed then there are too many ways to get access to that functionality indirectly. There is not yet any feasible way to generically determine whether a given operation is "safe" or not. Further information and references available at [2]_. This proposal does not attempt to restrict functionality, but simply exposes the fact that the functionality is being used. Particularly for intrusion scenarios, detection is significantly more important than early prevention (as early prevention will generally drive attackers to use an alternate, less-detectable, approach). The availability of audit hooks alone does not change the attack surface of Python in any way, but they enable defenders to integrate Python into their environment in ways that are currently not possible. Since audit hooks have the ability to safely prevent an operation occuring, this feature does enable the ability to provide some level of sandboxing. In most cases, however, the intention is to enable logging rather than creating a sandbox. Relationship to PEP 551 ======================= This API was originally presented as part of `PEP 551 <https://www.python.org/dev/peps/pep-0551/>`_ Security Transparency in the Python Runtime. For simpler review purposes, and due to the broader applicability of these APIs beyond security, the API design is now presented separately. PEP 551 is an informational PEP discussing how to integrate Python into a secure or audited environment. References ========== .. [1] Python Performance Benchmark Suite `<https://github.com/python/performance>`_ .. [2] Python Security model - Sandbox `<https://python-security.readthedocs.io/security.html#sandbox>`_ Copyright ========= Copyright (c) 2019 by Microsoft Corporation. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).

7 25

bpo-36558: Change time.mktime() return type from float to int?
by Victor Stinner 16 Apr '19

16 Apr '19

Hi, time.mktime() looks "inconsistent" to me and I would like to change it, but I'm not sure how it impacts backward compatibility. https://bugs.python.org/issue36558 time.mktime() returns a floating point number: >>> type(time.mktime(time.localtime())) <class 'float'> The documentation says: "It returns a floating point number, for compatibility with :func:`.time`." time.time() returns a float because it has sub-second resolution, but the C function mktime() returns an integer number of seconds. Would it make sense to change mktime() return type from float to int? I would like to change mktime() return type to make the function more consistent: all inputs are integers, it sounds wrong to me to return float. The result should be integer as well. How much code would it break? I guess that the main impact are unit tests relying on repr(time.mktime(t)) exact value. But it's easy to fix the tests: use int(time.mktime(t)) or "%.0f" % time.mktime(t) to never get ".0", or use float(time.mktime(t))) to explicitly cast for a float (that which be a bad but quick fix). Note: I wrote and implemented the PEP 564 to avoid any precision loss. mktime() will not start loosing precision before year 285,422,891 (which is quite far in the future ;-)). Victor -- Night gathers, and now my watch begins. It shall not end until my death.

5 6

PEP 589 discussion (TypedDict) happening at typing-sig@
by Jukka Lehtosalo 15 Apr '19

15 Apr '19

Hi everyone, I submitted PEP 589 (TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys) for discussion to typing-sig [1]. Here's an excerpt from the abstract of the PEP: PEP 484 defines the type Dict[K, V] for uniform dictionaries, where each value has the same type, and arbitrary key values are supported. It doesn't properly support the common pattern where the type of a dictionary value depends on the string value of the key. This PEP proposes a type constructor typing.TypedDict to support the use case where a dictionary object has a specific set of string keys, each with a value of a specific type. Jukka Lehtosalo [1] https://mail.python.org/mailman3/lists/typing-sig.python.org/

2 1

Cannot find documented API in PEP-376 (Database of Installed Python Distributions)
by Stefano Borini 15 Apr '19

15 Apr '19

Hello, I am on a PEP scouting effort to check the current status of python packaging and its historical context, mostly for learning purposes. I noted that the PEP defines some functions for pkgutil (e.g. get_distributions), but I cannot find them. I tried to do some searching on the mailing list history, but I came up with pretty much nothing of value. It appears that the topic was last considered in 2009 (the year of the PEP). dist-info was then implemented, but I cannot find any information about the missing API, nor any additional PEP, except for a brief reference in PEP-427. Does anyone have some context for this? I understand it was 10 years ago, so it's mostly a curiosity. Thanks. -- Kind regards, Stefano Borini

2 1

No longer enable Py_TRACE_REFS by default in debug build
by Victor Stinner 15 Apr '19

15 Apr '19

Hi, When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI. I propose to modify the debug build (Py_DEBUG) to no longer imply Py_TRACE_REFS. Antoine Pitrou proposed this idea when the C API was discussed to get a stable ABI. https://bugs.python.org/issue36465 https://github.com/python/cpython/pull/12615 This change makes the debug build ABI closer to the release build ABI, but I am not sure how to compare these two ABI. Technically, C extensions still need to be recompiled. What do you think? -- I also wrote a PR to remove all code related to Py_TRACE_REFS: https://github.com/python/cpython/pull/12614 I don't think that it's right approach. I prefer to keep this special build around to see if anyone needs it, and wait one or two Python releases to decide what to do with it. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

10 33

Season of Docs
by Senthil Kumaran 15 Apr '19

15 Apr '19

Hello Python Developers, Google is running a program called Season of Docs ( https://developers.google.com/season-of-docs/) to encourage technical writers to improve the documentation of Open Source Projects. As Python-Dev, and Python Software Foundation, do you think: a) We should participate? b) If yes to a), are you willing to be a mentor and identify project ideas? If you are willing to mentor and have project ideas, please let us know and we can think about the next steps. The deadline for org application is April 23, 2019. This discussion started here https://discuss.python.org/t/will-python-apply-for-season-of-docs-and-allow… Thank you, Senthil

2 1

Collaboration on a set of Python snaps
by Alan Pope 15 Apr '19

15 Apr '19

Hi Python devs, I work on the Snapcraft [0] team at Canonical. I'm looking for a Python contributor to collaborate with us on making snaps of supported releases of Python available in the Snap Store [1]. Travis CI and Canonical are looking for someone (preferably North-America based) to participate in an in-person Snapcraft Summit in downtown Montreal, Canada from 11th to 13th June. We're sponsoring a number of software vendors, device manufacturers people from the robotics sector to come. We'd love someone from the Python project to join us. We’ve published this blog post to explain the event in more detail: https://snapcraft.io/blog/snapcraft-summit-montreal The goal would be to create snaps of the major supported releases of Python, and authoritatively publish them in the Snap Store. This would enable users of many different Linux distributions to easily obtain up to date supported versions of Python directly from the Python project. It also enables providers of CI systems (such as Travis) to the latest builds of Python are easily available to developers who use their services. We've done this previously with NodeJS [2] and Ruby [3] - among others. It would be great to have Python available via this method too. All the best, Al. [0] - https://snapcraft.io/ [1] - https://snapcraft.io/store [2] - https://snapcraft.io/node [3] - https://snapcraft.io/ruby -- Alan Pope Community Advocate Canonical - Ubuntu Engineering and Services +44 (0) 7973 620 164 alan.pope(a)canonical.com http://ubuntu.com/

1 0

checking "errno" for math operaton is safe to determine the error status?
by Xin, Peixing 14 Apr '19

14 Apr '19

Hi, Math experts: Looking at the codes below, for many math operations, CPython is checking errno to determine the error status even though the math function returns normal value back. Is it a safe solution? From the description here http://man7.org/linux/man-pages/man3/errno.3.html and https://wiki.sei.cmu.edu/confluence/pages/viewpage.action?pageId=87152351, it looks apis probably set the errno when normal result is returned. Or being a side effect by calling other APIs in the implementation. In this situation, CPython's math operation might raise exceptions however in fact the result is correct. https://github.com/python/cpython/blob/master/Modules/mathmodule.c#L956 https://github.com/python/cpython/blob/master/Modules/mathmodule.c#L864 Thanks, Peixing

3 4

Fixing the ctypes implementation of the PEP3118 buffer interface
by Eric Wieser 14 Apr '19

14 Apr '19

I've recently been adding better support to Numpy 1.16 for interoperability with ctypes. In doing so, I came across two bugs in the implementation of the PEP3118 buffer interface within ctypes, affecting `Structure`s and arrays. Rather than repeating the issue summaries here, I've linked their tracker issues below, and the patches I filed to fix them. * https://bugs.python.org/issue32782 (patch: https://github.com/python/cpython/pull/5576) * https://bugs.python.org/issue32780 (patch: https://github.com/python/cpython/pull/5561) I've seen little to no response on either the bug tracker or the github PRs regarding these, so at the recommendation of the "Lifecycle of a Pull Request" am emailing this list. Without these fixes, numpy has no choice but to ignore the broken buffer interface that ctypes provides, and instead try to parse the ctypes types manually. The sooner this makes a CPython release, the sooner numpy can remove those workarounds. Thanks, Eric

2 1

duplicate method names in tests
by Xavier de Gaye 13 Apr '19

13 Apr '19

The last post [1] in issue bpo-16079 lists the methods in Lib/test that have duplicate names and that should be fixed. Xavier [1] https://bugs.python.org/issue16079#msg340168

1 0

Removing PID check from signal handler
by Jeroen Demeyer 12 Apr '19

12 Apr '19

The signal handler (that receives signals from the OS) in Python starts with a check if (getpid() == main_pid) Looking at the comments, the intent was to do a check for the main *thread* but this is checking the *process* id. So this condition is basically always true. Therefore, I suggest to remove it in https://bugs.python.org/issue36601 If you have any objections or comments, I suggest to post them to that bpo. Jeroen.

3 2

Update PEP 394: Distributions can choose what does python command mean
by Miro Hrončok 12 Apr '19

12 Apr '19

Hello. Based on discussions in [1], Petr Viktorin and me have drafted a new update [2] to the PEP 394 (The "python" Command on Unix-Like Systems). The update gives distributors the opportunity to decide where does the "python" command lead to, whether it is present etc. Please, see the PR [2] for the suggested changes. [1]: https://mail.python.org/pipermail/python-dev/2019-February/156272.html [2]: https://github.com/python/peps/pull/989 Thanks, -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 3

Summary of Python tracker Issues
by Python tracker 12 Apr '19

12 Apr '19

ACTIVITY SUMMARY (2019-04-05 - 2019-04-12) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7063 ( +7) closed 41307 (+76) total 48370 (+83) Open issues with patches: 2815 Issues opened (60) ================== #25160: Stop using deprecated imp module; imp should now emit a real D https://bugs.python.org/issue25160 reopened by ncoghlan #35934: Add socket.create_server() utility function https://bugs.python.org/issue35934 reopened by vstinner #36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var https://bugs.python.org/issue36235 reopened by vstinner #36537: except statement block incorrectly assumes end of scope(?). https://bugs.python.org/issue36537 opened by Saim Raza #36538: _thread.interrupt_main() no longer interrupts Lock.wait https://bugs.python.org/issue36538 opened by gregory.p.smith #36540: PEP 570: Python Positional-Only Parameters https://bugs.python.org/issue36540 opened by pablogsal #36541: Make lib2to3 grammar more closely match Python https://bugs.python.org/issue36541 opened by thatch #36542: Allow to overwrite the signature for Python functions https://bugs.python.org/issue36542 opened by serhiy.storchaka #36543: Remove old-deprecated ElementTree features (part 2) https://bugs.python.org/issue36543 opened by serhiy.storchaka #36545: Python 3.5 OOM during test_socket on make https://bugs.python.org/issue36545 opened by dekken #36546: Add quantiles() to the statistics module https://bugs.python.org/issue36546 opened by rhettinger #36548: Make the repr of re flags more readable https://bugs.python.org/issue36548 opened by serhiy.storchaka #36550: Avoid creating AttributeError exceptions in the debugger https://bugs.python.org/issue36550 opened by blueyed #36551: Optimize list comprehensions with preallocate size and protect https://bugs.python.org/issue36551 opened by anthony shaw #36552: Replace OverflowError with ValueError when calculating length https://bugs.python.org/issue36552 opened by anthony shaw #36553: inspect.is_decorator_call(frame) https://bugs.python.org/issue36553 opened by smarie #36556: Trashcan causing duplicated __del__ calls https://bugs.python.org/issue36556 opened by jdemeyer #36557: Python (Launcher)3.7.3 CMDLine install/uninstall https://bugs.python.org/issue36557 opened by mattcher_h #36558: Change time.mktime() return type from float to int? https://bugs.python.org/issue36558 opened by vstinner #36560: test_functools leaks randomly 1 memory block https://bugs.python.org/issue36560 opened by vstinner #36563: pdbrc is read twice if current directory is the home directory https://bugs.python.org/issue36563 opened by blueyed #36564: Infinite loop with short maximum line lengths in EmailPolicy https://bugs.python.org/issue36564 opened by p-ganssle #36567: DOC: manpage directive doesn't create hyperlink https://bugs.python.org/issue36567 opened by cheryl.sabella #36568: Typo in socket.CAN_RAW_FD_FRAMES library documentation https://bugs.python.org/issue36568 opened by Carl Cerecke #36569: @staticmethod seems to work with setUpClass, but docs say it s https://bugs.python.org/issue36569 opened by Peter de Blanc #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 opened by telatoa #36573: zipfile zipfile.BadZipFile: Bad CRC-32 for file '11_02_2019.pd https://bugs.python.org/issue36573 opened by Jozef Cernak #36576: Some test_ssl and test_asyncio tests fail with OpenSSL 1.1.1 o https://bugs.python.org/issue36576 opened by vstinner #36580: unittest.mock does not understand dataclasses https://bugs.python.org/issue36580 opened by John Parejko2 #36581: __dir__ on unittest.mock not safe for all spec types https://bugs.python.org/issue36581 opened by Dylan Semler #36582: collections.UserString encode method returns a string https://bugs.python.org/issue36582 opened by trey #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 opened by serhiy.storchaka #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 opened by jdemeyer #36586: multiprocessing.Queue.close doesn't behave as documented https://bugs.python.org/issue36586 opened by graingert #36587: race in logging code when fork() https://bugs.python.org/issue36587 opened by cagney #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 opened by ZackerySpytz #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 opened by topnotcher #36593: Trace function interferes with MagicMock isinstance? https://bugs.python.org/issue36593 opened by nedbat #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 opened by ZackerySpytz #36595: IDLE: Add search to Squeezed Output text viewer. https://bugs.python.org/issue36595 opened by Shane Smith #36596: tarfile module considers anything starting with 512 bytes of z https://bugs.python.org/issue36596 opened by cks #36600: re-enable test in nntplib https://bugs.python.org/issue36600 opened by Marcin Niemira #36601: signals can be caught by any thread https://bugs.python.org/issue36601 opened by jdemeyer #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 opened by Epic_Wink #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 opened by cagney #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 opened by vstinner #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 opened by xtreak #36607: asyncio.all_tasks() crashes if asyncio is used in multiple thr https://bugs.python.org/issue36607 opened by Nick Davies #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 opened by webknjaz #36609: activate.ps1 in venv for Windows should encoded with BOM https://bugs.python.org/issue36609 opened by 정한솔 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 opened by kulikjak #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 opened by vstinner #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 opened by vrpolakatcisco #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 opened by gescheit #36614: Popen output on windows server 2019 https://bugs.python.org/issue36614 opened by weispinc #36615: why call _Py_set_inheritable(0) from os.open() when O_CLOEXEC? https://bugs.python.org/issue36615 opened by cagney #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 opened by jdemeyer #36617: The rich comparison operators are second class citizens https://bugs.python.org/issue36617 opened by bup #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 opened by vstinner #36619: when is os.posix_spawn(setsid=True) safe? https://bugs.python.org/issue36619 opened by cagney Most recent 15 issues with no replies (15) ========================================== #36619: when is os.posix_spawn(setsid=True) safe? https://bugs.python.org/issue36619 #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36616: Optimize thread state handling in function call code https://bugs.python.org/issue36616 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36607: asyncio.all_tasks() crashes if asyncio is used in multiple thr https://bugs.python.org/issue36607 #36606: calling super() causes __class__ to be not defined when sys.se https://bugs.python.org/issue36606 #36603: should pty.openpty() set pty/tty inheritable? https://bugs.python.org/issue36603 #36596: tarfile module considers anything starting with 512 bytes of z https://bugs.python.org/issue36596 #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 #36583: Do not swallow exceptions in the _ssl module https://bugs.python.org/issue36583 #36572: python-snappy install issue during Crossbar install with Pytho https://bugs.python.org/issue36572 #36567: DOC: manpage directive doesn't create hyperlink https://bugs.python.org/issue36567 Most recent 15 issues waiting for review (15) ============================================= #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 #36612: Unittest document is not clear on SetUpClass calls https://bugs.python.org/issue36612 #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 #36608: Replace bundled pip and setuptools with a downloader in the en https://bugs.python.org/issue36608 #36605: make tags should also parse Modules/_io/*.c and Modules/_io/*. https://bugs.python.org/issue36605 #36602: Recursive directory list with pathlib.Path.iterdir https://bugs.python.org/issue36602 #36601: signals can be caught by any thread https://bugs.python.org/issue36601 #36600: re-enable test in nntplib https://bugs.python.org/issue36600 #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 #36593: Trace function interferes with MagicMock isinstance? https://bugs.python.org/issue36593 #36590: Add Bluetooth RFCOMM Support for Windows https://bugs.python.org/issue36590 #36589: Incorrect error handling in curses.update_lines_cols() https://bugs.python.org/issue36589 #36585: test_posix.py fails due to unsupported RWF_HIPRI https://bugs.python.org/issue36585 Top 10 most discussed issues (10) ================================= #36551: Optimize list comprehensions with preallocate size and protect https://bugs.python.org/issue36551 24 msgs #30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection https://bugs.python.org/issue30458 13 msgs #36560: test_functools leaks randomly 1 memory block https://bugs.python.org/issue36560 13 msgs #36537: except statement block incorrectly assumes end of scope(?). https://bugs.python.org/issue36537 9 msgs #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 8 msgs #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 8 msgs #36389: Add gc.enable_object_debugger(): detect corrupted Python objec https://bugs.python.org/issue36389 7 msgs #36611: Debug memory allocators: remove useless "serialno" field to re https://bugs.python.org/issue36611 7 msgs #18748: io.IOBase destructor silence I/O error on close() by default https://bugs.python.org/issue18748 6 msgs #36573: zipfile zipfile.BadZipFile: Bad CRC-32 for file '11_02_2019.pd https://bugs.python.org/issue36573 6 msgs Issues closed (73) ================== #2281: Enhanced cPython profiler with high-resolution timer https://bugs.python.org/issue2281 closed by inada.naoki #12910: urllib.quote quotes too many chars, e.g., '()' https://bugs.python.org/issue12910 closed by orsenthil #14017: Make it easy to create a new TextIOWrapper based on an existin https://bugs.python.org/issue14017 closed by ncoghlan #16712: collections.abc.Sequence should not provide __reversed__ https://bugs.python.org/issue16712 closed by inada.naoki #17396: modulefinder fails if module contains syntax error https://bugs.python.org/issue17396 closed by ncoghlan #17561: Add socket.bind_socket() convenience function https://bugs.python.org/issue17561 closed by giampaolo.rodola #19417: Bdb: add a unittest file (test.test_bdb) https://bugs.python.org/issue19417 closed by xdegaye #19476: Add a dedicated specification for module "reloading" to the la https://bugs.python.org/issue19476 closed by eric.snow #21318: sdist fails with symbolic links do non-existing files https://bugs.python.org/issue21318 closed by cheryl.sabella #25922: canceling a repair install breaks the ability to uninstall, re https://bugs.python.org/issue25922 closed by cheryl.sabella #27181: Add geometric mean to `statistics` module https://bugs.python.org/issue27181 closed by rhettinger #28351: statistics.geometric_mean can enter infinite loop for Decimal https://bugs.python.org/issue28351 closed by cheryl.sabella #28626: Tutorial: rearrange discussion of output formatting to encoura https://bugs.python.org/issue28626 closed by cheryl.sabella #29209: Remove old-deprecated ElementTree features https://bugs.python.org/issue29209 closed by serhiy.storchaka #29707: os.path.ismount() always returns false for mount --bind on sam https://bugs.python.org/issue29707 closed by christian.heimes #30134: BytesWarning is missing from the documents https://bugs.python.org/issue30134 closed by inada.naoki #30661: Support tarfile.PAX_FORMAT in shutil.make_archive https://bugs.python.org/issue30661 closed by ncoghlan #31155: Encode set, frozenset, bytearray, and iterators as json arrays https://bugs.python.org/issue31155 closed by inada.naoki #31512: Add non-elevated symlink support for dev mode Windows 10 https://bugs.python.org/issue31512 closed by steve.dower #32534: Speed-up list.insert: use memmove() https://bugs.python.org/issue32534 closed by inada.naoki #33228: Use Random.choices in tempfile https://bugs.python.org/issue33228 closed by inada.naoki #33456: site.py: by default, a virtual environment is *not* isolated f https://bugs.python.org/issue33456 closed by vinay.sajip #33461: json.loads(encoding=<not None>) does not emit deprecation warn https://bugs.python.org/issue33461 closed by inada.naoki #33722: Document builtins in mock_open https://bugs.python.org/issue33722 closed by jcrotts #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 closed by vstinner #34139: Remove stale unix datagram socket before binding https://bugs.python.org/issue34139 closed by asvetlov #34144: venv activate.bat reset codepage fails on windows 10 https://bugs.python.org/issue34144 closed by cheryl.sabella #34160: ElementTree not preserving attribute order https://bugs.python.org/issue34160 closed by scoder #34805: Explicitly specify `MyClass.__subclasses__()` returns classes https://bugs.python.org/issue34805 closed by cheryl.sabella #35376: modulefinder skips nested modules with same name as top-level https://bugs.python.org/issue35376 closed by ncoghlan #35416: Fix potential resource warnings in distutils https://bugs.python.org/issue35416 closed by inada.naoki #35488: pathlib Path.match does not behave as described https://bugs.python.org/issue35488 closed by anthony shaw #35848: readinto is not a method on io.TextIOBase https://bugs.python.org/issue35848 closed by benjamin.peterson #35906: [CVE-2019-9947] Header Injection in urllib https://bugs.python.org/issue35906 closed by gregory.p.smith #35936: Give modulefinder some much-needed updates. https://bugs.python.org/issue35936 closed by ncoghlan #36050: Why does http.client.HTTPResponse._safe_read use MAXAMOUNT https://bugs.python.org/issue36050 closed by inada.naoki #36378: Add support to load from paths to json.load https://bugs.python.org/issue36378 closed by inada.naoki #36416: bytes.rpartition bug in online documentation https://bugs.python.org/issue36416 closed by inada.naoki #36495: Out-of-bounds array reads in Python/ast.c https://bugs.python.org/issue36495 closed by levkivskyi #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 closed by Ivan.Pozdeev #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 closed by inada.naoki #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 closed by serhiy.storchaka #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 closed by serhiy.storchaka #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 closed by steve.dower #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 closed by inada.naoki #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 closed by spaceman_spiff #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 closed by Manjusaka #36539: Distutils VC 6.0 Errors When Using mingw-w64 GCC https://bugs.python.org/issue36539 closed by danyeaw #36544: cannot import hashlib when openssl is missing https://bugs.python.org/issue36544 closed by xdegaye #36547: bedevere is not working https://bugs.python.org/issue36547 closed by xtreak #36549: str.capitalize should titlecase the first character not upperc https://bugs.python.org/issue36549 closed by steve.dower #36554: unittest.TestCase: "subTest" cannot be used together with "deb https://bugs.python.org/issue36554 closed by dmaurer #36555: PEP484 @overload vs. str/bytes https://bugs.python.org/issue36555 closed by levkivskyi #36559: "import random" should import hashlib on demand (nor load Open https://bugs.python.org/issue36559 closed by rhettinger #36561: Python argparse doesn't work in the presence of my custom modu https://bugs.python.org/issue36561 closed by xtreak #36562: Can't call a method from a module built in Python C API https://bugs.python.org/issue36562 closed by jjppof #36565: Reference hunting (python3 -m test -R 3:3) doesn't work if the https://bugs.python.org/issue36565 closed by vstinner #36566: Support password masking in getpass.getpass() https://bugs.python.org/issue36566 closed by cheryl.sabella #36570: ftplib timeouts for misconfigured server https://bugs.python.org/issue36570 closed by giampaolo.rodola #36571: Lib/smtplib.py have some pep8 issues https://bugs.python.org/issue36571 closed by Marcin Niemira #36574: Error with self in python https://bugs.python.org/issue36574 closed by steven.daprano #36575: Use _PyTime_GetPerfCounter() in lsprof https://bugs.python.org/issue36575 closed by inada.naoki #36577: setup doesn't report missing _ssl and _hashlib https://bugs.python.org/issue36577 closed by christian.heimes #36578: multiprocessing pool + subprocess ValueError: empty range for https://bugs.python.org/issue36578 closed by SilentGhost #36579: test_venv: test_with_pip() hangs on PPC64 AIX 3.x https://bugs.python.org/issue36579 closed by vstinner #36584: cython nametuple TypeError https://bugs.python.org/issue36584 closed by serhiy.storchaka #36588: change sys.platform() to just "aix" for AIX https://bugs.python.org/issue36588 closed by vstinner #36591: Should be a typing.UserNamedTuple https://bugs.python.org/issue36591 closed by levkivskyi #36592: is behave different for integers in 3.6 and 3.7 https://bugs.python.org/issue36592 closed by eric.smith #36597: Travis CI: doctest failure https://bugs.python.org/issue36597 closed by inada.naoki #36598: mock side_effect should be checked for iterable not callable https://bugs.python.org/issue36598 closed by xtreak #36599: doctest document says dict order is unstable https://bugs.python.org/issue36599 closed by inada.naoki #36604: Add recipe to itertools https://bugs.python.org/issue36604 closed by rhettinger

1 0

PEP-582 and multiple Python installations
by Calvin Spealman 12 Apr '19

12 Apr '19

(I originally posted this to python-ideas, where I was told none of this PEP's authors subscribe so probably no one will see it there, so I'm posting it here to raise the issue where it can get seen and hopefully discussed) While the PEP does show the version number as part of the path to the actual packages, implying support for multiple versions, this doesn't seem to be spelled out in the actual text. Presumably __pypackages__/3.8/ might sit beside __pypackages__/3.9/, etc. to keep future versions capable of installing packages for each version, the way virtualenv today is bound to one version of Python. I'd like to raise a potential edge case that might be a problem, and likely an increasingly common one: users with multiple installations of the *same* version of Python. This is actually a common setup for Windows users who use WSL, Microsoft's Linux-on-Windows solution, as you could have both the Windows and Linux builds of a given Python version installed on the same machine. The currently implied support for multiple versions would not be able to separate these and could create problems if users pip install a Windows binary package through Powershell and then try to run a script in Bash from the same directory, causing the Linux version of Python to try to use Windows python packages. I'm not actually sure what the solution here is. Mostly I wanted to raise the concern, because I'm very keen on WSL being a great entry path for new developers and I want to make that a better experience, not a more confusing one. Maybe that version number could include some other unique identify, maybe based on Python's own executable. A hash maybe? I don't know if anything like that already exists to uniquely identify a Python build or installation. -- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma(a)redhat.com M: +1.336.210.5107 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

6 5

(no subject)
by Robert Okadar 10 Apr '19

10 Apr '19

Hi community, I have developed a tkinter GUI component, Python v3.7. It runs very well in Linux but seeing a huge performance impact in Windows 10. While in Linux an almost real-time performance is achieved, in Windows it is slow to an unusable level. The code is somewhat stripped down from the original, but the performance difference is the same anyway. The columns can be resized by clicking on the column border and dragging it. Resizing works only for the top row (but it resizes the entire column). In this demo, all bindings are avoided to exclude influence on the component performance and thus not included. If you resize the window (i.e., if you maximize it), you must call the function table.fit() from IDLE shell. Does anyone know where is this huge difference in performance coming from? Can anything be done about it? All the best, -- Robert Okadar IT Consultant Schedule an *online meeting <https://calendly.com/aranea-network/60min>* with me! Visit *aranea-mreze.hr* <http://aranea-mreze.hr> or call * +385 91 300 8887*

4 4

Need help to fix HTTP Header Injection vulnerability
by Victor Stinner 10 Apr '19

10 Apr '19

Hi, In May 2017, user "Orange" found a vulnerability in the urllib fix for CVE-2016-5699 (HTTP Header Injection vulnerability): https://bugs.python.org/issue30458 It allows to inject arbitrary HTTP headers. Copy of their message: """ Hi, the patch in CVE-2016-5699 can be broke by an addition space. http://www.cvedetails.com/cve/CVE-2016-5699/ https://hg.python.org/cpython/rev/bf3e1c9b80e9 https://hg.python.org/cpython/rev/1c45047c5102 import urllib, urllib2 urllib.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211') urllib2.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211') """ Last month, the same bug has been rediscovered by user "ragdoll.guo": https://bugs.python.org/issue36276 Almost one year after the bug has been reported, no one came with a solution. I'm not comfortable with having known security issues impacting HTTP. Can someone please have a look at the issue and try to write a change to fix the issue? According to Karthikeyan Singaravelan, the Go language fixed a similar issue in Go 1.12: throw an error if the URL contains any control character. If we decide that the issue is not a security issue, we should document the behavior properly and close the issue. See also this related issue: "urlopen URL with unescaped space" https://bugs.python.org/issue14826 Victor -- Night gathers, and now my watch begins. It shall not end until my death.

5 8

New Python Initialization API
by Victor Stinner 09 Apr '19

09 Apr '19

Hi, I would like to add a new C API to initialize Python. I would like your opinion on the whole API before making it public. The code is already implemented. Doc of the new API: https://pythondev.readthedocs.io/init_config.html To make the API public, _PyWstrList, _PyInitError, _PyPreConfig, _PyCoreConfig and related functions should be made public. By the way, I would suggest to rename "_PyCoreConfig" to just "PyConfig" :-) I don't think that "core init" vs "main init" is really relevant: more about that below. Let's start with two examples using the new API. Example of simple initialization to enable isolated mode: _PyCoreConfig config = _PyCoreConfig_INIT; config.isolated = 1; _PyInitError err = _Py_InitializeFromConfig(&config); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); Example using the pre-initialization to enable the UTF-8 Mode (and use the "legacy" Py_Initialize() function): _PyPreConfig preconfig = _PyPreConfig_INIT; preconfig.utf8_mode = 1; _PyInitError err = _Py_PreInitialize(&preconfig); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } /* at this point, Python will only speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Since November 2017, I'm refactoring the Python Initialization code to cleanup the code and prepare a new ("better") API to configure Python Initialization. I just fixed the last issues that Nick Coghlan asked me to fix (add a pre-initialization step: done, fix mojibake: done). My work is inspired by Nick Coghlan's PEP 432, but it is not implementing it directly. I had other motivations than Nick even if we are somehow going towards the same direction. Nick wants to get a half-initialized Python ("core init"), configure Python using the Python API and Python objects, and then finish the implementation ("main init"). I chose a different approach: put *everything* into a single C structure (_PyCoreConfig) using C types. Using the structure, you should be able to do what Nick wanted to do, but with C rather than Python. Nick: please tell me if I'm wrong :-) This work is also connected to Eric Snow's work on sub-interpreters (PEP 554) and moving global variables into structures. For example, I'm using his _PyRuntime structure to store a new "preconfig" state (pre-initialization configuration, more about that below). In November 2017, when I started to work on the Python Initialization (bpo-32030), I identified the following problems: * Many parts of the code were interdependent * Code executed early in Py_Main() used the Python API before the Python API was fully initialized. Like code parsing -W command line option which used PyUnicode_FromWideChar() and PyList_Append(). * Error handling used Py_FatalError() which didn't let the caller to decide how to handle the error. Moreover, exit() was used to exit Python, whereas libpython shouldn't do that: a library should not exit the whole process! (imagine when Python is embedded inside an application) One year and a half later, I implemented the following solutions: * Py_Main() and Py_Initialize() code has been reorganized to respect priorities between global configuration variables (ex: Py_IgnoreEnvironmentFlag), environment variables (ex: PYTHONPATH), command line arguments (ex: -X utf8), configuration files (ex: pyenv.cfg), and the new _PyPreConfig and _PyCoreConfig structures which store the whole configuration. * Python Initialization no longer uses the Python API but only C types like wchar_t* strings, a new _PyWstrList structure and PyMem_RawMalloc() memory allocator (PyMem_Malloc() is no longer used during init). * The code has been modified to use a new _PyInitError structure. The caller of the top function gets control to cleanup everything before handling the error (display a fatal error message or simply exit Python). The new _PyCoreConfig structure has the top-priority and provides a single structure for all configuration parameters. It becomes possible to override the code computing the "path configuration" like sys.path to fully control where Python looks to import modules. It becomes possible to use an empty list of paths to only allow builtin modules. A new "pre-initialization" steps is responsible to configure the bare minimum before the Python initialization: memory allocators and encodings (LC_CTYPE locale and the UTF-8 mode). The LC_CTYPE is no longer coerced and the UTF-8 Mode is no longer enabled automatically depending on the user configuration to prevent mojibake. Previously, calling Py_DecodeLocale() to get a Unicode wchar_t* string from a bytes wchar* string created mojibake when called before Py_Initialize() if the LC_CTYPE locale was coerced and/or if the UTF-8 Mode was enabled. The pre-initialization step ensures that the encodings and memory allocators are well defined *before* Py_Initialize() is called. Since the new API is currently private, I didn't document it in Python. Moreover, the code changed a lot last year :-) But it should now be way more stable. I started to document it in a separated webpage: https://pythondev.readthedocs.io/init_config.html The plan is to put it in the Python documentation once it becomes public. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

8 27

Deprecating "instance method" class
by Jeroen Demeyer 08 Apr '19

08 Apr '19

During my investigations related to low-level function/method classes, I came across the "instance method" class. There is a C API for it: https://docs.python.org/3.7/c-api/method.html However, it's not used/exposed anywhere in CPython, except as _testcapi.instancemethod (for testing its functionality) This class was discussed at https://mail.python.org/pipermail/python-3000/2007-December/011456.html and implemented in https://bugs.python.org/issue1587 Reading that old thread, there are use cases presented related to classic classes, wrapping Kogut (http://kokogut.sourceforge.net/kogut.html) objects and Pyrex. But classic classes no longer exist and the latter two use cases aren't actually needed if you read the thread to the end. So there are no surviving use cases from that thread. Does anybody know actual use cases or any code in the wild using it? To me, the fact that it's only exposed in the C API is a good sign that it's not really useful. So, should we deprecate the instance method class? Jeroen.

7 21

bedevere pipelines hang on github
by Xavier de Gaye 08 Apr '19

08 Apr '19

bedevere/issue-number and bedevere/news are not triggered for some reason at https://github.com/python/cpython/pull/12708 and hang forever with "Expected — Waiting for status to be reported ". Xavier

3 3

Summary of Python tracker Issues
by Python tracker 05 Apr '19

05 Apr '19

ACTIVITY SUMMARY (2019-03-29 - 2019-04-05) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7056 ( +8) closed 41231 (+55) total 48287 (+63) Open issues with patches: 2816 Issues opened (44) ================== #36260: Cpython/Lib vulnerability found and request a patch submission https://bugs.python.org/issue36260 reopened by krnick #36474: RecursionError resets trace function set via sys.settrace https://bugs.python.org/issue36474 opened by blueyed #36475: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle https://bugs.python.org/issue36475 opened by eric.snow #36476: Runtime finalization assumes all other threads have exited. https://bugs.python.org/issue36476 opened by eric.snow #36478: backport of pickle fixes to Python 3.5.7 uses C99 for loops https://bugs.python.org/issue36478 opened by Anthony Sottile #36479: Exit threads when interpreter is finalizing rather than runtim https://bugs.python.org/issue36479 opened by eric.snow #36481: telnetlib process_rawq() callback https://bugs.python.org/issue36481 opened by Quanthir #36484: Can't reorder TLS 1.3 ciphersuites https://bugs.python.org/issue36484 opened by Eman Alashwali #36485: Establish a uniform way to clear all caches in a given module https://bugs.python.org/issue36485 opened by serhiy.storchaka #36486: Bugs and inconsistencies in unicodedata https://bugs.python.org/issue36486 opened by dscorbett #36487: Make C-API docs clear about what the "main" interpreter is https://bugs.python.org/issue36487 opened by nanjekyejoannah #36488: os.sendfile() on BSD and macOS does not return bytes sent on E https://bugs.python.org/issue36488 opened by giampaolo.rodola #36489: add filename_extension_map and/or content-types_map dict(s) to https://bugs.python.org/issue36489 opened by Daniel Black #36490: Modernize function signature format in Archiving section of sh https://bugs.python.org/issue36490 opened by CAM-Gerlach #36494: bdb.Bdb.set_trace: should set f_trace_lines = True https://bugs.python.org/issue36494 opened by blueyed #36495: Out-of-bounds array reads in Python/ast.c https://bugs.python.org/issue36495 opened by blarsen #36497: Undocumented behavior in csv.Sniffer (preferred delimiters) https://bugs.python.org/issue36497 opened by thomas #36500: Add "regen-*" equivalent projects for Windows builds https://bugs.python.org/issue36500 opened by anthony shaw #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 opened by Ivan.Pozdeev #36502: The behavior of str.isspace() for U+00A0 and U+202F is differe https://bugs.python.org/issue36502 opened by Jun #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 opened by Michael.Felt #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 opened by ZackerySpytz #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 opened by bigbigliang #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 opened by vstinner #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 opened by Paul Monson #36511: Add Windows ARM32 buildbot https://bugs.python.org/issue36511 opened by Paul Monson #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 opened by stefanhoelzl #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 opened by Paul Monson #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 opened by doko #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 opened by gjj2828 #36517: typing.NamedTuple does not support mixins https://bugs.python.org/issue36517 opened by rectalogic #36518: Avoid conflicts when pass arbitrary keyword arguments to Pytho https://bugs.python.org/issue36518 opened by serhiy.storchaka #36519: Blake2b/s implementations have minor GIL issues https://bugs.python.org/issue36519 opened by gwk #36520: Email header folded incorrectly https://bugs.python.org/issue36520 opened by Jonathan Horn #36521: Consider removing docstrings from co_consts in code objects https://bugs.python.org/issue36521 opened by rhettinger #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 opened by Marcin Niemira #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 opened by AMDmi3 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 opened by xtreak #36529: Python from WindowsStore: can't install package using "-m pip" https://bugs.python.org/issue36529 opened by Ilya Kazakevich #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 opened by eelizondo #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 opened by spaceman_spiff #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 opened by gregory.p.smith #36534: tarfile: handling Windows (path) illegal characters in archive https://bugs.python.org/issue36534 opened by CristiFati #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 opened by Manjusaka Most recent 15 issues with no replies (15) ========================================== #36535: Windows build failure when use the code from the GitHub master https://bugs.python.org/issue36535 #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 #36529: Python from WindowsStore: can't install package using "-m pip" https://bugs.python.org/issue36529 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 #36520: Email header folded incorrectly https://bugs.python.org/issue36520 #36517: typing.NamedTuple does not support mixins https://bugs.python.org/issue36517 #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 #36511: Add Windows ARM32 buildbot https://bugs.python.org/issue36511 #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 Most recent 15 issues waiting for review (15) ============================================= #36532: Example of logging.formatter with new str.format style https://bugs.python.org/issue36532 #36531: PyType_FromSpec wrong behavior with multiple Py_tp_members https://bugs.python.org/issue36531 #36528: Remove duplicate tests in Lib/tests/re_tests.py https://bugs.python.org/issue36528 #36527: unused parameter warnings in Include/object.h (affecting build https://bugs.python.org/issue36527 #36523: missing docs for IOBase writelines https://bugs.python.org/issue36523 #36518: Avoid conflicts when pass arbitrary keyword arguments to Pytho https://bugs.python.org/issue36518 #36516: Python Launcher can not recognize pyw file as Python GUI Scrip https://bugs.python.org/issue36516 #36515: unaligned memory access in the _sha3 extension https://bugs.python.org/issue36515 #36513: Add support for building arm32 nuget package https://bugs.python.org/issue36513 #36512: future_factory argument for Thread/ProcessPoolExecutor https://bugs.python.org/issue36512 #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 #36508: python-config --ldflags must not contain LINKFORSHARED ("-Xlin https://bugs.python.org/issue36508 #36504: Signed integer overflow in _ctypes.c's PyCArrayType_new() https://bugs.python.org/issue36504 #36503: remove references to aix3 and aix4 in \*.py https://bugs.python.org/issue36503 #36501: Remove POSIX.1e ACLs in tests that rely on default permissions https://bugs.python.org/issue36501 Top 10 most discussed issues (10) ================================= #36485: Establish a uniform way to clear all caches in a given module https://bugs.python.org/issue36485 13 msgs #36466: Adding a way to strip annotations from compiled bytecode https://bugs.python.org/issue36466 12 msgs #36469: Stuck during interpreter exit, attempting to take the GIL https://bugs.python.org/issue36469 10 msgs #36506: [security] CVE-2019-10268: An arbitrary execution vulnerabilit https://bugs.python.org/issue36506 8 msgs #6721: Locks in the standard library should be sanitized on fork https://bugs.python.org/issue6721 7 msgs #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 6 msgs #35866: concurrent.futures deadlock https://bugs.python.org/issue35866 5 msgs #36384: ipaddress Should not reject IPv4 addresses with leading zeroes https://bugs.python.org/issue36384 5 msgs #30661: Support tarfile.PAX_FORMAT in shutil.make_archive https://bugs.python.org/issue30661 4 msgs #35224: PEP 572: Assignment Expressions https://bugs.python.org/issue35224 4 msgs Issues closed (53) ================== #17110: sys.argv docs should explaining how to handle encoding issues https://bugs.python.org/issue17110 closed by inada.naoki #20844: SyntaxError: encoding problem: iso-8859-1 on Windows https://bugs.python.org/issue20844 closed by inada.naoki #21269: Provide args and kwargs attributes on mock call objects https://bugs.python.org/issue21269 closed by xtreak #22831: Use "with" to avoid possible fd leaks https://bugs.python.org/issue22831 closed by serhiy.storchaka #24214: UTF-8 incremental decoder doesn't support surrogatepass correc https://bugs.python.org/issue24214 closed by serhiy.storchaka #25451: tkinter: PhotoImage transparency methods https://bugs.python.org/issue25451 closed by serhiy.storchaka #29202: Improve dict iteration https://bugs.python.org/issue29202 closed by inada.naoki #31182: Suggested Enhancements to zipfile & tarfile command line inter https://bugs.python.org/issue31182 closed by brett.cannon #32413: Document that locals() may return globals() https://bugs.python.org/issue32413 closed by brett.cannon #32531: gdb.execute can not put string value. https://bugs.python.org/issue32531 closed by berker.peksag #32538: Multiprocessing Manager on 3D list - no change of the list pos https://bugs.python.org/issue32538 closed by berker.peksag #33261: inspect.isgeneratorfunction fails on hand-created methods https://bugs.python.org/issue33261 closed by petr.viktorin #34430: Symmetrical chaining futures in asyncio.future.wrap_future https://bugs.python.org/issue34430 closed by huji #35272: sqlite3 get the connected database url https://bugs.python.org/issue35272 closed by berker.peksag #35403: support application/wasm in mimetypes and http.server https://bugs.python.org/issue35403 closed by martin.panter #35838: ConfigParser: document optionxform must be idempotent https://bugs.python.org/issue35838 closed by inada.naoki #36010: Please provide a .zip Windows release of Python that is not cr https://bugs.python.org/issue36010 closed by steve.dower #36026: Different error message when sys.settrace is used https://bugs.python.org/issue36026 closed by inada.naoki #36085: Enable better DLL resolution https://bugs.python.org/issue36085 closed by steve.dower #36157: Document PyInterpreterState_Main(). https://bugs.python.org/issue36157 closed by eric.snow #36293: Nonblocking read sys.stdin raises error https://bugs.python.org/issue36293 closed by martin.panter #36322: Argument typo in dbm.ndbm.open https://bugs.python.org/issue36322 closed by brett.cannon #36377: Python 'datastructures.html' docs page needs improvement becau https://bugs.python.org/issue36377 closed by rhettinger #36404: Document PendingDeprecationWarning is not so useful. https://bugs.python.org/issue36404 closed by inada.naoki #36426: exec() issue when used inside function https://bugs.python.org/issue36426 closed by ncoghlan #36434: Zipfile breaks if signalled during write() https://bugs.python.org/issue36434 closed by serhiy.storchaka #36440: more helpful diagnostics for parser module https://bugs.python.org/issue36440 closed by pablogsal #36442: Different ValueError for the same operation in List and Tuple https://bugs.python.org/issue36442 closed by serhiy.storchaka #36445: bus error in test_gil test on armhf running with 64bit kernel https://bugs.python.org/issue36445 closed by doko #36448: Message "You will need to rebuild pythoncore to see the change https://bugs.python.org/issue36448 closed by steve.dower #36468: Treeview: wrong color change https://bugs.python.org/issue36468 closed by ned.deily #36472: Some old PR with CLA not signed https://bugs.python.org/issue36472 closed by brett.cannon #36473: dictkeysobject: Add maximum iteration check for .values() and https://bugs.python.org/issue36473 closed by inada.naoki #36477: Subinterpreters are not finalized during runtime finalization. https://bugs.python.org/issue36477 closed by eric.snow #36480: .strip() unexpected output on Windows https://bugs.python.org/issue36480 closed by eric.smith #36482: let struct's internal cache use FIFO policy https://bugs.python.org/issue36482 closed by rhettinger #36483: Missing line in documentation example https://bugs.python.org/issue36483 closed by martin.panter #36491: sum function's start optional parameter documented in help but https://bugs.python.org/issue36491 closed by rhettinger #36492: Deprecate passing some conflicting arguments by keyword https://bugs.python.org/issue36492 closed by serhiy.storchaka #36493: Add math.midpoint(a,b) function https://bugs.python.org/issue36493 closed by scoder #36496: Local variables can be used uninitialized in _PyPreConfig_Read https://bugs.python.org/issue36496 closed by vstinner #36498: combining dict comprehensing and lists lead to IndexError https://bugs.python.org/issue36498 closed by SilentGhost #36499: unpickling of a datetime object in 3.5 fails when pickled with https://bugs.python.org/issue36499 closed by josh.r #36505: PYTHON-CAN with vector https://bugs.python.org/issue36505 closed by SilentGhost #36507: frozenset type breaks ZFC https://bugs.python.org/issue36507 closed by rhettinger #36510: Regular Expression Dot-Star patter matching - re- text skippi https://bugs.python.org/issue36510 closed by SilentGhost #36514: -m switch revisited https://bugs.python.org/issue36514 closed by ronaldoussoren #36522: http/client.py does not print duplicate header values in debug https://bugs.python.org/issue36522 closed by serhiy.storchaka #36524: identity operator https://bugs.python.org/issue36524 closed by SilentGhost #36525: Deprecate instancemethod https://bugs.python.org/issue36525 closed by christian.heimes #36526: python crash when loading some .pyc file https://bugs.python.org/issue36526 closed by serhiy.storchaka #36530: Document codecs decode_encode() and encode_decode() APIs https://bugs.python.org/issue36530 closed by gregory.p.smith #36536: is there a python implementation of the cpython commandline in https://bugs.python.org/issue36536 closed by larry

1 0

how to rerun the job “Azure Pipelines PR”?
by Xin, Peixing 02 Apr '19

02 Apr '19

Hi, Experts: Anyone can tell how to rerun the job “Azure Pipelines PR” for my PR? Sometimes my PR failed but this is caused by externals. The next day this external issue was fixed then I might want to rerun this specific job on my PR to get the new result. How can I reach this? [cid:image001.png@01D4E97B.F67B1E20] Thanks, Peixing

3 2

Strange umask(?)/st_mode issue
by Steve Dower 01 Apr '19

01 Apr '19

Hi I'm trying to track down the cause of the failed Linux tests on this build (it's also been happening on random PR builds for the last day or two, but this is the first I've seen it happen on already merged code): https://dev.azure.com/Python/cpython/_build/results?buildId=40189 One of the failed tests is this one from test_tarfile.py (and I think this is representative of the others): def test_file_mode(self): # Test for issue #8464: Create files with correct # permissions. if os.path.exists(tmpname): support.unlink(tmpname) original_umask = os.umask(0o022) try: tar = tarfile.open(tmpname, self.mode) tar.close() mode = os.stat(tmpname).st_mode & 0o777 self.assertEqual(mode, 0o644, "wrong file permissions") finally: os.umask(original_umask) It's failing because the mode of the new TAR file is 0o666 rather than 0o644. I'd guess that this means the umask() call is not having effect? But honestly this is beyond me - I don't have any idea what would cause Python to ignore that, and I don't see anything in the commit history that could be at fault. I'm hoping someone can tell me that it's definitely a platform configuration issue or it's definitely interference between two of our tests (which run in random order, which would explain the intermittent nature of this, but I haven't spotted any tests that seem likely either). To see the other failed PR builds, the full list is at https://dev.azure.com/Python/cpython/_build?definitionId=9 and most of the ones from today have failed because of whatever is causing it. Any help? Thanks, Steve

6 14

Removing PendingDeprecationWarning
by Inada Naoki 01 Apr '19

01 Apr '19

Hi, all. I'm thinking about removing PendingDeprecationWarning. (previous discussion: https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038) It was added "not be printed by default" version of DeprecationWarning. But DeprecationWarning is not printed by default now. We use PendingDeprecationWarning for N-2 release, and change it to DeprecationWarning for N-1 release. But this workflow seems not worth enough for now. I want to stop using PendingDeprecationWarning for new deprecation. More aggressively, I want to remove PendingDeprecationWarning class, and `PendingDeprecationWarning = DeprecationWarning` for backward compatibility. How do you think? May I do it in Python 3.8? -- Inada Naoki <songofacandy(a)gmail.com>

10 34