Mailman 3 November 2015 - Python-Dev

PEP 448 review
by Guido van Rossum 29 Mar '23

29 Mar '23

I'm back, I've re-read the PEP, and I've re-read the long thread with "(no subject)". I think Georg Brandl nailed it: """ *I like the "sequence and dict flattening" part of the PEP, mostly because itis consistent and should be easy to understand, but the comprehension syntaxenhancements seem to be bad for readability and "comprehending" what the codedoes.The call syntax part is a mixed bag on the one hand it is nice to be consistent with the extended possibilities in literals (flattening), but on the other hand there would be small but annoying inconsistencies anyways (e.g. the duplicate kwarg case above).* """ Greg Ewing followed up explaining that the inconsistency between dict flattening and call syntax is inherent in the pre-existing different rules for dicts vs. keyword args: {'a':1, 'a':2} results in {'a':2}, while f(a=1, a=2) is an error. (This form is a SyntaxError; the dynamic case f(a=1, **{'a': 1}) is a TypeError.) For me, allowing f(*a, *b) and f(**d, **e) and all the other combinations for function calls proposed by the PEP is an easy +1 -- it's a straightforward extension of the existing pattern, and anybody who knows what f(x, *a) does will understand f(x, *a, y, *b). Guessing what f(**d, **e) means shouldn't be hard either. Understanding the edge case for duplicate keys with f(**d, **e) is a little harder, but the error messages are pretty clear, and it is not a new edge case. The sequence and dict flattening syntax proposals are also clean and logical -- we already have *-unpacking on the receiving side, so allowing *x in tuple expressions reads pretty naturally (and the similarity with *a in argument lists certainly helps). From here, having [a, *x, b, *y] is also natural, and then the extension to other displays is natural: {a, *x, b, *y} and {a:1, **d, b:2, **e}. This, too, gets a +1 from me. So that leaves comprehensions. IIRC, during the development of the patch we realized that f(*x for x in xs) is sufficiently ambiguous that we decided to disallow it -- note that f(x for x in xs) is already somewhat of a special case because an argument can only be a "bare" generator expression if it is the only argument. The same reasoning doesn't apply (in that form) to list, set and dict comprehensions -- while f(x for x in xs) is identical in meaning to f((x for x in xs)), [x for x in xs] is NOT the same as [(x for x in xs)] (that's a list of one element, and the element is a generator expression). The basic premise of this part of the proposal is that if you have a few iterables, the new proposal (without comprehensions) lets you create a list or generator expression that iterates over all of them, essentially flattening them: >>> xs = [1, 2, 3] >>> ys = ['abc', 'def'] >>> zs = [99] >>> [*xs, *ys, *zs] [1, 2, 3, 'abc', 'def', 99] >>> But now suppose you have a list of iterables: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xss[0], *xss[1], *xss[2]] [1, 2, 3, 'abc', 'def', 99] >>> Wouldn't it be nice if you could write the latter using a comprehension? >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs for xs in xss] [1, 2, 3, 'abc', 'def', 99] >>> This is somewhat seductive, and the following is even nicer: the *xs position may be an expression, e.g.: >>> xss = [[1, 2, 3], ['abc', 'def'], [99]] >>> [*xs[:2] for xs in xss] [1, 2, 'abc', 'def', 99] >>> On the other hand, I had to explore the possibilities here by experimenting in the interpreter, and I discovered some odd edge cases (e.g. you can parenthesize the starred expression, but that seems a syntactic accident). All in all I am personally +0 on the comprehension part of the PEP, and I like that it provides a way to "flatten" a sequence of sequences, but I think very few people in the thread have supported this part. Therefore I would like to ask Neil to update the PEP and the patch to take out the comprehension part, so that the two "easy wins" can make it into Python 3.5 (basically, I am accepting two-thirds of the PEP :-). There is some time yet until alpha 2. I would also like code reviewers (Benjamin?) to start reviewing the patch <http://bugs.python.org/issue2292>, taking into account that the comprehension part needs to be removed. -- --Guido van Rossum (python.org/~guido)

7 17

PEP 1, PEP Purpose and Guidelines
by barry＠zope.com 18 May '21

18 May '21

It has been a while since I posted a copy of PEP 1 to the mailing lists and newsgroups. I've recently done some updating of a few sections, so in the interest of gaining wider community participation in the Python development process, I'm posting the latest revision of PEP 1 here. A version of the PEP is always available on-line at http://www.python.org/peps/pep-0001.html Enjoy, -Barry -------------------- snip snip -------------------- PEP: 1 Title: PEP Purpose and Guidelines Version: $Revision: 1.36 $ Last-Modified: $Date: 2002/07/29 18:34:59 $ Author: Barry A. Warsaw, Jeremy Hylton Status: Active Type: Informational Created: 13-Jun-2000 Post-History: 21-Mar-2001, 29-Jul-2002 What is a PEP? PEP stands for Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python. The PEP should provide a concise technical specification of the feature and a rationale for the feature. We intend PEPs to be the primary mechanisms for proposing new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and documenting dissenting opinions. Because the PEPs are maintained as plain text files under CVS control, their revision history is the historical record of the feature proposal[1]. Kinds of PEPs There are two kinds of PEPs. A standards track PEP describes a new feature or implementation for Python. An informational PEP describes a Python design issue, or provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementors are free to ignore informational PEPs or follow their advice. PEP Work Flow The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers for each PEP and changes its status. The PEP process begins with a new idea for Python. It is highly recommended that a single PEP contain a single key proposal or new idea. The more focussed the PEP, the more successfully it tends to be. The PEP editor reserves the right to reject PEP proposals if they appear too unfocussed or too broad. If in doubt, split your PEP into several well-focussed ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the appropriate forums, and attempts to build community consensus around the idea. The PEP champion (a.k.a. Author) should first attempt to ascertain whether the idea is PEP-able. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a patch submission to the SourceForge patch manager[2] or feature request tracker[3]. The PEP champion then emails the PEP editor <peps(a)python.org> with a proposed title and a rough, but fleshed out, draft of the PEP. This draft must be written in PEP style as described below. If the PEP editor approves, he will assign the PEP a number, label it as standards track or informational, give it status 'draft', and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or not in keeping with the Python philosophy. The BDFL (Benevolent Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbitrator of the draft's PEP-ability. If a pre-PEP is rejected, the author may elect to take the pre-PEP to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org mailing list) to help flesh it out, gain feedback and consensus from the community at large, and improve the PEP for re-submission. The author of the PEP is then responsible for posting the PEP to the community forums, and marshaling community support for it. As updates are necessary, the PEP author can check in new versions if they have CVS commit permissions, or can email new PEP versions to the PEP editor for committing. Standards track PEPs consists of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted before a reference implementation is begun, unless a reference implementation will aid people in studying the PEP. Standards Track PEPs must include an implementation - in the form of code, patch, or URL to same - before it can be considered Final. PEP authors are responsible for collecting community feedback on a PEP before submitting it for review. A PEP that has not been discussed on python-list(a)python.org and/or python-dev(a)python.org will not be accepted. However, wherever possible, long open-ended discussions on public mailing lists should be avoided. Strategies to keep the discussions efficient include, setting up a separate SIG mailing list for the topic, having the PEP author accept private comments in the early design phases, etc. PEP authors should use their discretion here. Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. Once a PEP has been accepted, the reference implementation must be completed. When the reference implementation is complete and accepted by the BDFL, the status will be changed to `Final.' A PEP can also be assigned status `Deferred.' The PEP author or editor can assign the PEP this status when no progress is being made on the PEP. Once a PEP is deferred, the PEP editor can re-assign it to draft status. A PEP can also be `Rejected'. Perhaps after all is said and done it was not a good idea. It is still important to have a record of this fact. PEPs can also be replaced by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of an API can replace version 1. PEP work flow is as follows: Draft -> Accepted -> Final -> Replaced ^ +----> Rejected v Deferred Some informational PEPs may also have a status of `Active' if they are never meant to be completed. E.g. PEP 1. What belongs in a successful PEP? Each PEP should have the following parts: 1. Preamble -- RFC822 style headers containing meta-data about the PEP, including the PEP number, a short descriptive title (limited to a maximum of 44 characters), the names, and optionally the contact info for each author, etc. 2. Abstract -- a short (~200 word) description of the technical issue being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly labelled as placed in the public domain (see this PEP as an example) or licensed under the Open Publication License[4]. 4. Specification -- The technical specification should describe the syntax and semantics of any new language feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Python platforms (CPython, JPython, Python .NET). 5. Motivation -- The motivation is critical for PEPs that want to change the Python language. It should clearly explain why the existing language specification is inadequate to address the problem that the PEP solves. PEP submissions without sufficient motivation may be rejected outright. 6. Rationale -- The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale should provide evidence of consensus within the community and discuss important objections or concerns raised during discussion. 7. Backwards Compatibility -- All PEPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The PEP must explain how the author proposes to deal with these incompatibilities. PEP submissions without a sufficient backwards compatibility treatise may be rejected outright. 8. Reference Implementation -- The reference implementation must be completed before any PEP is given status 'Final,' but it need not be completed before the PEP is accepted. It is better to finish the specification and rationale first and reach consensus on it before writing code. The final implementation must include test code and documentation appropriate for either the Python language reference or the standard library reference. PEP Template PEPs are written in plain ASCII text, and should adhere to a rigid style. There is a Python script that parses this style and converts the plain text PEP to HTML for viewing on the web[5]. PEP 9 contains a boilerplate[7] template you can use to get started writing your PEP. Each PEP must begin with an RFC822 style header preamble. The headers must appear in the following order. Headers marked with `*' are optional and are described below. All other headers are required. PEP: <pep number> Title: <pep title> Version: <cvs version string> Last-Modified: <cvs date string> Author: <list of authors' real names and optionally, email addrs> * Discussions-To: <email address> Status: <Draft | Active | Accepted | Deferred | Final | Replaced> Type: <Informational | Standards Track> * Requires: <pep numbers> Created: <date created on, in dd-mmm-yyyy format> * Python-Version: <version number> Post-History: <dates of postings to python-list and python-dev> * Replaces: <pep number> * Replaced-By: <pep number> The Author: header lists the names and optionally, the email addresses of all the authors/owners of the PEP. The format of the author entry should be address(a)dom.ain (Random J. User) if the email address is included, and just Random J. User if the address is not given. If there are multiple authors, each should be on a separate line following RFC 822 continuation line conventions. Note that personal email addresses in PEPs will be obscured as a defense against spam harvesters. Standards track PEPs must have a Python-Version: header which indicates the version of Python that the feature will be released with. Informational PEPs do not need a Python-Version: header. While a PEP is in private discussions (usually during the initial Draft phase), a Discussions-To: header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To: header is necessary if the PEP is being discussed privately with the author, or on the python-list or python-dev email mailing lists. Note that email addresses in the Discussions-To: header will not be obscured. Created: records the date that the PEP was assigned a number, while Post-History: is used to record the dates of when new versions of the PEP are posted to python-list and/or python-dev. Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001. PEPs may have a Requires: header, indicating the PEP numbers that this PEP depends on. PEPs may also have a Replaced-By: header indicating that a PEP has been rendered obsolete by a later document; the value is the number of the PEP that replaces the current document. The newer PEP must have a Replaces: header containing the number of the PEP that it rendered obsolete. PEP Formatting Requirements PEP headings must begin in column zero and the initial letter of each word must be capitalized as in book titles. Acronyms should be in all capitals. The body of each section must be indented 4 spaces. Code samples inside body sections should be indented a further 4 spaces, and other indentation can be used as required to make the text readable. You must use two blank lines between the last line of a section's body and the next section heading. You must adhere to the Emacs convention of adding two spaces at the end of every sentence. You should fill your paragraphs to column 70, but under no circumstances should your lines extend past column 79. If your code samples spill over column 79, you should rewrite them. Tab characters must never appear in the document at all. A PEP should include the standard Emacs stanza included by example at the bottom of this PEP. A PEP must contain a Copyright section, and it is strongly recommended to put the PEP in the public domain. When referencing an external web page in the body of a PEP, you should include the title of the page in the text, with a footnote reference to the URL. Do not include the URL in the body text of the PEP. E.g. Refer to the Python Language web site [1] for more details. ... [1] http://www.python.org When referring to another PEP, include the PEP number in the body text, such as "PEP 1". The title may optionally appear. Add a footnote reference that includes the PEP's title and author. It may optionally include the explicit URL on a separate line, but only in the References section. Note that the pep2html.py script will calculate URLs automatically, e.g.: ... Refer to PEP 1 [7] for more information about PEP style ... References [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html If you decide to provide an explicit URL for a PEP, please use this as the URL template: http://www.python.org/peps/pep-xxxx.html PEP numbers in URLs must be padded with zeros from the left, so as to be exactly 4 characters wide, however PEP numbers in text are never padded. Reporting PEP Bugs, or Submitting PEP Updates How you report a bug, or submit a PEP update depends on several factors, such as the maturity of the PEP, the preferences of the PEP author, and the nature of your comments. For the early draft stages of the PEP, it's probably best to send your comments and changes directly to the PEP author. For more mature, or finished PEPs you may want to submit corrections to the SourceForge bug manager[6] or better yet, the SourceForge patch manager[2] so that your changes don't get lost. If the PEP author is a SF developer, assign the bug/patch to him, otherwise assign it to the PEP editor. When in doubt about where to send your changes, please check first with the PEP author and/or PEP editor. PEP authors who are also SF committers, can update the PEPs themselves by using "cvs commit" to commit their changes. Remember to also push the formatted PEP text out to the web by doing the following: % python pep2html.py -i NUM where NUM is the number of the PEP you want to push out. See % python pep2html.py --help for details. Transferring PEP Ownership It occasionally becomes necessary to transfer ownership of PEPs to a new champion. In general, we'd like to retain the original author as a co-author of the transferred PEP, but that's really up to the original author. A good reason to transfer ownership is because the original author no longer has the time or interest in updating it or following through with the PEP process, or has fallen off the face of the 'net (i.e. is unreachable or not responding to email). A bad reason to transfer ownership is because you don't agree with the direction of the PEP. We try to build consensus around a PEP, but if that's not possible, you can always submit a competing PEP. If you are interested assuming ownership of a PEP, send a message asking to take over, addressed to both the original author and the PEP editor <peps(a)python.org>. If the original author doesn't respond to email in a timely manner, the PEP editor will make a unilateral decision (it's not like such decisions can be reversed. :). References and Footnotes [1] This historical record is available by the normal CVS commands for retrieving older revisions. For those without direct access to the CVS tree, you can browse the current and past PEP revisions via the SourceForge web site at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=… [2] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse [4] http://www.opencontent.org/openpub/ [5] The script referred to here is pep2html.py, which lives in the same directory in the CVS tree as the PEPs themselves. Try "pep2html.py --help" for details. The URL for viewing PEPs on the web is http://www.python.org/peps/ [6] http://sourceforge.net/tracker/?group_id=5470&atid=305470 [7] PEP 9, Sample PEP Template http://www.python.org/peps/pep-0009.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

8 14

congrats on 3.5! Alas, windows 7 users are having problems installing it
by Laura Creighton 15 Sep '19

15 Sep '19

webmaster has already heard from 4 people who cannot install it. I sent them to the bug tracker or to python-list but they seem not to have gone either place. Is there some guide I should be sending them to, 'how to debug installation problems'? Laura

5 6

PEP 506 secrets module
by Steven D'Aprano 15 Apr '16

15 Apr '16

Hi, As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement. https://www.python.org/dev/peps/pep-0506/ There is code and tests here: https://bitbucket.org/sdaprano/secrets or you can run hg clone https://sdaprano@bitbucket.org/sdaprano/secrets The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4. -- Steve

13 28

Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
by Nick Coghlan 08 Dec '15

08 Dec '15

Hi folks, I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence. As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet. Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/ There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first. Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe. Regards, Nick. P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's. ================================ PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Robert Kuska <rkuska(a)redhat.com>, Marc-André Lemburg <mal(a)lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour. Rationale ========= PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure. PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase. These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). Requirements for capability detection ===================================== As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour. Recommendation for an environment variable based security downgrade =================================================================== Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself. In these cases, a configuration mechanism is needed that provides: * an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _https_verify_envvar = 'PYTHONHTTPSVERIFY' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to: * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Recommendation for backporting to earlier Python versions ========================================================= Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _cert_verification_config = '<path to configuration file>' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- This approach is currently only defined for \*nix system Python installations. The recommended configuration file name is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Combining the recommendations ============================= If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Copyright ========= This document has been placed into the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

16 56

Avoiding CPython performance regressions
by Fabio Zadrozny 02 Dec '15

02 Dec '15

Hi python-dev, I've seen that on and off CPython had attempts to measure benchmarks over time to avoid performance regressions (i.e.: https://speed.python.org), but had nothing concrete so far, so, I ended up creating a hosted service for that (https://www.speedtin.com) and I'd like to help in setting up a structure to run the benchmarks from https://hg.python.org/benchmarks/ and properly upload them to SpeedTin (if CPython devs are Ok with that) -- note that I don't really have server to run the benchmarks, only to host the data (but https://speed.python.org seems to indicate that such a server is available...). There's a sample report at: https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time/ (it has real data from running using the PyPy benchmarks as I only discovered about the benchmarks from https://hg.python.org/benchmarks/ later on -- also, it doesn't seem to support Python 3 right now, so, it's probably not that useful for the current Python dev, but it does have some nice insight on CPython 2.7.x performance over time). Later on, the idea is being able to compare across different Python implementations which use the same benchmark set... (although that needs other implementations to also post to the data to SpeedTin). Note that uploading the data to SpeedTin should be pretty straightforward (by using https://github.com/fabioz/pyspeedtin, so, the main issue would be setting up o machine to run the benchmarks). Best Regards, Fabio

6 16

Deleting with setting C API functions
by Serhiy Storchaka 02 Dec '15

02 Dec '15

Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item, PyMappingMethods.mp_ass_subscript are used not only for setting attribute/item value, but also for deleting attribute/item (if value is NULL). This is not documented and should be. [1] Correspondingly public API functions like PyObject_SetAttr, PyObject_SetItem, PySequence_SetItem, PySequence_SetSlice, PyMapping_SetItemString can be used for deleting. But all these functions have special counterparts for deleting: PyObject_DelAttr etc. The question is wherever deleting ability of Set-functions is intentional, should we document this or deprecate and then delete? [1] http://bugs.python.org/issue25701

3 4

Help with a book
by Jacob Zimmerman 30 Nov '15

30 Nov '15

Hello Python Developers! I'm writing a book about descriptors, and I'm hoping to make it a comprehensive guide, covering just about everything there is to know about descriptors, including many tips to get past certain knowledge barriers and how to avoid certain pitfalls. I'm hoping for it to become the definitive guide for those who want to really understand descriptors. Raymond's Descriptor HowTo Guide is great, but focused. I want to expand on what that guide teaches. Anyway, I am currently reading through my initial rough draft and making corrections for a revised draft. Once I'm done with that, I'd like a few volunteers to read through it. First, to find spelling, grammatical, or consistency mistakes, though those should be few in number, but also to give input as to whether something seems too repetitive, if some more information should be given in a spot, and if I'm even blatantly wrong about anything (hopefully not). Anyone who helps will be able to receive a free ebook of the final copy as well as mention in the Special Thanks. I appreciate any help anyone can give to assist in making this book as great as it can be. Thanks, Jacob Zimmerman Twitter: @jacobz_20 Blog: http://programmingideaswithjake.wordpress.com

4 4

Can not pass configuration on mac os
by maochongzhao 28 Nov '15

28 Nov '15

Hi, I am new here wishing to contribute to python. I tried to go through the tutorial for seting up Cpython, while I cannot pass the configuration. My system is Mac OS and I have installed the command tool of Xcode and hg. But on terminal it said that no such file or directory when I type “./configure --with-pydebug”. What was wrong with my action? How can I fix the issue? This may a little question, but thanks for your help and tutoring! Sincerely, Chongzhao Mao

2 1

crash bug in weakref_richcompare?
by Luke Deller 28 Nov '15

28 Nov '15

Hi I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others. This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip). The code that worries me is at the end of the "weakref_richcompare" function: return PyObject_RichCompare(PyWeakref_GET_OBJECT(self), PyWeakref_GET_OBJECT(other), op); At this point the code has established that the referents are still alive, and it is trying to compare the referents. However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison. This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call. The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg: 00 python27!try_3way_compare+0x15 [objects\object.c @ 712] 01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901] 02 python27!do_richcmp+0x2c [objects\object.c @ 935] 03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982] 04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212] (In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all) Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr. I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly? Thanks, Luke. ********************************************************************************************** Important Note This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the sender immediately and delete this email. Any views expressed in this email are not necessarily the views of IRESS Limited. It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system. IRESS Limited does not warrant that the information is free of a virus or any other defect or error. **********************************************************************************************

2 1