Mailman 3 April 2014 - Python-Dev

Patches in need of review
by Claudiu Popa April 24, 2014

April 24, 2014

Hello! This approach worked for Nikolaus and I hope that it could work for me. I have a couple of languishing patches, waiting for a core dev to review, reject or commit them. I consider them ready to go, but another pair of eyes could unveil unknown problems to me. I pinged some issues from this list on core-mentorship a month ago, without luck though. Here they are: * http://bugs.python.org/issue19385 `dbm.dumb should be consistent when the database is closed` This patch adds a check for operations with closed dbm.dumb databases. The problem is that currently this fails with AttributeError: >>> import dbm.dumb as d >>> db = d.open('test.dat', 'c') >>> db.close() >>> db.keys() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/tank/libs/cpython/Lib/dbm/dumb.py", line 212, in keys return list(self._index.keys()) AttributeError: 'NoneType' object has no attribute 'keys' With this patch, the error is consistent with the other dbm flavours. * http://bugs.python.org/issue16104 `Use multiprocessing in compileall script` This patch adds a new command line argument to `compileall`, also a new argument to `compileall.compile_dir`, which controls the number of worker processes used to compile a given directory. The title of the issue is actually misleading a little, because the patch uses `concurrent.futures` instead. * http://bugs.python.org/issue19714 `Add tests for importlib.machinery.WindowsRegistryFinder.` This is Windows specific and adds tests for the finder in the title. * http://bugs.python.org/issue17442 `code.InteractiveInterpreter doesn't display the exception cause` This patch adds the ability for `code.InteractiveInterpreter` instances to display the exception cause as seen below: - currently >>> try: ... 1 / 0 ... except ZeroDivisionError as exc: ... raise IOError from exc ... Traceback (most recent call last): File "<console>", line 4, in <module> OSError - with the patch >>> try: ... 1 / 0 ... except ZeroDivisionError as exc: ... raise IOError from exc ... Traceback (most recent call last): File "<stdin>", line 2, in <module> ZeroDivisionError: division by zero The above exception was the direct cause of the following exception: Traceback (most recent call last): File "<stdin>", line 4, in <module> OSError Thank you for your time! Claudiu

2 1

this is what happens if you freeze all the modules required for startup
by Brett Cannon April 23, 2014

April 23, 2014

It was realized during PyCon that since we are freezing importlib we could now consider freezing all the modules to cut out having to stat or read them from disk. So for day 1 of the sprints I I decided to hack up a proof-of-concept to see what kind of performance gain it would get. Freezing everything except encodings.__init__, os, and _sysconfigdata, it speeds up startup by 11% compared to default. Compared to 2.7 it shaves 14% from the slowdown (27% slower vs. 41% slower). The full results are at the end of the email. Now the question is whether the maintenance cost of having to rebuild Python for a select number of stdlib modules is enough to warrant putting in the effort to make this work. My guess is the best approach would be adding a Lib/_frozen directory where any modules that we treat like this would be kept to act as a reminder that you need to rebuild for them (I would probably move importlib/_boostrap.py as well to make this consistent). Thoughts? -------------------------------------- default vs the freezing: ### normal_startup ### Min: 0.524812 -> 0.473339: 1.11x faster Avg: 0.534403 -> 0.481245: 1.11x faster Significant (t=61.80) Stddev: 0.00466 -> 0.00391: 1.1909x smaller ### startup_nosite ### Min: 0.307359 -> 0.291939: 1.05x faster Avg: 0.317667 -> 0.300156: 1.06x faster Significant (t=26.29) Stddev: 0.00543 -> 0.00385: 1.4099x smaller --------- 2.7 vs the freezing: ### normal_startup ### Min: 0.367571 -> 0.465264: 1.27x slower Avg: 0.374404 -> 0.476662: 1.27x slower Significant (t=-90.26) Stddev: 0.00313 -> 0.00738: 2.3603x larger ### startup_nosite ### Min: 0.164510 -> 0.290544: 1.77x slower Avg: 0.169833 -> 0.301109: 1.77x slower Significant (t=-286.30) Stddev: 0.00211 -> 0.00407: 1.9310x larger --------- As a baseline, 2.7 vs default: ### normal_startup ### Min: 0.368916 -> 0.521758: 1.41x slower Avg: 0.376784 -> 0.531883: 1.41x slower Significant (t=-172.82) Stddev: 0.00423 -> 0.00474: 1.1207x larger ### startup_nosite ### Min: 0.165156 -> 0.309090: 1.87x slower Avg: 0.171516 -> 0.319004: 1.86x slower Significant (t=-283.45) Stddev: 0.00334 -> 0.00399: 1.1948x larger

22 48

Open Source Organizational Culture
by Marius Storm-Olsen April 22, 2014

April 22, 2014

Hi, I would like to request your participation in a survey on Open Source Organizational Culture, which will provide valuable insight into how Open Source projects are run, how their participants act, how they might change going forward, and how particular Open Source projects compare with one another and with traditional business cultures. The survey will take 10-15 minutes to complete. http://bit.ly/osocas2014 Why? ---- The survey will be used as part of my thesis on Open Source Organizational Culture at BI Norwegian Business School (www.bi.no/en, or www.bi.edu), but in true Open Source spirit the raw - but anonymized - results will be open for all. So, your Open Source project will be able to massage and dissect the results any way you wish, and see how you compare with other projects out there. Up until now, most research in Open Source culture has been based on mining mailing lists to find out how people act, who they interact with, and how projects organize themselves. In this research we would rather ask the participants directly about how a project is managed and what should change for the project to be spectacularly successful. When? ----- The survey is open now through May 1st. Where? ------ The bit.ly address brings you to the following survey https://osocas.2014.sgizmo.com/s3/ Remember that you can save your progress at any time and come back to the survey at a later point when you have time to finish it. Who are you? ------------ My name is Marius Storm-Olsen, and I am currently working on a thesis on Open Source Organizational Culture. I've been an active part of Open Source for years, most notably on the Qt and Git projects. Although I have my own experiences to draw on in the thesis, they do not qualify for the Open Source community at large, hence the survey. How to help? ------------ If you want to help, feel free to forward this email to any Open Source project you would want to participate the survey. Once you have send the invitation, please either send me an email with the name of the project, or update the table shown on https://github.com/mstormo/OSOCAS/wiki I do hope you can participate, and thank you for your consideration! Best regards, Marius Storm-Olsen

1 1

PEP 469: Restoring the iterkeys/values/items() methods
by Nick Coghlan April 21, 2014

April 21, 2014

After spending some time talking to the folks at the PyCon Twisted sprints, they persuaded me that adding back the iterkeys/values/items methods for mapping objects would be a nice way to eliminate a key porting hassle for them (and likely others), without significantly increasing the complexity of Python 3. I personally put this one in the same category as PEP 414 - not particularly useful from a Python 3 perspective, but not really harmful either, and helpful enough from a transition perspective to be worth doing. Regards, Nick. ============================================== PEP: 469 Title: Simplified migration of iterator-based mapping code to Python 3 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-04-18 Python-Version: 3.5 Post-History: 2014-04-18 Abstract ======== For Python 3, PEP 3106 changed the design of the ``dict`` builtin and the mapping API in general to replace the separate list based and iterator based APIs in Python 2 with a merged, memory efficient set and multiset view based API. This means that Python 3 code always requires an additional qualifier to reliably reproduce classic Python 2 mapping semantics: * List based (e.g. ``d.keys()``): ``list(d.keys())`` * Iterator based (e.g. ``d.iterkeys()``): ``iter(d.keys())`` Some Python 2 code that uses ``d.keys()`` may be migrated to Python 3 (or the common subset of Python 2 and Python 3) without alteration, but *all* code using the iterator based API requires modification. Code that is migrating to the common subset of Python 2 and 3 and needs to retain the memory efficient implementation that avoids creating an unnecessary list object must switch away from using a method to instead using a helper function (such as those provided by the ``six`` module) To simplify the process of migrating Python 2 code that uses the existing iterator based APIs to Python 3, this PEP proposes the reintroduction of the Python 2 spelling of the iterator based semantics in Python 3.5, by restoring the following methods to the builtin ``dict`` API and the ``collections.abc.Mapping`` ABC definition: * ``iterkeys()`` * ``itervalues()`` * ``iteritems()`` Proposal ======== Methods with the following exact semantics will be added to the builtin ``dict`` type and ``collections.abc.Mapping`` ABC:: def iterkeys(self): return iter(self.keys()) def itervalues(self): return iter(self.values()) def iteritems(self): return iter(self.items()) These semantics ensure that the methods also work as expected for subclasses of these base types. Rationale ========= Similar in spirit to PEP 414 (which restored explicit Unicode literal support in Python 3.3), this PEP is aimed primarily at helping users that currently feel punished for making use of a feature that needed to be requested explicitly in Python 2, but was effectively made the default behaviour in Python 3. Users of list-based iteration in Python 2 that aren't actually relying on those semantics get a free memory efficiency improvement when migrating to Python 3, and face no additional difficulties when migrating via the common subset of Python 2 and 3. By contrast, users that actually want the increased efficiency may have faced a three phase migration process by the time they have fully migrated to Python 3: * original migration to the iterator based APIs after they were added in Python 2.2 * migration to a separate function based API in order to run in the common subset of Python 2 and 3 * eventual migration back to unprefixed method APIs when finally dropping Python 2.7 support at some point in the future The view based APIs that were added to Python 2.7 don't actually help with the transition process, as they don't exist in Python 3 and hence aren't part of the common subset of Python 2 and Python 3, and also aren't supported by most Python 2 mappings (including the collection ABCs). This PEP proposes to just eliminate all that annoyance by making the iterator based APIs work again in Python 3.5+. As with the restoration of Unicode literals, it does add a bit of additional noise to the definition of Python 3, but it does so while bringing a significant benefit in increasing the size of the common subset of Python 2 and Python 3 and so simplifying the process of migrating to Python 3 for affected Python 2 users. Acknowledgements ================ Thanks to the folks at the Twisted sprint table at PyCon for a very vigorous discussion of this idea (and several other topics), and especially to Hynek Schlawack for acting as a moderator when things got a little too heated :) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

27 52

Fw: fail to fetch python-dev for raspian with details of error message
by Ken Chan April 21, 2014

April 21, 2014

Dear sir, I am working on WebIOPi where library of python-dev is needed, so I use "sudo apt-get install python-dev" to install it. Then returned message of: fail to fetch http://mirrordirector.raspbian.org/raspbian/ wheezy...... details of error message: Found Python 2.7.3rc2... Trying to install python-dev using apt-get Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libexpat1-dev libssl-dev libssl-doc python2.7-dev The following NEW packages will be installed: libexpat1-dev libssl-dev libssl-doc python-dev python2.7-dev 0 upgraded, 5 newly installed, 0 to remove and 3 not upgraded. Need to get 31.3 MB of archives. After this operation, 42.2 MB of additional disk space will be used. Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main python2.7-dev armhf 2.7.3~rc2-2.1 404 Not Found Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main python-dev all 2.7.3~rc2-1 [912 B] Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libexpat1-dev armhf 2.1.0-1 [210 kB] Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libssl-dev armhf 1.0.1c-4+rpi1 [1,494 kB] Get:4 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libssl-doc all 1.0.1c-4+rpi1 [1,204 kB] Fetched 2,909 kB in 21s (136 kB/s) Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/p/python2.7/python2.7… 404 Not Found E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Cannot install for Python 2.7.3rc2 : missing development headers Found Python 3.2.3... Trying to install python3-dev using apt-get Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libexpat1-dev libpython3.2 libssl-dev libssl-doc python3.2-dev The following NEW packages will be installed: libexpat1-dev libpython3.2 libssl-dev libssl-doc python3-dev python3.2-dev 0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded. Need to get 31.0 MB/33.9 MB of archives. After this operation, 46.8 MB of additional disk space will be used. Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main libpython3.2 armhf 3.2.3-2 404 Not Found Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main python3.2-dev armhf 3.2.3-2 404 Not Found Get:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main python3-dev all 3.2.3-5 [1,060 B] Fetched 1,060 B in 0s (5,711 B/s) Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/p/python3.2/libpython… 404 Not Found Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/p/python3.2/python3.2… 404 Not Found E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Cannot install for Python 3.2.3 : missing development headers ERROR: WebIOPi cannot be installed - please check errors above Would you please to advise a solution. Many many thanks! Ken TIML MOM Limited 交通基建管理合約有限公司 P Please think of the environment before printing this email ------------------------------------------------------------------------ Disclaimer: This e-mail message (together with any attachments) is confidential to the addressee and may also be privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message is strictly prohibited. Please also notify the sender immediately by return e-mail and delete it from your system. Internet communications cannot be guaranteed to be secure or error-free. The sender and the entity through which this message is sent therefore

2 1

subprocess.Popen and win32
by David Aguilar April 20, 2014

April 20, 2014

Hi, I just joined python-dev because I found the need to add some code to paper over python3's subprocess API, and I'm wondering whether I'm missing something. On python2 and python3, the (only?) way to get utf-8 arguments to subprocess was to ensure that all unicode strings are encoded into bytes before subprocess sees them. This has worked for a long time (currently compatible across python2 and 3). On python3, this still works for normal platforms, but on windows we can't pass a list of byte strings. We have to pass a list of unicode strings. This means that the application code ends up needing to do this: https://github.com/git-cola/git-cola/commit/1109aeb4354c49931d9b0435d2b7cfd… basically, def start_command(cmd): if sys.platform == 'win32': # Python on windows always goes through list2cmdline() internally inside # of subprocess.py so we must provide unicode strings here otherwise # Python3 breaks when bytes are provided. cmd = [decode(c) for c in cmd] else: cmd = [encode(c) for c in cmd] return subprocess.Popen(cmd) That seems broken to me, so I wonder if this is a bug in the way python3 is handling Popen with list-of-bytestring on win32? I'm not a windows user, but I was able to install python3 under wine and the same traceback happens without the paper bag fix. This is what the traceback looks like; it dies in list2cmdline (which I am not calling directly, Popen does it under the covers): File "E:\Program Files (E)\git-cola\share\git-cola\lib\cola\core.py", line 109, in start_command universal_newlines=universal_newlines) File "C:\Python32\lib\subprocess.py", line 744, in __init__ restore_signals, start_new_session) File "C:\Python32\lib\subprocess.py", line 936, in _execute_child args = list2cmdline(args) File "C:\Python32\lib\subprocess.py", line 564, in list2cmdline needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: Type str doesn't support the buffer API This is an issue for folks that use python to write cross-platform code. The unix code paths expect list-of-bytes, but win32 only expects list-of-unicode, which pushes the burden onto the application programmer. It's my opinion that the win32 code path on python3 is the odd man out. If it allowed list-of-bytes like python2/win32 and python2+3/unix then this wouldn't be an issue. Is this an actual problem, or is it something that should be handled by application-level code as I've done? Thanks, -- David

3 3

Python usability study (Was: Language Summit notes)
by anatoly techtonik April 20, 2014

April 20, 2014

On Thu, Apr 10, 2014 at 2:24 PM, Kushal Das <kushaldas(a)gmail.com> wrote: > Glyph wants a PSF fund to a usability study on Python. There were a > few other suggestion on PSF support for tooling development. +2 on initiative -- anatoly t.

1 0

PEP 466 (round 2): Network security enhancements for Python 2.7
by Nick Coghlan April 19, 2014

April 19, 2014

Several significant changes in this revision: - scope narrowed to just Python 2.7 plus permission for commercial redistributors to use the same strategy in their long term support releases - far more explicit that this is about inviting potential corporate contributors to address the situation for the benefit of the overall Python ecosystem, not offering to fix it for them for free - clarified that third party integration testing services would need to be updated to support testing against multiple Python 2.7 minor releases - explicit sections on why I don't think the status quo is sustainable, why I don't think Python 2.8 would actually solve the problem, and why I think a PyPI based solution not only wouldn't solve the problem, but would be rather difficult to get working in the first place - be completely explicit that I am *not* speaking on behalf of Red Hat at this point and have no authority to make commitments on their behalf. Instead, I'm looking for upstream consensus that 1) this is a genuine problem that needs to be solved; 2) we're open to corporate assistance in solving it; and 3) we have a pretty good idea what help we actually want. If all that happens, *then* I can take up the issue internally to try to get us some help in maintaining the proposed solution (hopefully other folks with corporate influence can do the same, and we may actually get some ongoing assistance with upstream maintenance out of this, rather than having our downstream redistributors continue to take us for granted). Diff: http://hg.python.org/peps/rev/2e82209dda21 Updated web version: http://www.python.org/dev/peps/pep-0466/ Advance warning: while I was able to get this revision turned around pretty quickly, future revisions are likely to take a fair bit longer. It was already a rather busy month before I decided to start this discussion on top of everything else :) Cheers, Nick. PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. Accordingly, this PEP relaxes the normal restrictions by allowing enhancements to be applied in Python 2.7 maintenance releases for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers Proposed backports for these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else disclaim support for the use of older versions in roles that involving connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible changes in these modules to Python 2.7 maintenance releases. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that needs to evolve faster than the standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to review and merge corporate contributions that put this policy into effect. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole. Documentation ------------- All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors applying this policy to any Python release series that is either in security fix only mode, or has been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the redistributor. Integration testing ------------------- Third party integration testing services would likely need to start offering users a choice of multiple Python 2.7.x versions to test against, to ensure that the application is correctly degrading gracefully if it attempts to use newer networking features on maintenance releases that are too old to provide them. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, and currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. Now that we're fully aware of the impact the limitations in Python 2 may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements available in Python 3. While (as far as I am aware) Ubuntu has successfully switched to Python 3.4 as its main Python interpreter for its 14.04 LTS release, Fedora still has a lot of work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and those changes won't affect the core tools regardless. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my currently preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support). This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective as a solution to the original problem (the widespread use of the aging network security infrastructure in Python 2). Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While it initially appears to be an attractive and easier to manage approach, there are actually several significant problems with this idea. Firstly, this PEP encompasses a non-trivial portion of the standard library. It's not just the underlying SSL support, but also the libraries for other network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the SSL infrastructure to provide secure links, and that's just the protocols in the standard library. Even if an API compatible ``ssl2`` module was made available, it would need to be imported and injected into ``sys.modules`` as ``ssl`` before importing any other module that needed it. Secondly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than taking the risk of trying to upgrade it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means going around to each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to *opt out* of the security enhancements, which most of them are unlikely to do. Open Questions ============== * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases? Currently we just upgrade to the appropriate OpenSSL maintenance releases, rather than switching to the latest feature release. In particular, is it possible Windows C extensions may be linking against the Python provided OpenSSL module? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list thread [1]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

11 20

Software integrators vs end users (was Re: Language Summit notes)
by Nick Coghlan April 19, 2014

April 19, 2014

On 16 Apr 2014 21:03, "R. David Murray" <rdmurray(a)bitdance.com> wrote: > There is an 'Installing Python on Windows' link further down the google > results that links to a fairly good page from python-guide.org, whose > first link lets you download the 2.7.6 msi. I guess that's the 32 > bit version. But it then tells you go to python.org to make sure you > get the latest release, and tells you to click on a link that doesn't > exist any more (the "windows installer" link). > > So, yeah. > > Usability. As part of thrashing out the respective distribution ecosystem roles of pip and conda (still a work in progress), we're at least converging on the notion that there are actually now *two* main ways of consuming Python: as a "software integrator" (the way most of us have traditionally consumed it, and the way that dominates most project documentation outside the scientific Python community) and as an "end user" (the way Linux system administrators have long consumed it, and the way scientists, financial analysts and folks just learning Python are likely best off consuming it). Making these different personas explicit is a process that has barely begun (this email is mostly based on some conversations I had in person at PyCon and via email during the sprints), but here's the gist (based on listing examples): Software integrators: * Linux distributions and other operating system vendors * Sumo redistributions (commercial or otherwise) * "Python based environments" (PTVS, Enthought Canopy, wakari.io, Python Anywhere, etc) * Software-as-a-Service developers * Device manufacturers * PC OEMs * creators of corporate "Standard Operating Environment" definitions * System integrators (IBM, Boeing et al) * Application developers (from simple CLI tools to OpenStack) End users: * system administrators * scientists (whether in academia or corporations) * financial analysts * engineers * data miners * graphic artists * animation designers * <<< Folks trying out Python for the first time >>> For end users, Python is likely consumed as *part of something else*. For Linux admins, it's a way of scripting the system, for graphic artists and animators, it's likely embedded as part of a tool like Blender or Maya, for scientists, financial analysts, engineers and data miners, it likely makes sense to use it as part of a larger visualisation environment like IPython Notebook, Python (x, y), Enthought Canopy, or a hosted system like Wakari or Python Anywhere. Folks just learning Python are also in the latter list, and we currently ask them (on the python.org download pages) to jump straight into the "software integrator" role to get their environment set up, rather than setting out to impress them by recommending one of the pre-integrated sumo distributions that offers quick and easy access to powerful visualisation and data analysis tools. While providing access to the CPython default interactive prompt in the browser is cool, it's less impressive as the default experience we provide after someone has downloaded and installed the interpreter. Instead, we likely want to *really* impress them by making it easy for them to explore the full power of things like IPython Notebooks. My own current preferred approach for that is the fully open source "Anaconda" distribution from Continuum Analytics, specifically because it *is* fully open source, and you can "pip install conda" to bootstrap their package manager in other contexts. And, importantly, because conda environments can *manage the Python runtime itself*, it is able to work more consistently across different platforms than the upstream tools by reliably isolating itself from the system Python on POSIX platforms. It should even by possible to get conda to manage alternate Python implementations like Stackless, PyPy, Jython, IronPython, etc (although I don't believe anyone is using it that way as yet). However, conda *isn't* optimised for the software integrator use case - the additional work it does to improve cross platform consistency actually *gets in the way*, when you're trying to integrate Python closely with a larger system (like a Linux distribution), and "a large set of automatically provided libraries" isn't actually a feature in that context. While conda does offer some options (like miniconda) to make that kind of integration task easier, I strongly believe that realm is better handled by consuming CPython and the standard library, along with pip and related tools, directly, and accepting the additional platform specific details that come along with that. Making this "software integrator" and "end user" split deliberately and consciously, and pushing the former toward consuming things in a more DIY fashion, and the latter towards a pre-assembled sumo distribution should help greatly in allowing us to better balance the starkly different needs of the two groups, and provide an extremely high quality "out of the box" experience for new users, while still allowing software integrators to dive in and customise things to suit their own needs. >From a Python 2->3 migration perspective, blessing at least one sumo distribution also provides a way to make it easy to gain ready access to backports from the Python 3 standard library (for example, Anaconda currently includes at least the backports.ssl_match_hostname module). Regards, Nick.

8 26

dict and required hashing
by Jim J. Jewett April 19, 2014

April 19, 2014

(1) I believe the recent consensus was that the number of comparisons made in a dict lookup is an implementation detail. (Please correct me if I am wrong.) (2) Is "the item will be hashed at least once" a language guarantee? For small mappings, it might well be more efficient to just store the 2-3 key/value pairs and skip the bucket calculation. On the other hand, if a key is not hashable, discovering that long after it has already been added to the dict is suboptimal. Of course, that sort of delayed exception can already happen if it is the __eq__ method that is messed up ... -jJ

5 6