Mailman 3 March 2014 - Python-Dev

Jython site now seems to redirect to docs.python.org???
by Paul Moore March 28, 2014

March 28, 2014

I'm not sure if this is a result of the recent website reorg, but www.jython.org seems to be redirecting to docs.python.org for me. Presumably this is an error - where do I report it and/or is it a known issue? Thanks, Paul.

4 6

multiprocessing BaseManager's "serializer" argument
by Yuriy Taraday March 27, 2014

March 27, 2014

Hello. Is there any reason BaseManager's "serializer" argument has never been documented? We're going to use BaseManager for simple secure local RPC and for the "secure" part we can't use pickle. Can we hope that argument won't go away? Should I create a bug and provide some patch for the docs to add mention of this argument? -- Kind regards, Yuriy.

2 2

Re: [Python-Dev] cpython: asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668
by Antoine Pitrou March 27, 2014

March 27, 2014

On Thu, 27 Mar 2014 17:22:01 +0100 (CET) yury.selivanov <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/bcc77493249c > changeset: 89995:bcc77493249c > user: Yury Selivanov <yselivanov(a)sprymix.com> > date: Thu Mar 27 12:21:20 2014 -0400 > summary: > asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668 > > files: > Lib/test/test_asyncio/__init__.py | 22 ++++++++---------- > Misc/NEWS | 3 ++ > 2 files changed, 13 insertions(+), 12 deletions(-) Perhaps the tests.txt file should be removed from the repo? Regards Antoine.

1 0

PEP 466 (round 4): Python 2.7 network security enhancements
by Nick Coghlan March 27, 2014

March 27, 2014

Hmm, should probably have more entries in the post history... Anyway: * As Ben Darnell noted, the legacy SSL branch thing was a fundamentally bad idea. We can't stop downstream redistributors doing that if they really want to, but we don't have to encourage them * I also explained why I think selective backports are a bad idea * I've given a lot more context on how Red Hat actually manages to handle such long term support cycles, and that what I'm proposing here really isn't that radical from that perspective (and is, in fact, about the only available solution that can be effectively slotted into that model) * I've been a lot more explicit about the fact that long term maintenance releases are genuinely *different* from CPython's normal release lifetimes * more details on the specific support I think this proposal would need to become a practical reality * removed some unhelpful side comments and obliquely acknowledged the fact that it mat conceivably have been difficult for outsiders to see that it might actually be possible to make this case effectively :) Cheers, Nick. Diff: http://hg.python.org/peps/rev/4eb8de143801 Web: http://www.python.org/dev/peps/pep-0466/ =================================== PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in commercial use cases where upgrading to Python 3 in the near term may not be practical. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows Python 2.7 standard library components that have implications for the overall security of the internet to be updated in line with the corresponding Python 3 feature releases. Specifically, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the components of the ``random`` and ``os`` modules that the above modules rely on for cryptographic randomness * the version of OpenSSL bundled with the binary installers for Windows and Mac OS X Changes to these modules will still need to undergo normal backwards compatibility assessments to ensure their default behaviour remains consistent with earlier Python 2.7 releases, but otherwise they will be kept entirely aligned with the latest feature release of their Python 3 counterparts. This is designed to make it easier to implement secure networked software in Python, even for software that currently needs to remain compatible with the Python 2 series (which includes a lot of network infrastructure software). While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either adopt a similar approach to ensuring that the secure networking infrastructure keeps pace with the evolution of the internet, or else explicitly disclaim support for the use of older versions in roles that involve connecting directly to the public internet. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the normal restriction against adding new features in Python 2.7 maintenance releases, for the purpose of keeping their APIs aligned with their counterparts in the latest feature release of Python 3: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module Under this exemption, these modules are updated to provide identical functionality to their Python 3 counterparts after every new Python 3 feature release. The default behaviour of the backported modules will be adjusted as appropriate to meet the backwards compatibility requirements of a Python 2.7 maintenance release. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. Note that the ``sha`` and ``md5`` modules are not covered by this policy, but have been deprecated in favour of ``hashlib`` since Python 2.5 and have been removed entirely in the Python 3 series. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (specifically ``os.urandom``) * the ``random`` module This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. This is generally restricted to cases where the feature in question is needed by an update to one of the modules covered by the blanket exemption. Backwards Compatibility Considerations ====================================== This PEP does *not* grant Python 2.7 any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks and explicitly opting in to less secure configurations, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. In *all* cases where this policy is applied to backport enhancements to Python 2.7 maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect and handling the resulting increase in the Python 2 maintenance load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to contribute some of those changes back to the upstream Python community in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole, with the assurance that the existing core development team not only won't object to such contributions, but will actively encourage their incorporation into the next Python 2.7 maintenance release. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" section in their documentation in order to take advantage of this policy. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply this policy to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against specific Python 2.7 maintenance releases, to ensure that libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even after the latest Python 2.7 maintenance release has been synchronised with a new Python 3 feature release for the modules covered by this policy. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than even a slightly increased risk of regressions in maintenance releases. This policy largely excludes these environments from consideration where the modules covered by the exemption are concerned - this approach is entirely inappropriate for software connected to the public internet, and defence in depth security principles suggest that it is not appropriate for most private networks either. Downstream redistributors may still choose to cater to such environments, but they will need to handle the process of downgrading the security related modules and doing the associated regression testing themselves. The main CPython continuous integration infrastructure will not cover this scenario. Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may themselves be using newer versions of Python or other development languages, but need to interoperate with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python 2.7 maintenance releases as they are about updating to new feature releases within the same release series. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. Background ---------- The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them (in particular, it is still difficult for sotware to explicitly require some higher level security settings). In the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, but is currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Python 2.7 represents the only long term maintenance release the core development team has provided, and it is natural that there will be things that worked over a historically shorter maintenance lifespan that don't work over this longer support period. In the specific case of the problem described in this PEP, the simplest available solution is to acknowledge that long term maintenance of network security related modules *requires* the ability to add new features, even while retaining backwards compatibility for existing interfaces. It is worth comparing the approach described in this PEP with Red Hat's handling of its long term support commitments: it isn't the RHEL 6.0 release itself that receives 10 years worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing software. Subscribers *are* able to continue using a given point release after the next point release has become available, but a corresponding add-on subscription for `Extended Update Support <https://access.redhat.com/site/support/policy/updates/errata/#Extended_Upda…>`__ is needed to cover the additional backporting work involved. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly accepts that a more nuanced policy is appropriate in the case of network security related features, and the specific one it describes is deliberately designed such that it at least has some chance of being applied to Red Hat Enterprise Linux and its downstream derivatives. Alternative: advise developers of networked software to migrate to Python 3 --------------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my preferred approach to addressing the situation. Alternative: create and release Python 2.8 ------------------------------------------ With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Furthermore, while I can't make any commitments to actually addressing this issue on Red Hat platforms, I *can* categorically rule out the idea of a Python 2.8 being of any use to me in even attempting to get it addressed. Alternative: distribute the security enhancements via PyPI ---------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements by deliberately downgrading the provided network security infrastructure, which most of them are unlikely to do. Alternative: provide a "legacy SSL infrastructure" branch --------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network security infrastructure. It is the opinion of the PEP author that anyone that actually wants this is almost certainly making a mistake, and if they insist they really do want it in their specific situation, they're welcome to either make it themselves or arrange for a downstream redistributor to make it for them. If they are made publicly available, any such rebuilds should be referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official Python 2.7 releases that include more up to date network security infrastructure. After the first Python 2.7 maintenance release that has the security infrastructure updated to match Python 3.4, it would also be appropriate to refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". Alternative: selectively backport particular APIs ------------------------------------------------- An instinctively minimalist reaction to this proposal is to only backport particular APIs in the affected modules that are judged to be "security critical". However, this ends up providing a worse end user experience, as well as a worse developer experience. For end users, the selective backporting approach means learning not only the legacy Python 2.7 API and the current Python 3 APIs, but also the hybrid API created by the selective backporting process. It is much simpler to just learn the APIs of the original Python 2.7 feature release and the relevant Python 3 features releases, without trying to define a new hybrid API that only exists in later Python 2.7 maintenance branches. For developers, the main benefit of the "just align the available functionality with Python 3" approach is that it reduces the amount of design work needing when updating Python 2.7 with new security features. The "feature or fix?" and "security related or not?" debates are both handled in advance by this policy, leaving only the matter of ensuring that backwards compatibility is preserved as needed by adjusting the default behaviour in the Python 2.7 backport when appropriate. Open Questions ============== * MvL has indicated he is not prepared to tackle the task of trying to integrate a newer OpenSSL into the also aging Python 2.7 build infrastructure on Windows (unfortunately, we've looked into upgrading that build infrastructure, and the backwards compatibility issues appear to be effectively insurmountable). We would require a commitment from another trusted contributor to handle at least this task, and potentially also taking over the task of creating the official Python 2.7 Windows installers for the remaining Python 2.7 maintenance releases. * We would need commitments to create and review full backports of the components covered by this policy from Python 3.4 to Python 2.7, as well as support for handling any more specific security issues affecting these modules. * I believe I've addressed all the technical and scope questions I had, or others raised. That just leaves the question of "Do we agree this plan actually makes sense, given the constraints on downstream redistributors that would also like to see this problem solved?" :) Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2,5]_ References ========== .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html .. [2] https://mail.python.org/pipermail/python-dev/2014-March/133389.html .. [3] https://mail.python.org/pipermail/python-dev/2014-March/133438.html .. [4] https://mail.python.org/pipermail/python-dev/2014-March/133347.html .. [5] https://mail.python.org/pipermail/python-dev/2014-March/133442.html -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

9 21

PEP 466 (round 5): selected network security enhancements for Python 2.7
by Nick Coghlan March 26, 2014

March 26, 2014

Guido and Antoine persuaded me that selective backports would be a better idea for the network security enhancements than the wholesale module backports previously suggested, while Alex and Donald provided the necessary additional details, so here's a revised version of the PEP. Despite making it more explicit, I deleted more lines than I added, strongly suggesting that switching to selective backports was the right call :) I dealt with the SSL module the way Donald suggested: excluding the RAND_* functions, rather than listing everything else. I also changed the headings to make it clear the listed alternatives were rejected ideas, made the footnotes a bit more readable, and tidied up the wording in a few places. Diff: http://hg.python.org/peps/rev/8527f6e2beb0 Web: http://www.python.org/dev/peps/pep-0466/ ========================== PEP: 466 Title: Network Security Enhancement Exception for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This cadence works reasonably well during Python's normal 18-24 month feature release cycle, which is still applicable to the Python 3 series. However, the age of the standard library in Python 2 has now reached a point where it is sufficiently far behind the state of the art in network security protocols for it to be causing real problems in use cases where upgrading to Python 3 in the near term may not be feasible. In recognition of the additional practical considerations that have arisen during the 4+ year maintenance cycle for Python 2.7, this PEP allows Python 2.7 standard library components that have implications for the overall security of the internet to be updated in line with the corresponding Python 3 feature releases. Specifically, the exception allows a critical set of network security related features to be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release. While this PEP does not make any changes to the core development team's handling of security-fix-only branches that are no longer in active maintenance, it *does* recommend that commercial redistributors providing extended support periods for the Python standard library either backport these features to their supported versions, or else explicitly disclaim support for the use of older versions in roles that involve connecting directly to the public internet. Exemption Policy ================ Under this policy, the following features SHOULD be backported from Python 3.4 to the upcoming Python 2.7.7 maintenance release: * in the ``os`` module: * persistent file descriptor for ``os.urandom()``. * in the ``hmac`` module: * constant time comparison function (``hmac.compare_digest()``). * in the ``hashlib`` module: * password hashing function (``hashlib.pbkdf2_hmac()``). * details of hash algorithm availability (``hashlib.algorithms_guaranteed`` and ``hashlib.algorithms_guaranteed``). * in the ``ssl`` module: * this module is almost entirely synchronised with its Python 3 counterpart, bringing TLSv2, SSLContext manipulation, Server Name Identification, access to platform certificate stores, standard library support for peer hostname validation and more to the Python 2 series. * the only ``ssl`` module features *not* backported under this policy are the ``ssl.RAND_*`` functions that provide access to OpenSSL's random number generation capabilities - use ``os.urandom()`` instead. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of Python 2.7. Backwards Compatibility Considerations ====================================== As in the Python 3 series, the backported ``ssl.create_default_context()`` API is granted a backwards compatibility exemption that permits the protocol, options, cipher and other settings of the created SSL context to be made This PEP does *not* grant any exemptions to the usual backwards compatibility policy for maintenance releases. Instead, by explicitly encouraging the use of feature based checks, it is designed to make it easier to write more secure cross-version compatible Python software, while still limiting the risk of breaking currently working software when upgrading to a new Python 2.7 maintenance release. In all cases where this policy allows new features to be backported to the Python 2.7 release series, it is possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in a module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate warning and fallback behaviour if a desired feature is found to be missing. While some especially security sensitive software MAY fail outright if a desired security feature is unavailable, most software SHOULD instead emit a warning and continue operating using a slightly degraded security configuration. The backported APIs allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older maintenance releases of Python with less secure default behaviour) * explicitly opt in to less secure settings (to allow the use of newer Python feature releases in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but DOES NOT require checking for a specific maintenance release) Security related changes to other modules (such as higher level networking libraries and data format processing libraries) will continue to be made available as backports and new modules on the Python Package Index, as independent distribution remains the preferred approach to handling software that must continue to evolve to handle changing development requirements independently of the Python 2 standard library. Refer to the `Motivation and Rationale`_ section for a review of the characteristics that make the secure networking infrastructure worthy of special consideration. OpenSSL compatibility --------------------- Under this policy, OpenSSL may be upgraded to more recent feature releases in Python 2.7 maintenance releases. On Linux and most other POSIX systems, the specific version of OpenSSL used already varies, as CPython dynamically links to the system provided OpenSSL library by default. For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules are statically linked with OpenSSL and the associated symbols are not exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the ``egenix-pyopenssl`` binaries has not resulted in any reported compatibility issues [3]_ The Mac OS X binary installers historically followed the same policy as other POSIX installations and dynamically linked to the Apple provided OpenSSL libraries. However, Apple has now ceased updating these cross-platform libraries, instead requiring that even cross-platform developers adopt Mac OS X specific interfaces to access up to date security infrastructure on their platform. Accordingly, and independently of this PEP, the Mac OS X binary installers were already going to be switched to statically linker newer versions of OpenSSL [4]_ Other Considerations ==================== Maintainability --------------- This policy does NOT represent a commitment by volunteer contributors to actually backport network security related changes from the Python 3 series to the Python 2 series. Rather, it is intended to send a clear signal to potential corporate contributors that the core development team are willing to accept offers of corporate assistance in putting this policy into effect and handling the resulting increase in the Python 2 maintenance load. Backporting security related fixes and enhancements to earlier versions is a common service for commercial redistributors to offer to their customers. This policy represents an explicit invitation to implement those changes in the core development tree in cases where they are likely to have a broad impact that helps improve the security of the internet as a whole, with the assurance that the existing core development team not only won't object to such contributions, but will actively encourage their incorporation into the next Python 2.7 maintenance release. Documentation ------------- All modules covered by this policy MUST include a "Security Considerations" section in their documentation in order to take advantage of this policy. In addition to any other module specific contents, this section SHOULD enumerate key security enhancements and fixes (with CVE identifiers where applicable), along with the feature and maintenance releases that first included them. Security releases ----------------- This PEP does not propose any changes to the handling of security releases - those will continue to be source only releases that include only critical security fixes. However, the recommendations for library and application developers are deliberately designed to accommodate commercial redistributors that choose to apply this policy to additional Python release series that are either in security fix only mode, or have been declared "end of life" by the core development team. Whether or not redistributors choose to exercise that option will be up to the individual redistributor. Integration testing ------------------- Third party integration testing services should offer users the ability to test against specific Python 2.7 maintenance releases, to ensure that libraries, frameworks and applications can still test their handling of the legacy security infrastructure correctly (either failing or degrading gracefully, depending on the security sensitivity of the software), even after the features covered in this policy have been backported to the Python 2.7 series. Handling lower security environments with low risk tolerance ------------------------------------------------------------ For better or for worse (mostly worse), there are some environments where the risk of latent security defects is more tolerated than even a slightly increased risk of regressions in maintenance releases. This policy largely excludes these environments from consideration where the modules covered by the exemption are concerned - this approach is entirely inappropriate for software connected to the public internet, and defence in depth security principles suggest that it is not appropriate for most private networks either. Downstream redistributors may still choose to cater to such environments, but they will need to handle the process of downgrading the security related modules and doing the associated regression testing themselves. The main CPython continuous integration infrastructure will not cover this scenario. Evolution of this Policy ======================== The key requirement for a feature to be considered for inclusion in this policy is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols, password storage and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may themselves be using newer versions of Python or other development languages, but need to interoperate with clients or servers written using older versions of Python. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python 2.7 maintenance releases as they are about updating to new feature releases within the same release series. Motivation and Rationale ======================== The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While they represented reasonable security infrastructure in their time, the state of the art has moved on, and we need to investigate mechanisms for effectively providing more up to date network security infrastructure for users that, for whatever reason, are not currently in a position to migrate to Python 3. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them (in particular, it is still difficult for sotware to explicitly require some higher level security settings). In the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team, but is currently limited to OpenSSL maintenance releases for the version initially shipped with the corresponding Python feature release. With increased popularity comes increased responsibility, and this policy aims to acknowledge the fact that Python's popularity and adoption is at a sufficiently high level that some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, many users don't even consider the issue and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I anticipate that in the not too distant future, it will be considered too difficult to develop truly secure modern networked software using the Python 2 series (some developers would argue that we have already reached that point). Python 2.7 represents the only long term maintenance release the core development team has provided, and it is natural that there will be things that worked over a historically shorter maintenance lifespan that don't work over this longer support period. In the specific case of the problem described in this PEP, the simplest available solution is to acknowledge that long term maintenance of network security related modules *requires* the ability to add new features, even while retaining backwards compatibility for existing interfaces. It is worth comparing the approach described in this PEP with Red Hat's handling of its long term support commitments: it isn't the RHEL 6.0 release itself that receives 10 years worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x point releases within the series then receive a wide variety of new features, including security enhancements, all while meeting strict backwards compatibility guarantees for existing software. The policy described in this PEP brings our approach to long term maintenance more into line with this precedent - we retain our strict backwards compatibility requirements, but slightly relax the restrictions against adding new features. To date, downstream redistributors have respected our upstream policy of "no new features in Python maintenance releases". This PEP explicitly accepts that a more nuanced policy is appropriate in the case of network security related features, and the specific one it describes is deliberately designed such that it at least has some chance of being applied to Red Hat Enterprise Linux and its downstream derivatives. Rejected alternative: just advise developers to migrate to Python 3 ------------------------------------------------------------------- This alternative represents the status quo. Unfortunately, it has proven to be unworkable in practice, as the backwards compatibility implications mean that this is a non-trivial migration process for large applications and integration projects. While the tools for migration have evolved to a point where it is possible to migrate even large applications opportunistically and incrementally (rather than all at once) by updating code to run in the large common subset of Python 2 and Python 3, using the most recent technology often isn't a priority in commercial environments. Previously, this was considered an acceptable harm, as while it was an unfortunate problem for the affected developers to have to face, it was seen as an issue between them and their management chain to make the case for infrastructure modernisation, and this case would become naturally more compelling as the Python 3 series evolved. However, now that we're fully aware of the impact the limitations of the Python 2 standard library may be having on the evolution of internet security standards, I no longer believe that it is reasonable to expect platform and application developers to resolve all of the latent defects in an application's Unicode correctness solely in order to gain access to the network security enhancements already available in Python 3. While Ubuntu (and to some extent Debian as well) are committed to porting all default system services and scripts to Python 3, and to removing Python 2 from its default distribution images (but not from its archives), this is a mammoth task and won't be completed for the Ubuntu 14.04 LTS release (at least for the desktop image - it may be achieved for the mobile and server images). Fedora has even more work to do to migrate, and it will take a non-trivial amount of time to migrate the relevant infrastructure components. While Red Hat are also actively working to make it easier for users to use more recent versions of Python on our stable platforms, it's going to take time for those efforts to start having an impact on end users' choice of version, and any such changes also don't benefit the core platform infrastructure that runs in the integrated system Python by necessity. The OpenStack migration to Python 3 is also still in its infancy, and even though that's a project with an extensive and relatively robust automated test suite, it's still large enough that it is going to take quite some time to migrate fully to a Python 2/3 compatible code base. And that's just three of the highest profile open source projects that make heavy use of Python. Given the likely existence of large amounts of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3, there are likely to be many cases where reimplementation (perhaps even in Python 3) proves easier than migration. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: the existence of clients and servers with outdated network security infrastructure becomes something that developers of secure networked services need to take into account as part of their security design, and that's a problem that inhibits the adoption of better security standards. As Terry Reedy noted, if we try to persist with the status quo, the likely outcome is that commercial redistributors will attempt to do something like this on behalf of their customers *anyway*, but in a potentially inconsistent and ad hoc manner. By drawing the scope definition process into the upstream project we are in a better position to influence the approach taken to address the situation and to help ensure some consistency across redistributors. The problem is real, so *something* needs to change, and this PEP describes my preferred approach to addressing the situation. Rejected alternative: create and release Python 2.8 --------------------------------------------------- With sufficient corporate support, it likely *would* be possible to create and release Python 2.8 (it's highly unlikely such a project would garner enough interest to be achievable with only volunteers). However, this wouldn't actually solve the problem, as the aim is to provide a *relatively low impact* way to incorporate enhanced security features into integrated products and deployments that make use of Python 2. Upgrading to a new Python feature release would mean both more work for the core development team, as well as a more disruptive update that most potential end users would likely just skip entirely. Attempting to create a Python 2.8 release would also bring in suggestions to backport many additional features from Python 3 (such as ``tracemalloc`` and the improved coroutine support), making the migration from Python 2.7 to this hypothetical 2.8 release even riskier and more disruptive. This is not a recommended approach, as it would involve substantial additional work for a result that is actually less effective in achieving the original aim (which is to eliminate the current widespread use of the aging network security infrastructure in the Python 2 series). Furthermore, while I can't make any commitments to actually addressing this issue on Red Hat platforms, I *can* categorically rule out the idea of a Python 2.8 being of any use to me in even attempting to get it addressed. Rejected alternative: distribute the security enhancements via PyPI ------------------------------------------------------------------- While this initially appears to be an attractive and easier to manage approach, it actually suffers from several significant problems. Firstly, this is complex, low level, cross-platform code that integrates with the underlying operating system across a variety of POSIX platforms (including Mac OS X) and Windows. The CPython BuildBot fleet is already set up to handle continuous integration in that context, but most of the freely available continuous integration services just offer Linux, and perhaps paid access to Windows. Those services work reasonably well for software that largely runs on the abstraction layers offered by Python and other dynamic languages, as well as the more comprehensive abstraction offered by the JVM, but won't suffice for the kind of code involved here. The OpenSSL dependency for the network security support also qualifies as the kind of "complex binary dependency" that isn't yet handled well by the ``pip`` based software distribution ecosystem. Relying on a third party binary dependency also creates potential compatibility problems for ``pip`` when running on other interpreters like ``PyPy``. Another practical problem with the idea is the fact that ``pip`` itself relies on the ``ssl`` support in the standard library (with some additional support from a bundled copy of ``requests``, which in turn bundles ``backport.ssl_match_hostname``), and hence would require any replacement module to also be bundled within ``pip``. This wouldn't pose any insurmountable difficulties (it's just another dependency to vendor), but it *would* mean yet another copy of OpenSSL to keep up to date. This approach also has the same flaw as all other "improve security by renaming things" approaches: they completely miss the users who most need help, and raise significant barriers against being able to encourage users to do the right thing when their infrastructure supports it (since "use this other module" is a much higher impact change than "turn on this higher security setting"). Deprecating the aging SSL infrastructure in the standard library in favour of an external module would be even more user hostile than accepting the slightly increased risk of regressions associated with upgrading it in place. Last, but certainly not least, this approach suffers from the same problem as the idea of doing a Python 2.8 release: likely not solving the actual problem. Commercial redistributors of Python are set up to redistribute *Python*, and a pre-existing set of additional packages. Getting new packages added to the pre-existing set *can* be done, but means approaching each and every redistributor and asking them to update their repackaging process accordingly. By contrast, the approach described in this PEP would require redistributors to deliberately *opt out* of the security enhancements by deliberately downgrading the provided network security infrastructure, which most of them are unlikely to do. Rejected variant: provide a "legacy SSL infrastructure" branch -------------------------------------------------------------- Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl`` branch that preserved the exact feature set of the Python 2.7.6 network security infrastructure. It is the opinion of the PEP author that anyone that actually wants this is almost certainly making a mistake, and if they insist they really do want it in their specific situation, they're welcome to either make it themselves or arrange for a downstream redistributor to make it for them. If they are made publicly available, any such rebuilds should be referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them from the official Python 2.7 releases that include more up to date network security infrastructure. After the first Python 2.7 maintenance release that has the security infrastructure updated to match Python 3.4, it would also be appropriate to refer to Python 2.7.6 and earlier releases as "Python 2.7 with Legacy SSL". Rejected variant: synchronise particular modules entirely with Python 3 ----------------------------------------------------------------------- Earlier versions of this PEP suggested synchronising the ``hmac``, ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts. This approach proved too vague to build a compelling case for the exception, and has thus been replaced by the current more explicit proposal. Open Questions ============== * MvL has indicated he is not prepared to tackle the task of trying to integrate a newer OpenSSL into the also aging Python 2.7 build infrastructure on Windows (unfortunately, we've looked into upgrading that build infrastructure, and the backwards compatibility issues appear to be effectively insurmountable). We would require a commitment from another trusted contributor to handle at least this task, and potentially also taking over the task of creating the official Python 2.7 Windows installers for the remaining Python 2.7 maintenance releases. * We would need commitments to create and review full backports of the components covered by this policy from Python 3.4 to Python 2.7, as well as support for handling any more specific security issues affecting these modules. * Did I miss anything important in the switch to a more restrictive proposal? Disclosure of Interest ====================== The author of this PEP currently works for Red Hat on test automation tools. If this proposal is accepted, I will be strongly encouraging Red Hat to take advantage of the resulting opportunity to help improve the overall security of the Python ecosystem. However, I do not speak for Red Hat in this matter, and cannot make any commitments on Red Hat's behalf. Acknowledgements ================ Thanks to Christian Heimes and other for their efforts in greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set of essential security features that allowed the proposal to be made more fine-grained than backporting entire modules from Python 3.4 [7,8]_. Christian and Donald also provided valuable feedback on a preliminary draft of this proposal. Thanks also to participants in the python-dev mailing list threads [1,2,5,6]_ References ========== .. [1] PEP 466 discussion (round 1) (https://mail.python.org/pipermail/python-dev/2014-March/133334.html) .. [2] PEP 466 discussion (round 2) (https://mail.python.org/pipermail/python-dev/2014-March/133389.html) .. [3] Marc-Andre Lemburg's OpenSSL feedback for Windows (https://mail.python.org/pipermail/python-dev/2014-March/133438.html) .. [4] Ned Deily's OpenSSL feedback for Mac OS X (https://mail.python.org/pipermail/python-dev/2014-March/133347.html) .. [5] PEP 466 discussion (round 3) (https://mail.python.org/pipermail/python-dev/2014-March/133442.html) .. [6] PEP 466 discussion (round 4) (https://mail.python.org/pipermail/python-dev/2014-March/133472.html) .. [7] Donald Stufft's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133500.html) .. [8] Alex Gaynor's recommended set of backported features (https://mail.python.org/pipermail/python-dev/2014-March/133503.html) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 10

C code: %s vs %U
by Ethan Furman March 26, 2014

March 26, 2014

%s is a string. %U is unicode? If so, then %s should only be used when it is certain the string in question has no unicode in it? -- ~Ethan~

5 7

PEP 466: Proposed policy change for handling network security enhancements
by Nick Coghlan March 26, 2014

March 26, 2014

Folks, I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements. I now see these as categorically different from most other enhancements, as they have implications for the evolution of internet security as a whole, rather than being limited to affecting the security of individual applications written in Python. It is similar to the previous IDLE policy exception PEP, where we decided that cross version consistency of IDLE superseded the general policy against backporting enhancements to maintenance branches. This new PEP proposes a similar exception for network security related changes, such that the *capabilities* of all still maintained versions should be kept equivalent, with only the *default settings* varying across up to date feature releases. The policy is also designed to make it easier for us to ratchet up the default security settings in future feature releases. However, it's more complex than the IDLE PEP (since it's more invasive), and there are still a few open questions listed. This topic involves a complex balance between encouraging and supporting good security practices and limiting the risk of failures for users upgrading to new maintenance releases, so I'd ask that folks take time to read and consider the implications of the full PEP in the broader context of today's internet before posting comments on specific details, or indicating a preference one way or the other in terms of the overall proposal. Full PEP included inline below, and available in more readable form at http://www.python.org/dev/peps/pep-0466/ Regards, Nick. PEP: 466 Title: Network Security Enhancement Exception for All Branches Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com>, Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014 Abstract ======== Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version. This PEP relaxes the latter restriction allowing enhancements to be applied to maintenance branches for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers Changes to these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python. Exemption Policy ================ Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases: * the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) This exemption applies to *all* proposals to backport backwards compatible features in these modules to active maintenance branches. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software. As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython. In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features: * the ``os`` module (primarily ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related. Backwards Compatibility Considerations ====================================== This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release. In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version. It is then up to library and framework code to provide an appropriate error message or fallback behaviour if a desired feature is found to be missing. Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature: * explicitly opt in to more secure settings (to allow the use of enhanced security features in older versions of Python) * explicitly opt in to less secure settings (to allow the use of newer Python versions in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT depend on the specific maintenance release) Documentation Requirements ========================== All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation. In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), and the Evolution of this Policy ======================== The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages. The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases. Motivation and Rationale ======================== This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community. The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago. These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood. While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team. With increased popularity comes increased responsibility, and this policy is about recognising the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community. As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly. Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2. The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out. My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I consider it too difficult to develop truly secure modern networked software using the Python 2 series without introducing this network security enhancement policy, as doing so would mean reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts. Requiring that latent defects in an application's Unicode correctness be addressed in order to migrate to Python 3 is not a reasonable alternative recommendation, especially given the likely existence of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. By making it more feasible to enhance the security of the Python 2 series, we can help contribute to the evolution of a more secure internet for all concerned. For the Python 2 series, this proposal will most obviously impact the remaining maintenance releases of Python 2.7. However, commercial redistributors that continue to offer full support for Python 2.6, and will also do so for Python 2.7 after upstream maintenance ends, may choose to take this policy into account when deciding what changes to backport to their own maintenance updates. Providing commercial redistributors such as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and Red Hat that option is one of the benefits offered by requiring feature detection based forward compatibility for the modules covered by this policy. Open Questions ============== * What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases? * Are there any other security relevant modules that should be covered by either a blanket or conditional exemption? * Should we enumerate a specific list of "other networking modules" rather than leaving it implicit? Acknowledgement =============== Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole. Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

28 109

[RELEASED] Python 3.4.0
by Larry Hastings March 26, 2014

March 26, 2014

On behalf of the Python development team, I'm thrilled to announce the official release of Python 3.4. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series include: * PEP 428, a "pathlib" module providing object-oriented filesystem paths * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, a new "statistics" module * PEP 451, standardizing module metadata for Python's module import system * PEP 453, a bundled installer for the *pip* package manager * PEP 454, a new "tracemalloc" module for tracing Python memory allocations * PEP 456, a new hash algorithm for Python strings and binary data * PEP 3154, a new and improved protocol for pickled objects * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O To download Python 3.4.0 visit: http://www.python.org/download/releases/3.4.0/ This is a production release. Please report any issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors)

10 10

SSLSocket.send() for non-blocking sockets
by Nikolaus Rath March 26, 2014

March 26, 2014

Hello, I'd like to hear some more opinions on http://bugs.python.org/issue20951. I think the possible courses of action are: 1. Document the current behavior. This has the drawback that (a) it still remains rather counterintuitive, and (b) one still needs extra code to distinguish between a zero-return because the write would block, and a zero-return because the peer has closed the connection. I'm not sure about the best way to do this check. I would probably try to run select() on the underlying raw socket. Is there a better way? 2. Change the behavior immediately, potentially breaking some applications that worked around it, but unbreaking others that relied on the documented behavior. 3. Deprecate the current behavior, and introduce a transition period where the new behavior is controlled by a flag. Drawbacks: no direct breakage of anything, but additional complexity may indirectly cause bugs. Thoughts? Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«

2 2

Re: [Python-Dev] 3.3 branch is now in security fix mode
by april March 25, 2014

March 25, 2014

Sent from my iPhone

1 0