[Python-Dev] PEP 466: Network security enhancements for Python 2.7.7
Guido van Rossum
guido at python.org
Sat Apr 19 01:34:50 CEST 2014
Thanks, Nick. I hereby approve this PEP. You can update the status
On Wed, Apr 16, 2014 at 3:04 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I've reworded the PEP to make it clear it is now just about
> backporting a specific set of enhancements to 2.7.7, as well as
> switching to updating to new OpenSSL feature releases in the binary
> The idea of an open ended backport policy is now listed as a rejected
> variant. I believe that change addresses Guido's main remaining
> concern, so I think this version is ready for pronouncement.
> PEP: 466
> Title: Network Security Enhancements for Python 2.7.7
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>,
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Mar-2014
> Post-History: 23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014,
> Most CPython tracker issues are classified as errors in behaviour or
> proposed enhancements. Most patches to fix behavioural errors are
> applied to all active maintenance branches. Enhancement patches are
> restricted to the default branch that becomes the next Python version.
> This cadence works reasonably well during Python's normal 18-24 month
> feature release cycle, which is still applicable to the Python 3 series.
> However, the age of the standard library in Python 2 has now reached a
> where it is sufficiently far behind the state of the art in network
> protocols for it to be causing real problems in use cases where upgrading
> Python 3 in the near term may not be feasible.
> In recognition of the additional practical considerations that have arisen
> during the 4+ year maintenance cycle for Python 2.7, this PEP allows a
> critical set of network security related features to be backported from
> Python 3.4 to the upcoming Python 2.7.7 maintenance release.
> While this PEP does not make any changes to the core development team's
> handling of security-fix-only branches that are no longer in active
> maintenance, it *does* recommend that commercial redistributors providing
> extended support periods for the Python standard library either backport
> these features to their supported versions, or else explicitly disclaim
> support for the use of older versions in roles that involve connecting
> directly to the public internet.
> New security related features in Python 2.7.7
> Under this proposal, the following features will be backported from Python
> 3.4 to the upcoming Python 2.7.7 maintenance release:
> * in the ``os`` module:
> * persistent file descriptor for ``os.urandom()``.
> * in the ``hmac`` module:
> * constant time comparison function (``hmac.compare_digest()``).
> * in the ``hashlib`` module:
> * password hashing function (``hashlib.pbkdf2_hmac()``).
> * details of hash algorithm availability
> and ``hashlib.algorithms_available``).
> * in the ``ssl`` module:
> * this module is almost entirely synchronised with its Python 3
> counterpart, bringing TLSv1.x settings, SSLContext manipulation, Server
> Name Indication, access to platform certificate stores, standard
> library support for peer hostname validation and more to the Python 2
> * the only ``ssl`` module features *not* backported under this policy are
> the ``ssl.RAND_*`` functions that provide access to OpenSSL's random
> number generation capabilities - use ``os.urandom()`` instead.
> As a general change in maintenance policy, permission is also granted to
> upgrade to newer feature releases of OpenSSL when preparing the binary
> installers for new maintenance releases of Python 2.7.
> This PEP does NOT propose a general exception for backporting new features
> to Python 2.7 - every new feature proposed for backporting will still need
> to be justified independently. In particular, it will need to be explained
> why relying on and independently updated backport on the Python Package
> instead is not an acceptable solution.
> Backwards compatibility considerations
> As in the Python 3 series, the backported ``ssl.create_default_context()``
> API is granted a backwards compatibility exemption that permits the
> protocol, options, cipher and other settings of the created SSL context to
> be updated in maintenance releases to use higher default security settings.
> This allows them to appropriately balance compatibility and security at the
> time of the maintenance release, rather than at the time of the original
> feature release.
> This PEP does *not* grant any other exemptions to the usual backwards
> compatibility policy for maintenance releases. Instead, by explicitly
> encouraging the use of feature based checks, it is designed to make it
> to write more secure cross-version compatible Python software, while still
> limiting the risk of breaking currently working software when upgrading to
> a new Python 2.7 maintenance release.
> In all cases where this proposal allows new features to be backported to
> the Python 2.7 release series, it is possible to write cross-version
> compatible code that operates by "feature detection" (for example, checking
> for particular attributes in a module), without needing to explicitly check
> the Python version.
> It is then up to library and framework code to provide an appropriate
> and fallback behaviour if a desired feature is found to be missing. While
> some especially security sensitive software MAY fail outright if a desired
> security feature is unavailable, most software SHOULD instead emit a
> and continue operating using a slightly degraded security configuration.
> The backported APIs allow library and application code to perform the
> following actions after detecting the presence of a relevant
> network security related feature:
> * explicitly opt in to more secure settings (to allow the use of enhanced
> security features in older maintenance releases of Python with less
> secure default behaviour)
> * explicitly opt in to less secure settings (to allow the use of newer
> feature releases in lower security environments)
> * determine the default setting for the feature (this MAY require explicit
> Python version checks to determine the Python feature release, but DOES
> NOT require checking for a specific maintenance release)
> Security related changes to other modules (such as higher level networking
> libraries and data format processing libraries) will continue to be made
> available as backports and new modules on the Python Package Index, as
> independent distribution remains the preferred approach to handling
> software that must continue to evolve to handle changing development
> requirements independently of the Python 2 standard library. Refer to
> the `Motivation and Rationale`_ section for a review of the characteristics
> that make the secure networking infrastructure worthy of special
> OpenSSL compatibility
> Under this proposal, OpenSSL may be upgraded to more recent feature
> in Python 2.7 maintenance releases. On Linux and most other POSIX systems,
> the specific version of OpenSSL used already varies, as CPython dynamically
> links to the system provided OpenSSL library by default.
> For the Windows binary installers, the ``_ssl`` and ``_hashlib`` modules
> statically linked with OpenSSL and the associated symbols are not exported.
> Marc-Andre Lemburg indicates that updating to newer OpenSSL releases in the
> ``egenix-pyopenssl`` binaries has not resulted in any reported
> issues _
> The Mac OS X binary installers historically followed the same policy as
> other POSIX installations and dynamically linked to the Apple provided
> OpenSSL libraries. However, Apple has now ceased updating these
> cross-platform libraries, instead requiring that even cross-platform
> developers adopt Mac OS X specific interfaces to access up to date security
> infrastructure on their platform. Accordingly, and independently of this
> PEP, the Mac OS X binary installers were already going to be switched to
> statically linker newer versions of OpenSSL _
> Other Considerations
> A number of developers, including Alex Gaynor and Donald Stufft, have
> expressed interest in carrying out the feature backports covered by this
> policy, and assisting with any additional maintenance burdens that arise
> in the Python 2 series as a result.
> Steve Dower and Brian Curtin have offered to help with the creation of the
> Windows installers, allowing Martin von Löwis the opportunity to step back
> from the task of maintaining the 2.7 Windows installer.
> This PEP is primarily about establishing the consensus needed to allow them
> to carry out this work. For other core developers, this policy change
> shouldn't impose any additional effort beyond potentially reviewing the
> resulting patches for those developers specifically interested in the
> affected modules.
> Security releases
> This PEP does not propose any changes to the handling of security
> releases - those will continue to be source only releases that
> include only critical security fixes.
> However, the recommendations for library and application developers are
> deliberately designed to accommodate commercial redistributors that choose
> to apply these changes to additional Python release series that are either
> in security fix only mode, or have been declared "end of life" by the core
> development team.
> Whether or not redistributors choose to exercise that option will be up
> to the individual redistributor.
> Integration testing
> Third party integration testing services should offer users the ability
> to test against multiple Python 2.7 maintenance releases (at least 2.7.6
> and 2.7.7+), to ensure that libraries, frameworks and applications can
> test their handling of the legacy security infrastructure correctly (either
> failing or degrading gracefully, depending on the security sensitivity of
> the software), even after the features covered in this proposal have been
> backported to the Python 2.7 series.
> Handling lower security environments with low risk tolerance
> For better or for worse (mostly worse), there are some environments where
> the risk of latent security defects is more tolerated than even a slightly
> increased risk of regressions in maintenance releases. This proposal
> excludes these environments from consideration where the modules covered by
> the exemption are concerned - this approach is entirely inappropriate for
> software connected to the public internet, and defence in depth security
> principles suggest that it is not appropriate for most private networks
> Downstream redistributors may still choose to cater to such environments,
> but they will need to handle the process of downgrading the security
> related modules and doing the associated regression testing themselves.
> The main CPython continuous integration infrastructure will not cover this
> Motivation and Rationale
> The creation of this PEP was prompted primarily by the aging SSL support in
> the Python 2 series. As of March 2014, the Python 2.7 SSL module is
> approaching four years of age, and the SSL support in the still popular
> Python 2.6 release had its feature set locked six years ago.
> These are simply too old to provide a foundation that can be recommended
> in good conscience for secure networking software that operates over the
> public internet, especially in an era where it is becoming quite clearly
> evident that advanced persistent security threats are even more widespread
> and more indiscriminate in their targeting than had previously been
> understood. While they represented reasonable security infrastructure in
> their time, the state of the art has moved on, and we need to investigate
> mechanisms for effectively providing more up to date network security
> infrastructure for users that, for whatever reason, are not currently in
> a position to migrate to Python 3.
> While the use of the system OpenSSL installation addresses many of these
> concerns on Linux platforms, it doesn't address all of them (in particular,
> it is still difficult for sotware to explicitly require some higher level
> security settings). The standard library support can be bypassed by using a
> third party library like PyOpenSSL or Pycurl, but this still results in a
> security problem, as these can be difficult dependencies to deploy, and
> users will remain unaware that they might want them. Rather than explaining
> to potentially naive users how to obtain and use these libraries, it seems
> better to just fix the included batteries.
> In the case of the binary installers for Windows and Mac OS X that are
> published on python.org, the version of OpenSSL used is entirely within
> the control of the Python core development team, but is currently limited
> to OpenSSL maintenance releases for the version initially shipped with the
> corresponding Python feature release.
> With increased popularity comes increased responsibility, and this proposal
> aims to acknowledge the fact that Python's popularity and adoption is at a
> sufficiently high level that some of our design and policy decisions have
> significant implications beyond the Python development community.
> As one example, the Python 2 ``ssl`` module does not support the Server
> Name Indication standard. While it is possible to obtain SNI support
> by using the third party ``requests`` client library, actually doing so
> currently requires using not only ``requests`` and its embedded
> but also half a dozen or more additional libraries. The lack of support
> in the Python 2 series thus serves as an impediment to making effective
> use of SNI on servers, as Python 2 clients will frequently fail to handle
> it correctly.
> Another more critical example is the lack of SSL hostname matching in the
> Python 2 standard library - it is currently necessary to rely on a third
> party library, such as ``requests`` or ``backports.ssl_match_hostname`` to
> obtain that functionality in Python 2.
> The Python 2 series also remains more vulnerable to remote timing attacks
> on security sensitive comparisons than the Python 3 series, as it lacks a
> standard library equivalent to the timing attack resistant
> ``hmac.compare_digest()`` function. While appropriate secure comparison
> functions can be implemented in third party extensions, many users don't
> even consider the issue and use ordinary equality comparisons instead
> - while a standard library solution doesn't automatically fix that problem,
> it *does* make the barrier to resolution much lower once the problem is
> pointed out.
> Python 2.7 represents the only long term maintenance release the core
> development team has provided, and it is natural that there will be things
> that worked over a historically shorter maintenance lifespan that don't
> over this longer support period. In the specific case of the problem
> described in this PEP, the simplest available solution is to acknowledge
> that long term maintenance of network security related modules *requires*
> the ability to add new features, even while retaining backwards
> for existing interfaces.
> For those familiar with it, it is worth comparing the approach described in
> this PEP with Red Hat's handling of its long term open source support
> commitments: it isn't the RHEL 6.0 release itself that receives 10 years
> worth of support, but the overall RHEL 6 *series*. The individual RHEL 6.x
> point releases within the series then receive a wide variety of new
> features, including security enhancements, all while meeting strict
> backwards compatibility guarantees for existing software. The proposal
> covered in this PEP brings our approach to long term maintenance more into
> line with this precedent - we retain our strict backwards compatibility
> requirements, but make an exception to the restriction against adding new
> To date, downstream redistributors have respected our upstream policy of
> "no new features in Python maintenance releases". This PEP explicitly
> accepts that a more nuanced policy is appropriate in the case of network
> security related features, and the specific change it describes is
> deliberately designed such that it is potentially suitable for Red Hat
> Enterprise Linux and its downstream derivatives.
> Rejected alternative: just advise developers to migrate to Python 3
> This alternative represents the status quo. Unfortunately, it has proven
> to be unworkable in practice, as the backwards compatibility implications
> mean that this is a non-trivial migration process for large applications
> and integration projects. While the tools for migration have evolved to
> a point where it is possible to migrate even large applications
> opportunistically and incrementally (rather than all at once) by updating
> code to run in the large common subset of Python 2 and Python 3, using the
> most recent technology often isn't a priority in commercial environments.
> Previously, this was considered an acceptable harm, as while it was an
> unfortunate problem for the affected developers to have to face, it was
> seen as an issue between them and their management chain to make the case
> for infrastructure modernisation, and this case would become naturally
> more compelling as the Python 3 series evolved.
> However, now that we're fully aware of the impact the limitations of the
> Python 2 standard library may be having on the evolution of internet
> security standards, I no longer believe that it is reasonable to expect
> platform and application developers to resolve all of the latent defects
> in an application's Unicode correctness solely in order to gain access to
> the network security enhancements already available in Python 3.
> While Ubuntu (and to some extent Debian as well) are committed to porting
> default system services and scripts to Python 3, and to removing Python 2
> from its default distribution images (but not from its archives), this is
> a mammoth task and won't be completed for the Ubuntu 14.04 LTS release
> (at least for the desktop image - it may be achieved for the mobile and
> server images).
> Fedora has even more work to do to migrate, and it will take a non-trivial
> amount of time to migrate the relevant infrastructure components. While
> Red Hat are also actively working to make it easier for users to use more
> recent versions of Python on our stable platforms, it's going to take time
> for those efforts to start having an impact on end users' choice of
> and any such changes also don't benefit the core platform infrastructure
> that runs in the integrated system Python by necessity.
> The OpenStack migration to Python 3 is also still in its infancy, and even
> though that's a project with an extensive and relatively robust automated
> test suite, it's still large enough that it is going to take quite some
> to migrate fully to a Python 2/3 compatible code base.
> And that's just three of the highest profile open source projects that
> make heavy use of Python. Given the likely existence of large amounts of
> legacy code that lacks the kind of automated regression test suite needed
> to help support a migration from Python 2 to Python 3, there are likely to
> be many cases where reimplementation (perhaps even in Python 3) proves
> easier than migration. The key point of this PEP is that those situations
> affect more people than just the developers and users of the affected
> application: the existence of clients and servers with outdated network
> security infrastructure becomes something that developers of secure
> networked services need to take into account as part of their security
> design, and that's a problem that inhibits the adoption of better security
> As Terry Reedy noted, if we try to persist with the status quo, the likely
> outcome is that commercial redistributors will attempt to do something
> like this on behalf of their customers *anyway*, but in a potentially
> inconsistent and ad hoc manner. By drawing the scope definition process
> into the upstream project we are in a better position to influence the
> approach taken to address the situation and to help ensure some consistency
> across redistributors.
> The problem is real, so *something* needs to change, and this PEP describes
> my preferred approach to addressing the situation.
> Rejected alternative: create and release Python 2.8
> With sufficient corporate support, it likely *would* be possible to create
> and release Python 2.8 (it's highly unlikely such a project would garner
> enough interest to be achievable with only volunteers). However, this
> wouldn't actually solve the problem, as the aim is to provide a *relatively
> low impact* way to incorporate enhanced security features into integrated
> products and deployments that make use of Python 2.
> Upgrading to a new Python feature release would mean both more work for the
> core development team, as well as a more disruptive update that most
> potential end users would likely just skip entirely.
> Attempting to create a Python 2.8 release would also bring in suggestions
> to backport many additional features from Python 3 (such as ``tracemalloc``
> and the improved coroutine support), making the migration from Python 2.7
> to this hypothetical 2.8 release even riskier and more disruptive.
> This is not a recommended approach, as it would involve substantial
> additional work for a result that is actually less effective in achieving
> the original aim (which is to eliminate the current widespread use of the
> aging network security infrastructure in the Python 2 series).
> Furthermore, while I can't make any commitments to actually addressing
> this issue on Red Hat platforms, I *can* categorically rule out the idea
> of a Python 2.8 being of any use to me in even attempting to get it
> Rejected alternative: distribute the security enhancements via PyPI
> While this initially appears to be an attractive and easier to manage
> approach, it actually suffers from several significant problems.
> Firstly, this is complex, low level, cross-platform code that integrates
> with the underlying operating system across a variety of POSIX platforms
> (including Mac OS X) and Windows. The CPython BuildBot fleet is already set
> up to handle continuous integration in that context, but most of the
> freely available continuous integration services just offer Linux, and
> perhaps paid access to Windows. Those services work reasonably well for
> software that largely runs on the abstraction layers offered by Python and
> other dynamic languages, as well as the more comprehensive abstraction
> offered by the JVM, but won't suffice for the kind of code involved here.
> The OpenSSL dependency for the network security support also qualifies as
> the kind of "complex binary dependency" that isn't yet handled well by the
> ``pip`` based software distribution ecosystem. Relying on a third party
> binary dependency also creates potential compatibility problems for ``pip``
> when running on other interpreters like ``PyPy``.
> Another practical problem with the idea is the fact that ``pip`` itself
> relies on the ``ssl`` support in the standard library (with some additional
> support from a bundled copy of ``requests``, which in turn bundles
> ``backport.ssl_match_hostname``), and hence would require any replacement
> module to also be bundled within ``pip``. This wouldn't pose any
> insurmountable difficulties (it's just another dependency to vendor), but
> it *would* mean yet another copy of OpenSSL to keep up to date.
> This approach also has the same flaw as all other "improve security by
> renaming things" approaches: they completely miss the users who most need
> help, and raise significant barriers against being able to encourage users
> to do the right thing when their infrastructure supports it (since
> "use this other module" is a much higher impact change than "turn on this
> higher security setting"). Deprecating the aging SSL infrastructure in the
> standard library in favour of an external module would be even more user
> hostile than accepting the slightly increased risk of regressions
> with upgrading it in place.
> Last, but certainly not least, this approach suffers from the same problem
> as the idea of doing a Python 2.8 release: likely not solving the actual
> problem. Commercial redistributors of Python are set up to redistribute
> *Python*, and a pre-existing set of additional packages. Getting new
> packages added to the pre-existing set *can* be done, but means approaching
> each and every redistributor and asking them to update their
> repackaging process accordingly. By contrast, the approach described in
> this PEP would require redistributors to deliberately *opt out* of the
> security enhancements by deliberately downgrading the provided network
> security infrastructure, which most of them are unlikely to do.
> Rejected variant: provide a "legacy SSL infrastructure" branch
> Earlier versions of this PEP included the concept of a ``2.7-legacy-ssl``
> branch that preserved the exact feature set of the Python 2.7.6 network
> security infrastructure.
> In my opinion, anyone that actually wants this is almost certainly making a
> mistake, and if they insist they really do want it in their specific
> situation, they're welcome to either make it themselves or arrange for a
> downstream redistributor to make it for them.
> If they are made publicly available, any such rebuilds should be referred
> as "Python 2.7 with Legacy SSL" to clearly distinguish them from the
> Python 2.7 releases that include more up to date network security
> After the first Python 2.7 maintenance release that implements this PEP, it
> would also be appropriate to refer to Python 2.7.6 and earlier releases as
> "Python 2.7 with Legacy SSL".
> Rejected variant: synchronise particular modules entirely with Python 3
> Earlier versions of this PEP suggested synchronising the ``hmac``,
> ``hashlib`` and ``ssl`` modules entirely with their Python 3 counterparts.
> This approach proved too vague to build a compelling case for the
> and has thus been replaced by the current more explicit proposal.
> Rejected variant: open ended backport policy
> Earlier versions of this PEP suggested a general policy change related to
> future Python 3 enhancements that impact the general security of the
> That approach created unnecessary uncertainty, so it has been simplified to
> propose backport a specific concrete set of changes. Future feature
> backport proposals can refer back to this PEP as precedent, but it will
> still be necessary to make a specific case for each feature addition to
> the Python 2.7 long term support release.
> Disclosure of Interest
> The author of this PEP currently works for Red Hat on test automation
> If this proposal is accepted, I will be strongly encouraging Red Hat to
> advantage of the resulting opportunity to help improve the overall security
> of the Python ecosystem. However, I do not speak for Red Hat in this
> and cannot make any commitments on Red Hat's behalf.
> Thanks to Christian Heimes and other for their efforts in greatly improving
> Python's SSL support in the Python 3 series, and a variety of members of
> the Python community for helping me to better understand the implications
> of the default settings we provide in our SSL modules, and the impact that
> tolerating the use of SSL infrastructure that was defined in 2010
> (Python 2.7) or even 2008 (Python 2.6) potentially has for the security
> of the web as a whole.
> Thanks to Donald Stufft and Alex Gaynor for identifying a more limited set
> of essential security features that allowed the proposal to be made more
> fine-grained than backporting entire modules from Python 3.4 [7,8]_.
> Christian and Donald also provided valuable feedback on a preliminary
> draft of this proposal.
> Thanks also to participants in the python-dev mailing list threads
> [1,2,5,6]_, as well as the various folks I discussed this issue with at
> PyCon 2014 in Montreal.
> ..  PEP 466 discussion (round 1)
> ..  PEP 466 discussion (round 2)
> ..  Marc-Andre Lemburg's OpenSSL feedback for Windows
> ..  Ned Deily's OpenSSL feedback for Mac OS X
> ..  PEP 466 discussion (round 3)
> ..  PEP 466 discussion (round 4)
> ..  Donald Stufft's recommended set of backported features
> ..  Alex Gaynor's recommended set of backported features
> This document has been placed in the public domain.
> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
> Python-Dev mailing list
> Python-Dev at python.org
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Python-Dev