[Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7

[Donald Stufft]

On Mar 23, 2014, at 3:07 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Several significant changes in this revision:
> 
> - scope narrowed to just Python 2.7 plus permission for commercial
> redistributors to use the same strategy in their long term support
> releases
> - far more explicit that this is about inviting potential corporate
> contributors to address the situation for the benefit of the overall
> Python ecosystem, not offering to fix it for them for free
> - clarified that third party integration testing services would need
> to be updated to support testing against multiple Python 2.7 minor
> releases

For what it’s worth, I have an outstanding PR against Travis CI that
would make this trivial for them at least. They are a pretty popular
CI service especially for OSS projects. I made that PR for unrelated
reasons but it could at least serve as a template for other projects
to do the same thing.

> - explicit sections on why I don't think the status quo is
> sustainable, why I don't think Python 2.8 would actually solve the
> problem, and why I think a PyPI based solution not only wouldn't solve
> the problem, but would be rather difficult to get working in the first
> place
> - be completely explicit that I am *not* speaking on behalf of Red Hat
> at this point and have no authority to make commitments on their
> behalf. Instead, I'm looking for upstream consensus that 1) this is a
> genuine problem that needs to be solved; 2) we're open to corporate
> assistance in solving it; and 3) we have a pretty good idea what help
> we actually want. If all that happens, *then* I can take up the issue
> internally to try to get us some help in maintaining the proposed
> solution (hopefully other folks with corporate influence can do the
> same, and we may actually get some ongoing assistance with upstream
> maintenance out of this, rather than having our downstream
> redistributors continue to take us for granted).
> 
> Diff: http://hg.python.org/peps/rev/2e82209dda21
> Updated web version: http://www.python.org/dev/peps/pep-0466/
> 
> Advance warning: while I was able to get this revision turned around
> pretty quickly, future revisions are likely to take a fair bit longer.
> It was already a rather busy month before I decided to start this
> discussion on top of everything else :)
> 
> Cheers,
> Nick.
> 
> 
> PEP: 466
> Title: Network Security Enhancement Exception for Python 2.7
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>,
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 23-Mar-2014
> Post-History: 23-Mar-2014
> 
> 
> Abstract
> ========
> 
> Most CPython tracker issues are classified as errors in behaviour or
> proposed enhancements. Most patches to fix behavioural errors are
> applied to all active maintenance branches.  Enhancement patches are
> restricted to the default branch that becomes the next Python version.
> 
> This cadence works reasonably well during Python's normal 18-24 month
> feature release cycle, which is still applicable to the Python 3 series.
> However, the age of the standard library in Python 2 has now reached a point
> where it is sufficiently far behind the state of the art in network security
> protocols for it to be causing real problems in commercial use cases
> where upgrading to Python 3 in the near term may not be practical.
> 
> Accordingly, this PEP relaxes the normal restrictions by allowing
> enhancements to be applied in Python 2.7 maintenance releases for standard
> library components that have implications for the overall security of the
> internet. In particular, the exception will apply to:
> 
> * the ``ssl`` module
> * the ``hashlib`` module
> * the ``hmac`` module
> * the ``sha`` module (Python 2 only)
> * the components of other networking modules that make use of these modules
> * the components of the ``random`` and ``os`` modules that are relevant to
>  cryptographic applications
> * the version of OpenSSL bundled with the binary installers
> 
> Proposed backports for these modules will still need to undergo normal
> backwards compatibility assessments, but new features will be permitted where
> appropriate, making it easier to implement secure networked software in
> Python, even for software that needs to remain compatible with older feature
> releases of Python.
> 
> While this PEP does not make any changes to the core development team's
> handling of security-fix-only branches that are no longer in active
> maintenance, it *does* recommend that commercial redistributors providing
> extended support periods for the Python standard library either adopt a
> similar approach to ensuring that the secure networking infrastructure
> keeps pace with the evolution of the internet, or else disclaim support
> for the use of older versions in roles that involving connecting
> directly to the public internet.
> 
> 
> Exemption Policy
> ================
> 
> Under this policy, the following network security related modules are
> granted a blanket exemption to the restriction against adding new features
> in maintenance releases, for the purpose of keeping their APIs aligned with
> their counterparts in the latest feature release of Python 3:
> 
> * the ``ssl`` module
> * the ``hashlib`` module
> * the ``hmac`` module
> * the ``sha`` module (Python 2 only)
> 
> This exemption applies to *all* proposals to backport backwards compatible
> changes in these modules to Python 2.7 maintenance releases. This choice is
> made deliberately to ensure that the "feature or fix?" argument isn't simply
> replaced by a "security related or not?" argument. These particular modules
> are inherently security related, and all enhancements to them improve
> Python's capabilities as a platform for development of secure networked
> software.
> 
> As part of this policy, permission is also granted to upgrade to newer
> feature releases of OpenSSL when preparing the binary installers
> for new maintenance releases of CPython.
> 
> In addition to the above blanket exemption, a conditional exemption is
> granted for these modules that may include some network security related
> features:
> 
> * the ``os`` module (specifically ``os.urandom``)
> * the ``random`` module
> * networking related modules that depend on one or more of the network
>  security related modules listed above
> 
> This more limited exemption for these modules requires that the *specific*
> enhancement being proposed for backporting needs to be justified as being
> network security related. If the enhancement under discussion is designed
> to take advantage of a new feature in one of the network security related
> modules, then that will be taken as implying that the enhancement is
> security related.
> 
> 
> Backwards Compatibility Considerations
> ======================================
> 
> This PEP does *not* grant any general exemptions to the usual backwards
> compatibility policy for maintenance releases. Instead, by explicitly
> encouraging the use of feature based checks and explicitly opting in to
> less secure configurations, it is designed to make it easier to provide
> more "secure by default" behaviour in future feature releases, while still
> limiting the risk of breaking currently working software when upgrading to
> a new maintenance release.
> 
> In *all* cases where this policy is applied to backport enhancements to
> maintenance releases, it MUST be possible to write cross-version compatible
> code that operates by "feature detection" (for example, checking for
> particular attributes in the module), without needing to explicitly check
> the Python version.
> 
> It is then up to library and framework code to provide an appropriate warning
> and fallback behaviour if a desired feature is found to be missing. While
> some especially security sensitive software MAY fail outright if a desired
> security feature is unavailable, most software SHOULD instead continue
> operating using a slightly degraded security configuration.
> 
> Affected APIs SHOULD be designed to allow library and application code to
> perform the following actions after detecting the presence of a relevant
> network security related feature:
> 
> * explicitly opt in to more secure settings (to allow the use of enhanced
>  security features in older maintenance releases of Python)
> * explicitly opt in to less secure settings (to allow the use of newer Python
>  feature releases in lower security environments)
> * determine the default setting for the feature (this MAY require explicit
>  Python version checks to determine the Python feature release, but MUST
>  NOT require checking for a specific maintenance release)
> 
> Security related changes to other modules (such as data format processing
> libraries) will continue to be made available as backports and new modules
> on the Python Package Index, as independent distribution remains the
> preferred approach to handling software that needs to evolve faster than
> the standard library. Refer to the `Motivation and Rationale`_ section for
> a review of the characteristics that make the secure networking
> infrastructure worthy of special consideration.
> 
> 
> Other Considerations
> ====================
> 
> Maintainability
> ---------------
> 
> This policy does NOT represent a commitment by volunteer contributors to
> actually backport network security related changes from the Python 3 series
> to the Python 2 series. Rather, it is intended to send a clear signal to
> potential corporate contributors that the core development team are willing
> to review and merge corporate contributions that put this policy into
> effect.
> 
> Backporting security related fixes and enhancements to earlier versions is
> a common service for commercial redistributors to offer to their customers.
> This policy represents an explicit invitation to contribute some of those
> changes back to the upstream community in cases where they are likely to
> have a broad impact that helps improve the security of the internet as a
> whole.
> 
> 
> Documentation
> -------------
> 
> All modules that take advantage of this policy to backport network
> security related enhancements to earlier Python versions MUST include
> a "Security Considerations" section in their documentation.
> 
> In addition to any other module specific contents, this section MUST
> enumerate key security enhancements and fixes (with CVE identifiers where
> applicable), along with the feature and maintenance releases that first
> included them.
> 
> 
> Security releases
> -----------------
> 
> This PEP does not propose any changes to the handling of security
> releases - those will continue to be source only releases that
> include only critical security fixes.
> 
> However, the recommendations for library and application developers are
> deliberately designed to accommodate commercial redistributors applying
> this policy to any Python release series that is either in security
> fix only mode, or has been declared "end of life" by the core development
> team.
> 
> Whether or not redistributors choose to exercise that option will be up
> to the redistributor.
> 
> 
> Integration testing
> -------------------
> 
> Third party integration testing services would likely need to start
> offering users a choice of multiple Python 2.7.x versions to test against,
> to ensure that the application is correctly degrading gracefully if it
> attempts to use newer networking features on maintenance releases that
> are too old to provide them.
> 
> 
> Evolution of this Policy
> ========================
> 
> The key requirement for a module to be considered for inclusion in this
> policy (whether under a blanket or conditional exemption) is that it must
> have security implications *beyond* the specific application that is written
> in Python and the system that application is running on. Thus the focus on
> network security protocols and related cryptographic infrastructure - Python
> is a popular choice for the development of web services and clients, and
> thus the capabilities of widely used Python versions have implications for
> the security design of other services that may be using newer versions of
> Python or other development languages.
> 
> The intent behind this requirement is to minimise any impact that the
> introduction of this policy may have on the stability and compatibility of
> maintenance releases. It would be thoroughly counterproductive if end
> users became as cautious about updating to new Python maintenance releases
> as they are about updating to new feature releases.
> 
> 
> Motivation and Rationale
> ========================
> 
> This PEP can be seen as a more targeted version of the "faster standard
> library release cycle" proposals discussed in PEP 407 and PEP 413,
> focusing specifically on those areas which have implications beyond the
> Python community.
> 
> 
> Background
> ----------
> 
> The creation of this PEP was prompted primarily by the aging SSL support in
> the Python 2 series. As of March 2014, the Python 2.7 SSL module is
> approaching four years of age, and the SSL support in the still popular
> Python 2.6 release had its feature set locked six years ago.
> 
> These are simply too old to provide a foundation that can be recommended
> in good conscience for secure networking software that operates over the
> public internet, especially in an era where it is becoming quite clearly
> evident that advanced persistent security threats are even more widespread
> and more indiscriminate in their targeting than had previously been
> understood. While they represented reasonable security infrastructure in
> their time, the state of the art has moved on, and we need to investigate
> mechanisms for effectively providing more up to date network security
> infrastructure for users that, for whatever reason, are not currently in
> a position to migrate to Python 3.
> 
> While the use of the system OpenSSL installation addresses many of these
> concerns on Linux platforms, it doesn't address all of them, and in the
> case of the binary installers for Windows and Mac OS X that are published
> on python.org, the version of OpenSSL used is entirely within the control
> of the Python core development team, and currently limited to OpenSSL
> maintenance releases for the version initially shipped with the corresponding
> Python feature release.
> 
> With increased popularity comes increased responsibility, and this policy
> aims to acknowledge the fact that Python's popularity and adoption has now
> reached a level where some of our design and policy decisions have
> significant implications beyond the Python development community.
> 
> As one example, the Python 2 ``ssl`` module does not support the Server
> Name Identification standard. While it is possible to obtain SNI support
> by using the third party ``requests`` client library, actually doing so
> currently requires using not only ``requests`` and its embedded dependencies,
> but also half a dozen or more additional libraries. The lack of support
> in the Python 2 series thus serves as an impediment to making effective
> use of SNI on servers, as Python 2 clients will frequently fail to handle
> it correctly.
> 
> Another more critical example is the lack of SSL hostname matching in the
> Python 2 standard library - it is currently necessary to rely on a third
> party library, such as ``requests`` or ``backports.ssl_match_hostname`` to
> obtain that functionality in Python 2.
> 
> The Python 2 series also remains more vulnerable to remote timing attacks
> on security sensitive comparisons than the Python 3 series, as it lacks a
> standard library equivalent to the timing attack resistant
> ``hmac.compare_digest()`` function. While appropriate secure comparison
> functions can be implemented in third party extensions, may users don't
> even consider the problem and use ordinary equality comparisons instead
> - while a standard library solution doesn't automatically fix that problem,
> it *does* make the barrier to resolution much lower once the problem is
> pointed out.
> 
> My position on the ongoing transition from Python 2 to Python 3 has long
> been that Python 2 remains a supported platform for the core development
> team, and that commercial support will remain available well after upstream
> maintenance ends. However, in the absence of this network security
> enhancement policy, that position is difficult to justify when it comes to
> software that operates over the public internet. Just as many developers
> consider it too difficult to develop truly secure modern networked software
> in C/C++ (largely due to the challenges associated with manual
> memory management), I anticipate that in the not too distant future, it
> will be considered too difficult to develop truly secure modern networked
> software using the Python 2 series (some developers would argue that we
> have already reached that point).
> 
> 
> Alternative: advise developers of networked software to migrate to Python 3
> ---------------------------------------------------------------------------
> 
> This alternative represents the status quo. Unfortunately, it has proven
> to be unworkable in practice, as the backwards compatibility implications
> mean that this is a non-trivial migration process for large applications
> and integration projects.
> 
> Now that we're fully aware of the impact the limitations in Python 2 may be
> having on the evolution of internet security standards, I no longer believe
> that it is reasonable to expect platform and application developers to
> resolve all of the latent defects in an application's Unicode correctness
> solely in order to gain access to the network security enhancements
> available in Python 3.
> 
> While (as far as I am aware) Ubuntu has successfully switched to Python 3.4
> as its main Python interpreter for its 14.04 LTS release, Fedora still
> has a lot of work to do to migrate, and it will take a non-trivial amount
> of time to migrate the relevant infrastructure components. While Red Hat
> are also actively working to make it easier for users to use more recent
> versions of Python on our stable platforms, it's going to take time for
> those efforts to start having an impact on end users' choice of version,
> and those changes won't affect the core tools regardless.
> 
> The OpenStack migration to Python 3 is also still in its infancy, and even
> though that's a project with an extensive and relatively robust automated
> test suite, it's large enough that it is going to take quite some time
> to migrate.
> 
> And that's just three of the highest profile open source projects that
> make heavy use of Python. Given the likely existence of large amounts of
> legacy code that lacks the kind of automated regression test suite needed
> to help support a migration from Python 2 to Python 3. The key point of
> this PEP is that those situations affect more people than just the
> developers and users of the affected application: their existence becomes
> something that developers of secure networked services need to take into
> account as part of their security design.
> 
> As Terry Reedy noted, if we try to persist with the status quo, the likely
> outcome is that commercial redistributors will attempt to do something
> like this on behalf of their customers *anyway*, but in a potentially
> inconsistent and ad hoc manner. By drawing the scope definition process
> into the upstream project we are in a better position to influence the
> approach taken to address the situation and to help ensure some consistency
> across redistributors.
> 
> The problem is real, so *something* needs to change, and this PEP describes
> my currently preferred approach to addressing the situation.
> 
> 
> Alternative: create and release Python 2.8
> ------------------------------------------
> 
> With sufficient corporate support, it likely *would* be possible to create
> and release Python 2.8 (it's highly unlikely such a project would garner
> enough interest to be achievable with only volunteers). However, this
> wouldn't actually solve the problem, as the aim is to provide a *relatively
> low impact* way to incorporate enhanced security features into integrated
> products and deployments that make use of Python 2. Upgrading to a new
> Python feature release would mean both more work for the core development
> team, as well as a more disruptive update that most potential end users
> would likely just skip entirely.
> 
> Attempting to create a Python 2.8 release would also bring in suggestions
> to backport many additional features from Python 3 (such as ``tracemalloc``
> and the improved coroutine support).
> 
> This is not a recommended approach, as it would involve substantial
> additional work for a result that is actually less effective as a solution
> to the original problem (the widespread use of the aging network security
> infrastructure in Python 2).
> 
> 
> Alternative: distribute the security enhancements via PyPI
> ----------------------------------------------------------
> 
> While it initially appears to be an attractive and easier to manage
> approach, there are actually several significant problems with this
> idea.
> 
> Firstly, this PEP encompasses a non-trivial portion of the standard library.
> It's not just the underlying SSL support, but also the libraries for other
> network protocols like HTTP, FTP, IMAP, and POP3 that integrate with the
> SSL infrastructure to provide secure links, and that's just the protocols
> in the standard library.  Even if an API compatible ``ssl2`` module was
> made available, it would need to be imported and injected
> into ``sys.modules`` as ``ssl`` before importing any other module that
> needed it.
> 
> Secondly, this is complex, low level, cross-platform code that integrates
> with the underlying operating system across a variety of POSIX platforms
> (including Mac OS X) and Windows. The CPython BuildBot fleet is already set
> up to handle continuous integration in that context, but most of the
> freely available continuous integration services just offer Linux, and
> perhaps paid access to Windows. Those services work reasonably well for
> software that largely runs on the abstraction layers offered by Python and
> other dynamic languages, but won't suffice for the kind of code involved
> here.
> 
> The OpenSSL dependency for the network security support also qualifies as
> the kind of "complex binary dependency" that isn't yet handled well by the
> ``pip`` based software distribution ecosystem. Relying on a binary
> dependency also creates potential compatibility problems for ``pip`` when
> running on other interpreters like ``PyPy``.
> 
> Another practical problem with the idea is the fact that ``pip`` itself
> relies on the ``ssl`` support in the standard library (with some additional
> support from a bundled copy of ``requests``, which in turn bundles
> ``backport.ssl_match_hostname``), and hence would require any replacement
> module to also be bundled within ``pip``. This wouldn't pose any
> insurmountable difficulties (it's just another dependency to vendor), but
> it *would* mean yet another copy of OpenSSL to keep up to
> date.
> 
> This approach also has the same flaw as all other "improve security by
> renaming things" approaches: they completely miss the users who most need
> help, and raise significant barriers against being able to encourage users
> to do the right thing when their infrastructure supports it (since
> "use this other module" is a much higher impact change than "turn on this
> higher security setting"). Deprecating the aging SSL infrastructure in the
> standard library in favour of an external module would be even more user
> hostile than taking the risk of trying to upgrade it in place.
> 
> Last, but certainly not least, this approach suffers from the same problem
> as the idea of doing a Python 2.8 release: likely not solving the actual
> problem. Commercial redistributors of Python are set up to redistribute
> *Python*, and a pre-existing set of additional packages. Getting new
> packages added to the pre-existing set *can* be done, but means going
> around to each and every redistributor and asking them to update their
> repackaging process accordingly. By contrast, the approach described in
> this PEP would require redistributors to *opt out* of the security
> enhancements, which most of them are unlikely to do.
> 
> 
> Open Questions
> ==============
> 
> * What are the risks associated with allowing OpenSSL to be updated to
>  new feature versions in the Windows and Mac OS X binary installers for
>  maintenance releases? Currently we just upgrade to the appropriate
>  OpenSSL maintenance releases, rather than switching to the latest
>  feature release. In particular, is it possible Windows C extensions may
>  be linking against the Python provided OpenSSL module?
> 
> * Are there any other security relevant modules that should be covered
>  by either a blanket or conditional exemption?
> 
> 
> Disclosure of Interest
> ======================
> 
> The author of this PEP currently works for Red Hat on test automation tools.
> If this proposal is accepted, I will be strongly encouraging Red Hat to take
> advantage of the resulting opportunity to help improve the overall security
> of the Python ecosystem. However, I do not speak for Red Hat in this matter,
> and cannot make any commitments on Red Hat's behalf.
> 
> 
> Acknowledgements
> ================
> 
> Thanks to Christian Heimes for his recent efforts on greatly improving
> Python's SSL support in the Python 3 series, and a variety of members of
> the Python community for helping me to better understand the implications
> of the default settings we provide in our SSL modules, and the impact that
> tolerating the use of SSL infrastructure that was defined in 2010
> (Python 2.7) or even 2008 (Python 2.6) potentially has for the security
> of the web as a whole.
> 
> Christian and Donald Stufft also provided valuable feedback on a preliminary
> draft of this proposal.
> 
> Thanks also to participants in the python-dev mailing list thread [1]_
> 
> 
> References
> ==========
> 
> .. [1] https://mail.python.org/pipermail/python-dev/2014-March/133334.html
> 
> 
> Copyright
> =========
> 
> This document has been placed in the public domain.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140323/51def55b/attachment-0001.sig>