On 22 March 2014 at 21:11:24, Nick Coghlan (ncoghlan@gmail.com(mailto:ncoghlan@gmail.com)) wrote:

Folks,

I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements. I now see these as categorically different from most other enhancements, as they have implications for the evolution of internet security as a whole, rather than being limited to affecting the security of individual applications written in Python.

I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 is a security liability, any it becomes nothing short of irresponsible to use Python 2.7 for any form of networking code that hits the open internet. On top of that, the current state of the ssl module means that Python 2.7 and earlier cannot ever support a standard-compliant implementation of things like HTTP/2. That’s a fairly tragic state of affairs for 2.7, especially if we’re supposed to claim with a straight face that it’s acceptable to still use Python 2.7. Treat this as my strong +1. Additionally, I’m happy to volunteer my time and limited expertise to help make this happen. I’ll help work on back porting things, review code, write docs: whatever it takes to get this to happen.

On Sat, Mar 22, 2014, at 14:11, Nick Coghlan wrote:

Folks,

I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements.

I think the PEP should also address "security-mode" releases. Do the same exceptions apply? Does anyone really want to backport features to Python 3.1?

I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged.

On Mar 22, 2014, at 5:32 PM, Benjamin Peterson wrote:

Does anyone really want to backport features to Python 3.1?

Am 22.03.14 22:17, schrieb Cory Benfield:

I am 100%, overwhelmingly in favour of this. Without this PEP, Python 2.7 is a security liability, any it becomes nothing short of irresponsible to use Python 2.7 for any form of networking code that hits the open internet.

Agreed (although this might be a slight overstatement if taken literally). However, the right consequence should be to use Python 3.4 instead. Regards, Martin

Am 22.03.14 23:05, schrieb Donald Stufft:

I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged.

Does it actually? Unfortunately, I never got around to writing the PEP on security-only branches, so the terminology of PEP 466 is unclear. It says "maintenance brances", which could mean three things: - all branches that have ever been made off default in hg.python.org/cpython (i.e. back to 2.0). This would be the traditional definition of "maintenance branch" in a version control system, but is probably not intended by the PEP - all branches from which still releases are made (whether security-only or bug fix) - all bug fix branches Clarification of this is just a minor nit, though - although the maintainers of the old maintenance branches should (IMO) have a say whether they accept a certain change on a certain branch. Regards, Martin

Am 22.03.14 23:21, schrieb Donald Stufft:

"Just use Python 3.4" ignores the reality of production software. I wish it were that simple because I love 3.4

But I think so does the PEP (i.e. ignore the reality of production software). The risk of breaking existing installations (which might not be affected by the purported security threat being fixed) is just too high. Users that feel threatened by the lack of SSL features in Python should have long started planning on how to get to Python 3. It's not that they haven't been warned. Regards, Martin

On 23 March 2014 08:23, "Martin v. Löwis" wrote:

Am 22.03.14 23:05, schrieb Donald Stufft:

...

I think the pep doesn't mandate that someone does. It still requires someone to care enough to actually write the patch. It just allows such a patch to be merged.

Does it actually? Unfortunately, I never got around to writing the PEP on security-only branches, so the terminology of PEP 466 is unclear. It says "maintenance brances", which could mean three things: - all branches that have ever been made off default in hg.python.org/cpython (i.e. back to 2.0). This would be the traditional definition of "maintenance branch" in a version control system, but is probably not intended by the PEP - all branches from which still releases are made (whether security-only or bug fix) - all bug fix branches

Clarification of this is just a minor nit, though - although the maintainers of the old maintenance branches should (IMO) have a say whether they accept a certain change on a certain branch.

Both good points, I'll clarify them in the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Am 22.03.14 23:33, schrieb Nick Coghlan:

Hard to maintain legacy software is a fact of life, and way too much of it is exposed to the public internet. This PEP is about doing what we can to mitigate the damage caused both by other people's mistakes, and also the inherent challenges of migrating from the error prone POSIX text model to something more reasonable.

I *don't* think its reasonable to expect us to do this without support from the corporate users that caused the problem in the first place (by continuing to deploy older versions of Python without investing adequately in their upkeep), so I'd encourage everyone employed by a commercial user of Python to remind their management chains of the risks of failing to invest development time in any upstream dependencies that they expect to keep pace with the dynamic nature of the internet.

I hope indeed you are successful in activating resources. However, putting them on this backporting project seems like a waste. They should rather go into porting stuff to 3.x where people need it. As responsible maintainers, we should just advise our users that Python 2.7 is a dead horse, and that they should stop riding it. More professionally, we should set an official end-of-life date for 2.7 (alas, we should have done that two years ago). I hope that the language summit can agree to stopping bug fix releases for 2.7 in 2014. Regards, Martin

I agree wholeheartedly with the sentiment behind this PEP, but I have concerns about the implementation. If we introduce new APIs into the ssl module then we will see packages and applications that depend on Python 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a mess unless adoption of new Python minor versions is swift and consistent. The PEP says that new features should be testable with feature detection, but how far does that go? The Python 3 NPN interface is a method of SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext interface in a minor release, or introduce a new NPN interface that is not based on SSLContext just for older Pythons? In my experience most deployments of Python apps have better systems in place to deal with packages installed from PyPI than new versions of Python itself. Therefore it might be better to follow the lead of the unittest2 module and introduce a new ssl2 module that tracks the most recent version of Python's ssl module. My counter-proposal is: * Introduce a new module 'ssl2' which is essentially a backport of the current release's ssl module for all old versions of Python that anyone cares to support. * Third-party applications and libraries targeting anything but the latest version of Python are advised to use ssl2 instead of ssl (this might also be a good opportunity to bless certifi or an equivalent). * Minor releases of Python are allowed to make any security-related change that does not change any public interfaces (e.g. linking against a new version of openssl, or disabling SSLv2) * New interfaces in security-related modules are allowed only if those new interfaces are used by other stdlib modules (since stdlib modules are the only place where ssl2 is off-limits). For example, a backport of match_hostname could go in if httplib or urllib used it. NPN/ALPN would not be allowed under this rule since there is no stdlib code that uses them; third-party SPDY or HTTP/2 code would have to use ssl2 to get this interface. In this world, libraries like requests or tornado that want to use advanced ssl features can simply declare a dependency on ssl2, instead of saying "we require Python 2.7.7+, 3.2.6+, etc". In spite of my reservations about this PEP, I'd still love to see SSL improvements in 2.7 in any form. I'm currently planning a port for Tornado's ssl usage from the stdlib ssl module to PyOpenSSL (in order to get SNI and NPN/ALPN features), and I'd really rather not have to convert to a completely incompatible interface. -Ben On Sat, Mar 22, 2014 at 5:11 PM, Nick Coghlan wrote:

Folks,

I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements. I now see these as categorically different from most other enhancements, as they have implications for the evolution of internet security as a whole, rather than being limited to affecting the security of individual applications written in Python.

It is similar to the previous IDLE policy exception PEP, where we decided that cross version consistency of IDLE superseded the general policy against backporting enhancements to maintenance branches.

This new PEP proposes a similar exception for network security related changes, such that the *capabilities* of all still maintained versions should be kept equivalent, with only the *default settings* varying across up to date feature releases. The policy is also designed to make it easier for us to ratchet up the default security settings in future feature releases.

However, it's more complex than the IDLE PEP (since it's more invasive), and there are still a few open questions listed. This topic involves a complex balance between encouraging and supporting good security practices and limiting the risk of failures for users upgrading to new maintenance releases, so I'd ask that folks take time to read and consider the implications of the full PEP in the broader context of today's internet before posting comments on specific details, or indicating a preference one way or the other in terms of the overall proposal.

Full PEP included inline below, and available in more readable form at http://www.python.org/dev/peps/pep-0466/

Regards, Nick.

PEP: 466 Title: Network Security Enhancement Exception for All Branches Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014

Abstract ========

Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version.

This PEP relaxes the latter restriction allowing enhancements to be applied to maintenance branches for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to:

* the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers

Changes to these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python.

Exemption Policy ================

Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases:

* the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only)

This exemption applies to *all* proposals to backport backwards compatible features in these modules to active maintenance branches. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software.

As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython.

In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features:

* the ``os`` module (primarily ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above

This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related.

Backwards Compatibility Considerations ======================================

This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release.

In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version.

It is then up to library and framework code to provide an appropriate error message or fallback behaviour if a desired feature is found to be missing.

Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature:

* explicitly opt in to more secure settings (to allow the use of enhanced security features in older versions of Python) * explicitly opt in to less secure settings (to allow the use of newer Python versions in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT depend on the specific maintenance release)

Documentation Requirements ==========================

All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation.

In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), and the

Evolution of this Policy ========================

The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages.

The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases.

Motivation and Rationale ========================

This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community.

The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago.

These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood.

While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team.

With increased popularity comes increased responsibility, and this policy is about recognising the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community.

As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly.

Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2.

The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out.

My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I consider it too difficult to develop truly secure modern networked software using the Python 2 series without introducing this network security enhancement policy, as doing so would mean reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts.

Requiring that latent defects in an application's Unicode correctness be addressed in order to migrate to Python 3 is not a reasonable alternative recommendation, especially given the likely existence of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. By making it more feasible to enhance the security of the Python 2 series, we can help contribute to the evolution of a more secure internet for all concerned.

For the Python 2 series, this proposal will most obviously impact the remaining maintenance releases of Python 2.7. However, commercial redistributors that continue to offer full support for Python 2.6, and will also do so for Python 2.7 after upstream maintenance ends, may choose to take this policy into account when deciding what changes to backport to their own maintenance updates. Providing commercial redistributors such as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and Red Hat that option is one of the benefits offered by requiring feature detection based forward compatibility for the modules covered by this policy.

Open Questions ==============

* What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases?

* Are there any other security relevant modules that should be covered by either a blanket or conditional exemption?

* Should we enumerate a specific list of "other networking modules" rather than leaving it implicit?

Acknowledgement ===============

Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole.

Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal.

Copyright =========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com

On Sat, Mar 22, 2014 at 11:26 PM, "Martin v. Löwis" wrote:

Am 22.03.14 23:21, schrieb Donald Stufft:

...

"Just use Python 3.4" ignores the reality of production software. I wish it were that simple because I love 3.4

But I think so does the PEP (i.e. ignore the reality of production software). The risk of breaking existing installations (which might not be affected by the purported security threat being fixed) is just too high.

Users that feel threatened by the lack of SSL features in Python should have long started planning on how to get to Python 3. It's not that they haven't been warned.

Or not rely on the standard library for their security. Much as I realize it is necessary for rudimentary SSL support (for example) to exist in the standard library, any security-aware user is much better served by third-party packages they can update more easily than Python. I really wish we had no SSL *server* support in the standard library at all (but then I wish we didn't have so many middling-quality servers in the stdlib to start with :) Even when we're not considering virtualenvs, which are a great tool for preventing upgrades from breaking things, separately-installed packages (be they third-party or "official" python.org packages) are much easier to upgrade than whole Python installations. And a third-party package like PyOpenSSL has much more freedom in dealing with backward-incompatible changes (because users updating PyOpenSSL are much more likely to realize something regarding SSL could break, even if they aren't more likely to read the release documentation.) I think I'm +0 on the PEP: I understand Martin's concerns, and share them (heck, I'm preparing for an update from Python 2.7.3 to 2.7.6 at work and I'm already having nightmares) but I also understand the problem the PEP tries to solve. I wish we would separate out e.g. the SSL-related modules so they aren't part of the 'core' Python but not quite third-party packages either, but that's not something that's easily done. I do believe that besides accepting this PEP we should provide good guidance on how to write portable, secure software using the tools provided, say in a new chapter in the library docs -- and it shouldn't be afraid to suggest third-party modules if they are a much better solution :) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread!

As someone who is deeply biased towards improving the packaging tool chain and getting people to use it I think that most people will simply use the Stdlib even if a more secure alternative exists. Infact one does exist and I still see almost everyone using the stdlib ssl instead of pyopenssl. At best they have an optional dependency on it which many people who aren't security conscious won't even realize why they should install it.

On Mar 22, 2014, at 6:53 PM, Ben Darnell wrote:

I agree wholeheartedly with the sentiment behind this PEP, but I have concerns about the implementation. If we introduce new APIs into the ssl module then we will see packages and applications that depend on Python 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a mess unless adoption of new Python minor versions is swift and consistent. The PEP says that new features should be testable with feature detection, but how far does that go? The Python 3 NPN interface is a method of SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext interface in a minor release, or introduce a new NPN interface that is not based on SSLContext just for older Pythons?

In my experience most deployments of Python apps have better systems in place to deal with packages installed from PyPI than new versions of Python itself. Therefore it might be better to follow the lead of the unittest2 module and introduce a new ssl2 module that tracks the most recent version of Python's ssl module.

My counter-proposal is: * Introduce a new module 'ssl2' which is essentially a backport of the current release's ssl module for all old versions of Python that anyone cares to support. * Third-party applications and libraries targeting anything but the latest version of Python are advised to use ssl2 instead of ssl (this might also be a good opportunity to bless certifi or an equivalent). * Minor releases of Python are allowed to make any security-related change that does not change any public interfaces (e.g. linking against a new version of openssl, or disabling SSLv2) * New interfaces in security-related modules are allowed only if those new interfaces are used by other stdlib modules (since stdlib modules are the only place where ssl2 is off-limits). For example, a backport of match_hostname could go in if httplib or urllib used it. NPN/ALPN would not be allowed under this rule since there is no stdlib code that uses them; third-party SPDY or HTTP/2 code would have to use ssl2 to get this interface.

In this world, libraries like requests or tornado that want to use advanced ssl features can simply declare a dependency on ssl2, instead of saying "we require Python 2.7.7+, 3.2.6+, etc".

In spite of my reservations about this PEP, I'd still love to see SSL improvements in 2.7 in any form. I'm currently planning a port for Tornado's ssl usage from the stdlib ssl module to PyOpenSSL (in order to get SNI and NPN/ALPN features), and I'd really rather not have to convert to a completely incompatible interface.

-Ben

...

On Sat, Mar 22, 2014 at 5:11 PM, Nick Coghlan wrote: Folks,

I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements. I now see these as categorically different from most other enhancements, as they have implications for the evolution of internet security as a whole, rather than being limited to affecting the security of individual applications written in Python.

It is similar to the previous IDLE policy exception PEP, where we decided that cross version consistency of IDLE superseded the general policy against backporting enhancements to maintenance branches.

This new PEP proposes a similar exception for network security related changes, such that the *capabilities* of all still maintained versions should be kept equivalent, with only the *default settings* varying across up to date feature releases. The policy is also designed to make it easier for us to ratchet up the default security settings in future feature releases.

However, it's more complex than the IDLE PEP (since it's more invasive), and there are still a few open questions listed. This topic involves a complex balance between encouraging and supporting good security practices and limiting the risk of failures for users upgrading to new maintenance releases, so I'd ask that folks take time to read and consider the implications of the full PEP in the broader context of today's internet before posting comments on specific details, or indicating a preference one way or the other in terms of the overall proposal.

Full PEP included inline below, and available in more readable form at http://www.python.org/dev/peps/pep-0466/

Regards, Nick.

PEP: 466 Title: Network Security Enhancement Exception for All Branches Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan , Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Mar-2014 Post-History: 23-Mar-2014

Abstract ========

Most CPython tracker issues are classified as errors in behaviour or proposed enhancements. Most patches to fix behavioural errors are applied to all active maintenance branches. Enhancement patches are restricted to the default branch that becomes the next Python version.

This PEP relaxes the latter restriction allowing enhancements to be applied to maintenance branches for standard library components that have implications for the overall security of the internet. In particular, the exception will apply to:

* the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers

Changes to these modules will still need to undergo normal backwards compatibility assessments, but new features will be permitted where appropriate, making it easier to implement secure networked software in Python, even for software that needs to remain compatible with older feature releases of Python.

Exemption Policy ================

Under this policy, the following network security related modules are granted a blanket exemption to the restriction against adding new features in maintenance releases:

* the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only)

This exemption applies to *all* proposals to backport backwards compatible features in these modules to active maintenance branches. This choice is made deliberately to ensure that the "feature or fix?" argument isn't simply replaced by a "security related or not?" argument. These particular modules are inherently security related, and all enhancements to them improve Python's capabilities as a platform for development of secure networked software.

As part of this policy, permission is also granted to upgrade to newer feature releases of OpenSSL when preparing the binary installers for new maintenance releases of CPython.

In addition to the above blanket exemption, a conditional exemption is granted for these modules that may include some network security related features:

* the ``os`` module (primarily ``os.urandom``) * the ``random`` module * networking related modules that depend on one or more of the network security related modules listed above

This more limited exemption for these modules requires that the *specific* enhancement being proposed for backporting needs to be justified as being network security related. If the enhancement under discussion is designed to take advantage of a new feature in one of the network security related modules, then that will be taken as implying that the enhancement is security related.

Backwards Compatibility Considerations ======================================

This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release.

In *all* cases where this policy is applied to backport enhancements to maintenance releases, it MUST be possible to write cross-version compatible code that operates by "feature detection" (for example, checking for particular attributes in the module), without needing to explicitly check the Python version.

It is then up to library and framework code to provide an appropriate error message or fallback behaviour if a desired feature is found to be missing.

Affected APIs SHOULD be designed to allow library and application code to perform the following actions after detecting the presence of a relevant network security related feature:

* explicitly opt in to more secure settings (to allow the use of enhanced security features in older versions of Python) * explicitly opt in to less secure settings (to allow the use of newer Python versions in lower security environments) * determine the default setting for the feature (this MAY require explicit Python version checks to determine the Python feature release, but MUST NOT depend on the specific maintenance release)

Documentation Requirements ==========================

All modules that take advantage of this policy to backport network security related enhancements to earlier Python versions MUST include a "Security Considerations" section in their documentation.

In addition to any other module specific contents, this section MUST enumerate key security enhancements and fixes (with CVE identifiers where applicable), and the

Evolution of this Policy ========================

The key requirement for a module to be considered for inclusion in this policy (whether under a blanket or conditional exemption) is that it must have security implications *beyond* the specific application that is written in Python and the system that application is running on. Thus the focus on network security protocols and related cryptographic infrastructure - Python is a popular choice for the development of web services and clients, and thus the capabilities of widely used Python versions have implications for the security design of other services that may be using newer versions of Python or other development languages.

The intent behind this requirement is to minimise any impact that the introduction of this policy may have on the stability and compatibility of maintenance releases. It would be thoroughly counterproductive if end users became as cautious about updating to new Python maintenance releases as they are about updating to new feature releases.

Motivation and Rationale ========================

This PEP can be seen as a more targeted version of the "faster standard library release cycle" proposals discussed in PEP 407 and PEP 413, focusing specifically on those areas which have implications beyond the Python community.

The creation of this PEP was prompted primarily by the aging SSL support in the Python 2 series. As of March 2014, the Python 2.7 SSL module is approaching four years of age, and the SSL support in the still popular Python 2.6 release had its feature set locked six years ago.

These are simply too old to provide a foundation that can be recommended in good conscience for secure networking software that operates over the public internet, especially in an era where it is becoming quite clearly evident that advanced persistent security threats are even more widespread and more indiscriminate in their targeting than had previously been understood.

While the use of the system OpenSSL installation addresses many of these concerns on Linux platforms, it doesn't address all of them, and in the case of the binary installers for Windows and Mac OS X that are published on python.org, the version of OpenSSL used is entirely within the control of the Python core development team.

With increased popularity comes increased responsibility, and this policy is about recognising the fact that Python's popularity and adoption has now reached a level where some of our design and policy decisions have significant implications beyond the Python development community.

As one example, the Python 2 ``ssl`` module does not support the Server Name Identification standard. While it is possible to obtain SNI support by using the third party ``requests`` client library, actually doing so currently requires using not only ``requests`` and its embedded dependencies, but also half a dozen or more additional libraries. The lack of support in the Python 2 series thus serves as an impediment to making effective use of SNI on servers, as Python 2 clients will frequently fail to handle it correctly.

Another more critical example is the lack of SSL hostname matching in the Python 2 standard library - it is currently necessary to rely on a third party library, such as ``requests`` or ``backports.ssl_match_hostname`` to obtain that functionality in Python 2.

The Python 2 series also remains more vulnerable to remote timing attacks on security sensitive comparisons than the Python 3 series, as it lacks a standard library equivalent to the timing attack resistant ``hmac.compare_digest()`` function. While appropriate secure comparison functions can be implemented in third party extensions, may users don't even consider the problem and use ordinary equality comparisons instead - while a standard library solution doesn't automatically fix that problem, it *does* make the barrier to resolution much lower once the problem is pointed out.

My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends. However, in the absence of this network security enhancement policy, that position is difficult to justify when it comes to software that operates over the public internet. Just as many developers consider it too difficult to develop truly secure modern networked software in C/C++ (largely due to the challenges associated with manual memory management), I consider it too difficult to develop truly secure modern networked software using the Python 2 series without introducing this network security enhancement policy, as doing so would mean reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts.

Requiring that latent defects in an application's Unicode correctness be addressed in order to migrate to Python 3 is not a reasonable alternative recommendation, especially given the likely existence of legacy code that lacks the kind of automated regression test suite needed to help support a migration from Python 2 to Python 3. The key point of this PEP is that those situations affect more people than just the developers and users of the affected application: their existence becomes something that developers of secure networked services need to take into account as part of their security design. By making it more feasible to enhance the security of the Python 2 series, we can help contribute to the evolution of a more secure internet for all concerned.

For the Python 2 series, this proposal will most obviously impact the remaining maintenance releases of Python 2.7. However, commercial redistributors that continue to offer full support for Python 2.6, and will also do so for Python 2.7 after upstream maintenance ends, may choose to take this policy into account when deciding what changes to backport to their own maintenance updates. Providing commercial redistributors such as ActiveState, Attachmate, Canonical, Continuum Analytics, Enthought, and Red Hat that option is one of the benefits offered by requiring feature detection based forward compatibility for the modules covered by this policy.

Open Questions ==============

* What are the risks associated with allowing OpenSSL to be updated to new feature versions in the Windows and Mac OS X binary installers for maintenance releases?

* Are there any other security relevant modules that should be covered by either a blanket or conditional exemption?

* Should we enumerate a specific list of "other networking modules" rather than leaving it implicit?

Acknowledgement ===============

Thanks to Christian Heimes for his recent efforts on greatly improving Python's SSL support in the Python 3 series, and a variety of members of the Python community for helping me to better understand the implications of the default settings we provide in our SSL modules, and the impact that tolerating the use of SSL infrastructure that was defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has for the security of the web as a whole.

Christian and Donald Stufft also provided valuable feedback on a preliminary draft of this proposal.

Copyright =========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Am 22.03.14 23:39, schrieb Ned Deily:

Regarding the python.org binary installers, I think past practice has been for us to update third-party libraries as necessary in maintenance releases when there is good cause and with the concurrence of the release manager, so I don't see this as a big issue. For the OS X binary installer, the issue for OpenSSL has been that we dynamically link to the system-supplied OpenSSL libraries and that, for various reasons, Apple has deprecated (and frozen at non-current OpenSSL releases) the use of those libraries in favor of their own security frameworks. So, for multiple reasons, including the risk that OpenSSL may be dropped from an upcoming major release of OS X, we need to start supplying our own version with all OS X binary installers. That's the plan regardless of the outcome of this PEP.

The Windows case is similar but different. I also update OpenSSL releases between Python bug fix releases, but only to OpenSSL bug fix releases (likewise for all other third party software). So 2.7 uses 0.9.8 (currently 0.9.8y), 3.3 and 3.4 use 1.0.1 (currently 1.0.1e). I still wish it was possible to drop OpenSSL altogether on both OSX and Windows, and use the vendor TLS libraries instead. This would reduce the maintenance burden, and might even improve performance. However, it's getting difficult since the ssl module exposes more and more OpenSSL API as-is (rather than trying to provide an abstraction). For Windows, this approach was unfortunately hopeless as long as we had to support XP. Now that Vista becomes the baseline, I hope I can try to provide much of the ssl module natively using CryptoAPI. Regards, Martin

On 23 March 2014 09:07, Donald Stufft wrote:

As someone who is deeply biased towards improving the packaging tool chain and getting people to use it I think that most people will simply use the Stdlib even if a more secure alternative exists. Infact one does exist and I still see almost everyone using the stdlib ssl instead of pyopenssl. At best they have an optional dependency on it which many people who aren't security conscious won't even realize why they should install it.

I should probably mention explicitly in the PEP that security related packages almost always involve somewhat tricky binary dependencies, and while we're working on it, our packaging ecosystem will still sometimes deliver a poor user experience on that front. There's also an inherent divide between "using an old version of Python" and "willing to use new dependencies from pip". They're not disjoint sets (plenty of folks use the CentOS system Python + pip), but I expect there's a large enough population of conservative corporate users on older versions that won't use a pip based solution even if its available, thus not really solving the problem. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 23 March 2014 07:32, Benjamin Peterson wrote:

On Sat, Mar 22, 2014, at 14:11, Nick Coghlan wrote:

...

Folks,

I have just posted a proposal to change the way we treat enhancements that relate to Python's support for network security enhancements.

I think the PEP should also address "security-mode" releases. Do the same exceptions apply?

Does anyone really want to backport features to Python 3.1?

As I see it, RHEL is the primary culprit for people still caring about Python 2.6. With the RHEL7 beta published late last year and hence CentOS 7 and RHEL 7 itself presumably landing well before Python 2.7 enters security fix only mode, I think it's OK to leave this aspect in the hands of the distro vendors and other commercial redistributors. We create the problem, we can bear the cost of dealing with it on behalf of our users. In Red Hat's specific case, we're also finally starting to offer users better solutions that make it easier to switch to a newer language runtime without having to upgrade the entire underlying OS or run in an unsupported configuration. In the longer term, this will hopefully mean that the expectation for upstream projects to maintain compatibility with the system Python in RHEL will become limited to just those projects that we actually include as part of the distro. However, I agree the PEP needs to have a dedicated section discussing security fix only releases. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 23 Mar 2014 09:08:29 +1000 Nick Coghlan wrote:

On 23 March 2014 08:53, Ben Darnell wrote:

...

I agree wholeheartedly with the sentiment behind this PEP, but I have concerns about the implementation. If we introduce new APIs into the ssl module then we will see packages and applications that depend on Python 2.7.7+, just like with the introduction of bool in 2.2.1. This will be a mess unless adoption of new Python minor versions is swift and consistent. The PEP says that new features should be testable with feature detection, but how far does that go? The Python 3 NPN interface is a method of SSLContext, which didn't exist in 2.7. Do we backport the entire SSLContext interface in a minor release, or introduce a new NPN interface that is not based on SSLContext just for older Pythons?

The whole thing.

Have you first tried to do it and appreciate the amount of disruption it would cause?

It's not just ssl2: you needs os2, random2, ftplib2, poplib2, imaplib2, smtplib2. You also run headlong into the problem of a complex external dependency on OpenSSL (which the Linux distro repackagers currently handle for us).

So are you proposing to backport changes to all these modules too? This sounds crazy. Regards Antoine.

On Sat, Mar 22, 2014 at 7:08 PM, Nick Coghlan wrote:

The issue isn't really the ssl module itself - it's the other modules that *depend* on the ssl module (like httplib, urllib, poplib, ftplib, imaplib). You could technically try to monkeypatch or shadow the stdlib ssl module from a third party library, but that's a *really* dangerous approach to security (since it makes it hard to do proper integration testing).

I suppose the difference is that the applications I've worked with have *already* abandoned stdlib modules like httplib or urllib in favor of libraries like requests and tornado, so I see this as mainly a question of whether the stdlib provides a suitable foundation for those packages. The argument about the difficulty of integration testing cuts both ways - I can use tox to test Tornado with several versions of a third-party package, but cannot (as long as https://launchpad.net/~fkrull/+archive/deadsnakesonly provides one package per x.y release) easily test with both Python 2.7.6 and 2.7.7.

...

In my experience most deployments of Python apps have better systems in place to deal with packages installed from PyPI than new versions of Python itself. Therefore it might be better to follow the lead of the unittest2 module and introduce a new ssl2 module that tracks the most recent version of Python's ssl module.

My counter-proposal is: * Introduce a new module 'ssl2' which is essentially a backport of the current release's ssl module for all old versions of Python that anyone cares to support.

It's not just ssl2: you needs os2, random2, ftplib2, poplib2, imaplib2, smtplib2. You also run headlong into the problem of a complex external dependency on OpenSSL (which the Linux distro repackagers currently handle for us).

Just os2 and random2; as I said a few bullet points down the stdlib libraries that use ssl would be sufficient justification for additions to the ssl module's interface.

This is what the """reimplementing substantial portions of the standard library as third party modules to gain access to the newer underlying network security protocols and mechanisms, and then injecting those into the module namespace to override their standard library counterparts.""" comment is about, but I guess I should expand on that for the benefit of readers not as familiar with the relevant implementation details.

In my proposal there is no monkey-patching; the newer ssl interfaces would come solely from the ssl2 module.

...

* Third-party applications and libraries targeting anything but the latest version of Python are advised to use ssl2 instead of ssl (this might also be a good opportunity to bless certifi or an equivalent). * Minor releases of Python are allowed to make any security-related change that does not change any public interfaces (e.g. linking against a new version of openssl, or disabling SSLv2)

New versions of OpenSSL are nowhere near as useful when their constants aren't made available in the Python API.

...

* New interfaces in security-related modules are allowed only if those new interfaces are used by other stdlib modules (since stdlib modules are the only place where ssl2 is off-limits). For example, a backport of match_hostname could go in if httplib or urllib used it. NPN/ALPN would not be allowed under this rule since there is no stdlib code that uses them; third-party SPDY or HTTP/2 code would have to use ssl2 to get this interface.

In this world, libraries like requests or tornado that want to use advanced ssl features can simply declare a dependency on ssl2, instead of saying "we require Python 2.7.7+, 3.2.6+, etc".

There's a reason the developers of those libraries haven't already done that: it's genuinely hard. It's not that I didn't think of this approach, it's that I think it would actually be more difficult and less effective than the approach currently in the PEP.

One of the reasons it's genuinely hard is that no one has provided an interface compatible with the stdlib ssl module. An ssl2 module that backported Python 3.4's ssl module would be a *much* more attractive porting target than e.g. PyOpenSSL. -Ben

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 22 March 2014 23:49, Donald Stufft wrote:

In the case of requests they already have an optional dependency on pyopenssl. It's just many people either don't know they should use it, are unable to use it, or unwilling to use the python packaging tool chain because of its current flaws.

Do they use the new features in the Python 3.x ssl module when it's available to give the same level of security as having pyopenssl would, or do they use a "lowest common denominator" (i.e., 2.x compatible) level of security when using the stdlib? If the latter, that would be very, very sad. Paul

They detect for ssl to have the SSLContext and use it if it's available.

On Mar 22, 2014, at 7:54 PM, Paul Moore wrote:

...

On 22 March 2014 23:49, Donald Stufft wrote: In the case of requests they already have an optional dependency on pyopenssl. It's just many people either don't know they should use it, are unable to use it, or unwilling to use the python packaging tool chain because of its current flaws.

Do they use the new features in the Python 3.x ssl module when it's available to give the same level of security as having pyopenssl would, or do they use a "lowest common denominator" (i.e., 2.x compatible) level of security when using the stdlib? If the latter, that would be very, very sad.

Paul

On Sun, 23 Mar 2014 07:11:07 +1000 Nick Coghlan wrote:

It is similar to the previous IDLE policy exception PEP, where we decided that cross version consistency of IDLE superseded the general policy against backporting enhancements to maintenance branches.

But the consequences are different: the stdlib promises API stability between bugfix releases, while IDLE is a standalone GUI application.

This topic involves a complex balance between encouraging and supporting good security practices and limiting the risk of failures for users upgrading to new maintenance releases, so I'd ask that folks take time to read and consider the implications of the full PEP in the broader context of today's internet before posting comments on specific details, or indicating a preference one way or the other in terms of the overall proposal.

The fact that it involves a "complex balance" implies IMO that a blanket exemption is the wrong mechanism, and instead the exemption should probably be granted on a case-by-case basis.

This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release.

But enforcing "secure by default" can by construction break backwards compatibility, which is the very reason we are so conservative with such changes.

Thus the focus on network security protocols and related cryptographic infrastructure -

This is a bit limited. There are remotely-exploitable security issues which are still open on the bug tracker; they are not cryptography-related, but that shouldn't make a difference. (for example the dreaded XML issues have never been properly fixed, AFAICT)

My position on the ongoing transition from Python 2 to Python 3 has long been that Python 2 remains a supported platform for the core development team, and that commercial support will remain available well after upstream maintenance ends.

If this is about the difficulty of migrating to Python 3, then this PEP should focus specifically on that problem, and restrict the policy to Python 2.7. Upgrading from 3.X to 3.X+1 is easy, and so is backporting patches from 3.X to 3.X-1, conversely.

* Are there any other security relevant modules that should be covered by either a blanket or conditional exemption?

Every module can be remotely "security relevant", unfortunately. Regards Antoine.

On Sat, 22 Mar 2014 23:54:37 +0100 Thomas Wouters wrote:

Or not rely on the standard library for their security. Much as I realize it is necessary for rudimentary SSL support (for example) to exist in the standard library,

Unfortunately, "rudimentary SSL support" is worse than nothing. Regards Antoine.

On 23 Mar 2014 10:18, "Christian Heimes" wrote:

On 23.03.2014 01:01, Antoine Pitrou wrote:

...

This is a bit limited. There are remotely-exploitable security issues which are still open on the bug tracker; they are not cryptography-related, but that shouldn't make a difference.

(for example the dreaded XML issues have never been properly fixed, AFAICT)

True, you may blame me for the situation. Only a handful of people were interested in the XML issues. I ran out of steam and moved to more sapid topics, too.

The key difference there is we're not reliant on defusedxml to download defusedxml :) Anyway, folks have given me plenty of good feedback to chew over. I'm going to go run some errands and then try to get a new revision posted this afternoon. Cheers, Nick.

Christian

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Sun, 23 Mar 2014 01:01:38 +0100, Antoine Pitrou wrote:

On Sun, 23 Mar 2014 07:11:07 +1000 Nick Coghlan wrote:

...

This PEP does *not* grant any general exemptions to the usual backwards compatibility policy for maintenance releases. Instead, it is designed to make it easier to provide more "secure by default" behaviour in future feature releases, while still limiting the risk of breaking currently working software when upgrading to a new maintenance release.

But enforcing "secure by default" can by construction break backwards compatibility, which is the very reason we are so conservative with such changes.

Yeah, I couldn't figure out what this paragraph meant. What does backporting features in maintenance releases have to do with making feature releases secure by default? If we are not relaxing the backward compatibility rules we can't do secure by default changes in a maintenance release...and if we don't do that, thinking that very much legacy software is going to get upgraded to use the new "more secure" capabilities is, I suspect (but cannot prove, obviously) mostly wishful thinking. An exception might be be commercial packages, but even there I wouldn't bet very many bucks. It seems to me that shops who are being security concious enough to be candidates for modifying currently deployed packages to be more secure are also likely to have already addressed the security issues in some other way. Now, requests and other libraries could use these new features, and since they have faster release cycles, they could presumably make the increased security be the default sooner. So that's what I see as the argument in favor...but it is a much narrower impact area than "most python software that uses ssl", and certainly doesn't cover poplib, smtplib, etc. But perhaps I am being overly cynical. Regardless, like Martin, I prefer to focus on Python3. --David

On 23 March 2014 10:40, "Martin v. Löwis" wrote:

Am 23.03.14 01:15, schrieb Christian Heimes:

...

On 23.03.2014 01:01, Antoine Pitrou wrote:

...

This is a bit limited. There are remotely-exploitable security issues which are still open on the bug tracker; they are not cryptography-related, but that shouldn't make a difference.

(for example the dreaded XML issues have never been properly fixed, AFAICT)

True, you may blame me for the situation. Only a handful of people were interested in the XML issues. I ran out of steam and moved to more sapid topics, too

I don't think anybody wanted to assign blame (although I can sympathize with your urge to accept the blame). The fact is that this is a volunteer project: we do what we can and have fun doing.

Agreed completely - what I'm trying to do here is set up a plan that is at least *acceptable* to the upstream community, so we can then seek corporate support for actually putting it into practice (and I'd advise against us accepting *any* proposal to resolve the situation without receiving binding commitments to provide ongoing maintenance support - while I think this proposal is important, I'm under no illusions that actually implementing it will be fun, and it's not appropriate to ask people to do that in their own time). However, we have a *lot* of downstream users and redistributors that have been taking CPython core development for granted, and by so doing, they have allowed a situation to develop that has some rather negative implications for the overall security of networked communications in the Python ecosystem. Since some of those same corporate redistributors are a key enabler allowing users to stay on those old releases that are no longer supported upstream, and others are likely to be being conservative in their own Python 3 migrations, I believe they share a lot of the responsibility for helping to resolve it, either by facilitating the migration to Python 3, helping to improve the networking security situation in Python 2, or, preferably, both. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 23 Mar 2014 01:40:32 +0100 "Martin v. Löwis" wrote:

Am 23.03.14 01:15, schrieb Christian Heimes:

...

On 23.03.2014 01:01, Antoine Pitrou wrote:

...

This is a bit limited. There are remotely-exploitable security issues which are still open on the bug tracker; they are not cryptography-related, but that shouldn't make a difference.

(for example the dreaded XML issues have never been properly fixed, AFAICT)

True, you may blame me for the situation. Only a handful of people were interested in the XML issues. I ran out of steam and moved to more sapid topics, too

I don't think anybody wanted to assign blame (although I can sympathize with your urge to accept the blame). The fact is that this is a volunteer project: we do what we can and have fun doing.

Indeed this wasn't meant to blame anyone, simply to point out that improving security is often hampered by lack of manpower or motivation. Regards Antoine.

On Sat Mar 22 2014 at 8:55:36 PM, Nick Coghlan wrote:

...

On Sat, 22 Mar 2014 23:54:37 +0100 Thomas Wouters wrote:

...

Or not rely on the standard library for their security. Much as I

realize

...

it is necessary for rudimentary SSL support (for example) to exist in

On 23 March 2014 10:08, Antoine Pitrou wrote: the

...

...

standard library,

Unfortunately, "rudimentary SSL support" is worse than nothing.

I'm going to try to find a way to steal that line and get it into the PEP. I'm not sure yet if my proposal is the *right* answer, but this observation gets right to the heart of the problem that actually needs to be solved.

I'd actually welcome a competing PEP that seriously explored separating out the relevant parts of the Python 3.4 standard library as a PyPI project and attempted to tackle the problem by making that available at least as far back as 2.7 and then just deprecating the leaking batteries in the 2.7 standard library.

Now I have been reading this thread on my phone and I only have cursory understanding of what failure ssl has had as of late, so this might be stupid, but what if in Python 3.5 we made it so people passed in an explicit SSL object into the relevant APIs and threw an exception if one wasn't provided when trying to access something requiring SSL? Then the ssl module would start existing on PyPI to make sure it can evolve faster than every 18 months? I know this doesn't help older APIs but it would make it rather explicit going forward that if you want security you need the latest thing and that's found on PyPI so it would at least solve the issue for the future which is better than nothing. We could try to provide matching APIs that monkey-patch older versions of Python if that's so desired to make this backwards-compatible, but as we all know that can be rather brittle. -Brett

Just because I think that approach would be even more work and even more disruptive than what I propose in PEP 466, doesn't mean it isn't a better idea overall.

What we have essentially found is that where we could basically get away with an 18 month update cycle for improved network security support (extended out to a few years by certain major platform vendors), that approach *isn't* working when it comes to putting a feature release into long term maintenance mode. I don't think the situation isn't critical yet, but it's getting close, and I think we need to deal with it within the 12 months (and preferably sooner than that).

My PEP proposes one way of addressing the challenge: allowing a wider variety of changes in maintenance releases to reduce the variance in capability in this area.

Moving the affected modules out of the standard library proper and bundling the critical ones along with pip instead is indeed another alternative. However, that approach introduces additional issues of its own - I'll cover some of them in the next PEP update, but it would be good to have someone explicitly trying to make the case that a PyPI backport would be simpler for the overall ecosystem than my suggested approach.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ brett%40python.org

On Mar 22, 2014, at 9:17 PM, Ben Darnell wrote:

On Sat, Mar 22, 2014 at 8:55 PM, Nick Coghlan wrote: What we have essentially found is that where we could basically get away with an 18 month update cycle for improved network security support (extended out to a few years by certain major platform vendors), that approach *isn't* working when it comes to putting a feature release into long term maintenance mode. I don't think the situation isn't critical yet, but it's getting close, and I think we need to deal with it within the 12 months (and preferably sooner than that).

This PEP as written applies to both Python 2.x and 3.x, but the two situations are very different. 3.x is on a ~18 month update cycle, so why isn't the status quo acceptable there? Python 2.x has less than 18 months of support left, so could it get by with a single exceptional release instead of a general relaxing of the rules? (if it were up to me, I'd call that release Python 2.8 instead of 2.7.7) If this PEP is mainly about a one-shot update to the security components of Python 2.x, I'd like to see an explicit list of what is in scope for the update.

-Ben _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

As one of the instigators who convinced Nick to write this PEP I’m far less concerned about back porting things to earlier 3.x releases than I am about getting things into 2.7 at this point. Going from 3.2 to 3.4 is not terribly difficult if it requires any work at all. Going from 2.7 to 3.4 is often times a significant investment in resources that has to be taken by *every* network using project. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 23 Mar 2014 11:47, "Donald Stufft" wrote:

On Mar 22, 2014, at 9:33 PM, Brett Cannon wrote:

...

On Sat Mar 22 2014 at 8:55:36 PM, Nick Coghlan

wrote:

...

...

On 23 March 2014 10:08, Antoine Pitrou wrote:

...

On Sat, 22 Mar 2014 23:54:37 +0100 Thomas Wouters wrote:

...

Or not rely on the standard library for their security. Much as I

realize

...

...

...

it is necessary for rudimentary SSL support (for example) to exist in the standard library,

Unfortunately, "rudimentary SSL support" is worse than nothing.

I'm going to try to find a way to steal that line and get it into the PEP. I'm not sure yet if my proposal is the *right* answer, but this observation gets right to the heart of the problem that actually needs to be solved.

I'd actually welcome a competing PEP that seriously explored separating out the relevant parts of the Python 3.4 standard library as a PyPI project and attempted to tackle the problem by making that available at least as far back as 2.7 and then just deprecating the leaking batteries in the 2.7 standard library.

Now I have been reading this thread on my phone and I only have cursory understanding of what failure ssl has had as of late, so this might be stupid, but what if in Python 3.5 we made it so people passed in an explicit SSL object into the relevant APIs and threw an exception if one wasn't provided when trying to access something requiring SSL? Then the ssl module would start existing on PyPI to make sure it can evolve faster than every 18 months?

It's not really a failure so much as what was concerned secure in 2010 isn't the case in 2014.

I don't want to sounds like I'm trying to say that pip should dictate Python policy or anything but pip more or less depends on the fact that ssl is available in the stdlib at this point in time in order to be able to handle secure downloads.

Every 18 months isn't that big of a deal tbh, the larger concern is Python 2.7 which has had nearly 4 years since it's release and is likely to exist for far more years past that.

Yes, I think it's worth narrowing the scope to just Python 2.7, and leaving the process unchanged for 3.x. Cheers, Nick.

...

I know this doesn't help older APIs but it would make it rather explicit

going forward that if you want security you need the latest thing and that's found on PyPI so it would at least solve the issue for the future which is better than nothing. We could try to provide matching APIs that monkey-patch older versions of Python if that's so desired to make this backwards-compatible, but as we all know that can be rather brittle.

...

-Brett

...

Just because I think that approach would be even more work and even more disruptive than what I propose in PEP 466, doesn't mean it isn't a better idea overall.

What we have essentially found is that where we could basically get away with an 18 month update cycle for improved network security support (extended out to a few years by certain major platform vendors), that approach *isn't* working when it comes to putting a feature release into long term maintenance mode. I don't think the situation isn't critical yet, but it's getting close, and I think we need to deal with it within the 12 months (and preferably sooner than that).

My PEP proposes one way of addressing the challenge: allowing a wider variety of changes in maintenance releases to reduce the variance in capability in this area.

Moving the affected modules out of the standard library proper and bundling the critical ones along with pip instead is indeed another alternative. However, that approach introduces additional issues of its own - I'll cover some of them in the next PEP update, but it would be good to have someone explicitly trying to make the case that a PyPI backport would be simpler for the overall ecosystem than my suggested approach.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/brett%40python.org

...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy@udel.edu(mailto:tjreedy@udel.edu)) wrote:

Instead, I think the PEP should propose a special series of server enhancement releases that are based on the final 2.7 maintenance release (2.7.8 or 2.7.9) but which have have a different application-specific enhancement policy.

This is an interesting idea. My biggest problem with it is that, at least with the ssl library, these aren’t server-only problems. If we suggest that they are, we end up in the same position we’re in right now (that is, hurting the internet). For example, Python 2.7’s ssl module lacks the OP_NO_COMPRESSION option for OpenSSL, meaning that the application is at the mercy of the server to determine whether it’s vulnerable to the CRIME attack. Given that all modern browsers already disable TLS compression, we can assume that lots of server admins haven’t bothered disabling it on their end. This leaves pretty much anyone using HTTPS, client or server, on Python 2.7 at risk of the CRIME attack. This isn’t a server-only problem, so I feel like limiting the fixes to a ‘server’ release is not good enough.

Hi, 2014-03-22 22:11 GMT+01:00 Nick Coghlan :

In particular, the exception will apply to:

* the ``ssl`` module * the ``hashlib`` module * the ``hmac`` module * the ``sha`` module (Python 2 only) * the components of other networking modules that make use of these modules * the components of the ``random`` and ``os`` modules that are relevant to cryptographic applications * the version of OpenSSL bundled with the binary installers

The current Python development cycle is too slow to track latest security recommandations. Python 2.7 is now very far from Python 3.4, so backportng so many features is very complex and error-prone. IMO A third-party module backporting all these security enhancements would be easy to develop. There are already M2Crypto, cryptography and PyOpenSSL modules for example. You may reuse them. The drawback is that applications would be benefit immediatly from this work, they should be modified to use the new module. But usually, developers who care of security are able to do these modifications. The PEP is very generic. Can you try to list all missing security features of old Python versions? Victor

Hi, 2014-03-23 11:17 GMT+01:00 :

Quoting Victor Stinner :

...

The drawback is that applications would be benefit immediatly from this work, they should be modified to use the new module. But usually, developers who care of security are able to do these modifications.

I think asking developers to make significant modifications to their code is besides the point of the PEP.

I expect something like replacing "import ssl" with "from safestdlib import ssl", same for other modules. FYI I backported some Python 3.3 modules for Python 2.7 for my Trollius project, and it was not so hard. For ssl.SSLContext, it's an empty class, it's just a wrapper to ssl.wrap_socket() which raises errors when unsupported features are used. http://trollius.readthedocs.org/#backports I wrote Trollius to port OpenStack to Python 3, not to defer the port :-)

However, if they are willing to make changes, I'd still recommend that they port their code to Python 3, as that is the better long-term investment.

It's not always possible. For example, the OpenStack projet still have +30 dependencies which are not Python 3 compatible yet. https://wiki.openstack.org/wiki/Python3#Dependencies We are porting these dependencies, but it may still take one year to port the whole OpenStack project to Python 3. Come to help us :) Sorry, it's maybe not fair to take the worst example (OpenStack) :-) Victor

On Sat, Mar 22, 2014 at 11:31 PM, Terry Reedy wrote:

The download page for the final 2.7.z maintenance release could say something like "We recommend that you move to the most recent Python 3 version if at all possible. If you cannot do that and you want to use Python to run a server on the public internet, we urge you to instead use the latest version of ServerPython 2.7.1s. This series is based on Python 2.7.z but has been and will continue to be enhanced with security features backported from Python 3."

I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be. If you create a 2.8 release which is restricted to just the topic areas of the PEP (that is, no other stuff backported from 3.x, no requirement to add other non-security bug fixes, etc), the incremented minor version number tells people that a bit of extra care is required to upgrade. The lack of change in the code base outside the security apparatus should make update pretty trivial for most every non-networked application. If the PEP or something like it is approved, the work is still going to have to be done, no matter what you call it. Why not be transparent about it? Skip

On Sun, 23 Mar 2014 07:29:07 +0000 Cory Benfield wrote:

On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy@udel.edu(mailto:tjreedy@udel.edu)) wrote:

...

Instead, I think the PEP should propose a special series of server enhancement releases that are based on the final 2.7 maintenance release (2.7.8 or 2.7.9) but which have have a different application-specific enhancement policy.

This is an interesting idea. My biggest problem with it is that, at least with the ssl library, these aren’t server-only problems. If we suggest that they are, we end up in the same position we’re in right now (that is, hurting the internet).

For example, Python 2.7’s ssl module lacks the OP_NO_COMPRESSION option for OpenSSL,

This is easy to change in a bugfix release, though. Someone just has to open an issue and write a patch. Regards Antoine.

On 3/23/2014 11:37 AM, Donald Stufft wrote:

On Mar 23, 2014, at 11:34 AM, Antoine Pitrou wrote:

...

On Sun, 23 Mar 2014 07:29:07 +0000 Cory Benfield wrote:

...

This is an interesting idea. My biggest problem with it is that, at least with the ssl library, these aren’t server-only problems. If we suggest that they are, we end up in the same position we’re in right now (that is, hurting the internet).

For example, Python 2.7’s ssl module lacks the OP_NO_COMPRESSION option for OpenSSL,

This is easy to change in a bugfix release, though. Someone just has to open an issue and write a patch.

I already did open an issue and write a patch :)

There’s someone on that issue saying that flipping that without a way to flip it back would break their application.

http://bugs.python.org/issue20994, if anyone is looking for it. Eric.

On Mar 23, 2014, at 11:46 AM, Antoine Pitrou wrote:

On Sun, 23 Mar 2014 11:37:25 -0400 Donald Stufft wrote:

...

I already did open an issue and write a patch :)

There’s someone on that issue saying that flipping that without a way to flip it back would break their application.

You're right, I had forgotten about that :-)

I'd be surprised if disabling compression could altogether break an application. It may make it less efficient, though, if it relied on TLS to compress data for it (which doesn't sound like a good thing to rely on, anyway).

Regards

Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I tend to agree about that, but I was mostly echoing the fact that they’ve expressed that sentiment. Without SSLContext I couldn’t think of a reasonable way to enable them to turn it on/off that didn’t involve a whole new API just for some versions of 2.7. This seemed to me like a wholly worse idea than back porting SSLContext :/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mar 23, 2014, at 11:55 AM, Mark Lawrence wrote:

On 23/03/2014 15:46, Antoine Pitrou wrote:

...

On Sun, 23 Mar 2014 11:37:25 -0400 Donald Stufft wrote:

...

I already did open an issue and write a patch :)

There’s someone on that issue saying that flipping that without a way to flip it back would break their application.

You're right, I had forgotten about that :-)

I'd be surprised if disabling compression could altogether break an application. It may make it less efficient, though, if it relied on TLS to compress data for it (which doesn't sound like a good thing to rely on, anyway).

Regards

Antoine.

I've looked at the issue and can't find any such comment. The closest is from http://bugs.python.org/issue20994#msg214246 "Also some users will absolutely want to manually re-enable compression, please don't disable it entirely." Chinese whispers? :)

-- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language.

Mark Lawrence

--- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Ah, maybe he told me that in IRC then, I actually know him and was arguing with him about it there. Sorry I got my streams crossed. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 3/23/2014 3:29 AM, Cory Benfield wrote:

On 23 March 2014 at 04:32:17, Terry Reedy (tjreedy@udel.edu(mailto:tjreedy@udel.edu)) wrote:

...

Instead, I think the PEP should propose a special series of server enhancement releases that are based on the final 2.7 maintenance release (2.7.8 or 2.7.9) but which have have a different application-specific enhancement policy.

This is an interesting idea. My biggest problem with it is that, at least with the ssl library, these aren’t server-only problems. If we suggest that they are, we end up in the same position we’re in right now (that is, hurting the internet).

If I understood the problem and pep too narrowly, substitute 'security' or whatever phrase you want for 'server'. My last sentence was an admission that some details of what I said would need to be changed. The key idea is to admit the what is being proposed is a minor fork of 2.7 (somewhat like Stackless 2.7) that might break code and which would require separate code testing -- Terry Jan Reedy

On 3/23/2014 9:00 AM, Skip Montanaro wrote:

On Sat, Mar 22, 2014 at 11:31 PM, Terry Reedy wrote:

...

The download page for the final 2.7.z maintenance release could say something like "We recommend that you move to the most recent Python 3 version if at all possible. If you cannot do that and you want to use Python to run a server on the public internet, we urge you to instead use the latest version of ServerPython 2.7.1s. This series is based on Python 2.7.z but has been and will continue to be enhanced with security features backported from Python 3."

I'm unclear how this would be better than just biting the bullet and making a 2.8 release.

The first step is to 'bite the bullet' and admit that the proposal is for a fork of 2.7. (Martin use that word today and pointed out the analogy with Stackless Python.) I decided not to suggest '2.8' because a) we said there would be no 2.8 (though that could be reversed); b) once we say '2.8', there would be a mountain of proposals to change this and that; c) people will expect the change from 2.7 to 2.8 to be something like previous deltas, but it will not be; d) the security fork might have a change that would normally require a deprecation period, but there might not be; and e) once a 2.8 were released, it too would be closed to enhancements, so that 2.9, 2.10, 2.11.. would eventually be needed. I mentioned this as one possible solution, but one I do not like. In summary, "2.8 is to 2.7 as 2.7 is to 2.6, etcetera" would not be true, so lets not pretend or mislead people that is would be. The fork series, not being a regular CPython series, would not be subject to the regular CPython rule of needing a new minor version number for each set of enhancements and spacing the sets 18 to 24 months apart.

On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases.

No CPython x.y.z has a two-digit number for z. I know that this is a subtle hint, which is why I also propose a new name. I do not think that anyone expects Stackless Python 2.7 to be an exact drop-in replacement for any 2.7.z, although it is for code that is not affected by its specialized alterations.

On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be.

Consider 'ServerPython' a placeholder for a better name. In my penultimate sentence "There are still details to be filled in or altered," I was specifically thinking of this.

If you create a 2.8 release which is restricted to just the topic areas of the PEP (that is, no other stuff backported from 3.x, no requirement to add other non-security bug fixes, etc), the incremented minor version number tells people that a bit of extra care is required to upgrade. The lack of change in the code base outside the security apparatus should make update pretty trivial for most every non-networked application. If the PEP or something like it is approved, the work is still going to have to be done, no matter what you call it. Why not be transparent about it?

We agree on being transparent, even if we have different ideas on exactly what needs to be made clear. -- Terry Jan Reedy

On Mar 23, 2014, at 7:03 PM, Barry Warsaw wrote:

...

On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote:

I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be.

It seems to me that the problem the PEP addresses can largely be confined to the Python 2.7 standard library and the fact that it's bundled with the language. I want to stick to our no-Python-2.8 pledge, but I wonder if there isn't a middle ground where we fork the stdlib into a separate branch, and apply security specific changes there.

Python 2.7.x will always be the "standard stdlib". We would never release a specific Python 2.7 + "security stdlib" release, but downstream developers would be able to overlay this forked stdlib on top of the standard one. Volunteers or corporate sponsors could distribute binary installers with this combination of pure Python 2.7 language + "security enhanced stdlib", and Linux distros could do the necessary building and distributing for their own platforms if they so desired.

The trick is what do you call this new combination, how do you invoke it, and how do you keep it distinct and independent of the system's standard Python 2.7?

Maybe this is some scent of Python 2.8 by another name, but let's never call it Python 2.8.

It seems like this and the fork idea are both jumping through hoops to avoid the inevitable - a 2.7 security release or a 2.8 security only release. Especially as I know nick isn't the only one looking to hire FTEs for this specific work as it's actively hurting service providers/distributions and others. If we call it a security only release, and scope/review all patches to that it seems that it's pretty clear what the release is for. 2.x is going to be in the wild for at least another decade as these large legacy codebases are functioning, profitable and dedicating teams of engineers to port them to 3.x doesn't make sense to the business. As new services and things come online we'll see the gradual movement to 3 accelerate especially as people realize clients and service talk better on an interface such as http rather than large monolithic tragedies.

-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

On 3/23/2014 8:03 PM, Barry Warsaw wrote:

On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote:

...

I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be.

It seems to me that the problem the PEP addresses can largely be confined to the Python 2.7 standard library and the fact that it's bundled with the language. I want to stick to our no-Python-2.8 pledge, but I wonder if there isn't a middle ground where we fork the stdlib into a separate branch, and apply security specific changes there.

Python 2.7.x will always be the "standard stdlib". We would never release a specific Python 2.7 + "security stdlib" release, but downstream developers would be able to overlay this forked stdlib on top of the standard one. Volunteers or corporate sponsors could distribute binary installers with this combination of pure Python 2.7 language + "security enhanced stdlib", and Linux distros could do the necessary building and distributing for their own platforms if they so desired.

The trick is what do you call this new combination,

I have already agree that 'ServerPython' is just a placeholder.

how do you invoke it, and how do you keep it distinct and independent of the system's standard Python 2.7?

Maybe this is some scent of Python 2.8 by another name, but let's never call it Python 2.8.

I agree, no only because of the 'pledge', but because it is the wrong model. Nick's latest paraphrase of where he thinks his proposal is heading is "the SSL, hashlib and hmac modules are kept in sync with Python 3 feature releases, but use the same default settings as the original Python 2.7 release". '2.8' would imply a broader focus than just 3 modules, and it would only work for the first of a series of enhancement releases. -- Terry Jan Reedy

On Mar 23, 2014, at 8:31 PM, Jesse Noller wrote:

...

On Mar 23, 2014, at 7:03 PM, Barry Warsaw wrote:

...

On Mar 23, 2014, at 08:00 AM, Skip Montanaro wrote:

I'm unclear how this would be better than just biting the bullet and making a 2.8 release. On the one hand, the 2.7.x number suggests (based on the existing release protocol) that it should be a drop-in replacement for earlier 2.7 micro releases. On the other hand, calling it something like "ServerPython" implies that it's not necessary for network client applications, when, if I read the PEP correctly, it most certainly would be.

It seems to me that the problem the PEP addresses can largely be confined to the Python 2.7 standard library and the fact that it's bundled with the language. I want to stick to our no-Python-2.8 pledge, but I wonder if there isn't a middle ground where we fork the stdlib into a separate branch, and apply security specific changes there.

Python 2.7.x will always be the "standard stdlib". We would never release a specific Python 2.7 + "security stdlib" release, but downstream developers would be able to overlay this forked stdlib on top of the standard one. Volunteers or corporate sponsors could distribute binary installers with this combination of pure Python 2.7 language + "security enhanced stdlib", and Linux distros could do the necessary building and distributing for their own platforms if they so desired.

The trick is what do you call this new combination, how do you invoke it, and how do you keep it distinct and independent of the system's standard Python 2.7?

Maybe this is some scent of Python 2.8 by another name, but let's never call it Python 2.8.

It seems like this and the fork idea are both jumping through hoops to avoid the inevitable - a 2.7 security release or a 2.8 security only release. Especially as I know nick isn't the only one looking to hire FTEs for this specific work as it's actively hurting service providers/distributions and others.

If we call it a security only release, and scope/review all patches to that it seems that it's pretty clear what the release is for.

2.x is going to be in the wild for at least another decade as these large legacy codebases are functioning, profitable and dedicating teams of engineers to port them to 3.x doesn't make sense to the business. As new services and things come online we'll see the gradual movement to 3 accelerate especially as people realize clients and service talk better on an interface such as http rather than large monolithic tragedies.

...

-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com

---

Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

I agree, the bulk of the alternative suggestions feel more like trying to adhere to a policy for policy’s sake rather than actually figure out what is best for the users. Adding new APIs to 2.7 feels to me like a pretty backwards compatible thing to be honest. The biggest contenders are things like SSLContext and match_hostname. The PEP still says things must be backwards compatible so we can't change existing APIs in a way that the backported things aren't opt in. I mean how likely is it that there is code out there relying on the fact that SSLContext doesn't exist other than as a detection to determine if they can use SSLContext? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mar 23, 2014, at 9:31 PM, Barry Warsaw wrote:

On Mar 24, 2014, at 11:38 AM, Chris Angelico wrote:

...

Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the standard lib. Then you can go back to the standard 2.7 (if you want to) by unsetting PYTHONPATH.

It'd be nice if SEPython defined a modified sys.version for clarity, but otherwise, it'd be a vanilla Python 2.7.

That's certainly more in the direction of what I think is an appropriate upstream solution.

The thing is, there isn't one single "what's best for users" resolution. There are many, many competing requirements and I think it will be difficult to satisfy everyone. I'm particularly sensitive to complaints of unexpected changes between micro releases.

If it’s too much for a micro release call it 2.8 or 2.9 or something and give it a super limited scope. Proposals like modifying the sys.path to opt into the backported things or having an official unofficial stdlib just adds more things downstream users have to actually test against and more possible configurations or Pythons. Right now users have a singular method for determining what the runtime environment looks like for Python, the version. There are processes around selecting different Python versions for things, upgrading etc. This isn’t a new thing for users. Suddenly having to mess with PYTHONPATH or having to figure out if their distro used the “official stdlib” or the “official unofficial stdlib” just adds massive confusion. Things like ssl3 etc are “ok” but less than optimal IMO because one of the benefits of adding SSLContext and such to Python 2.x is that you can write code that works in 2.x and 3.x without having to special case 2.7.7 or 2.8 or whatever. However if you just add ssl3 then now you have a third path through the code that you have to deal with.

-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 24 Mar 2014 15:25, "Chris Angelico" wrote:

As has already been pointed out, this can already happen, but in an ad-hoc way. Making it official or semi-official would mean that a script written for Debian's "Python 2.7.10" would run on Red Hat's "Python 2.7.10", which would surely be an advantage.

And having it break on the official Windows and Mac OS X binaries would benefit end users, how? The position I am coming to is that the "enhanced security" release should be the default one that we publish binary installers for, but we should also ensure that downstream redistributors can easily do "Python 2.7 with legacy SSL" releases if they so choose. I'm happier forcing end users to rely on a redistributor to opt in to a lower security option than I am to knowingly publish too many additional releases with network security infrastructure that is (at best) rapidly approaching its use by date. Cheers, Nick.

ChrisA _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 24 March 2014 08:44, Nick Coghlan wrote:

The position I am coming to is that the "enhanced security" release should be the default one that we publish binary installers for, but we should also ensure that downstream redistributors can easily do "Python 2.7 with legacy SSL" releases if they so choose. I'm happier forcing end users to rely on a redistributor to opt in to a lower security option than I am to knowingly publish too many additional releases with network security infrastructure that is (at best) rapidly approaching its use by date.

I'm with you here Nick, the default should really be "Python 2.7 with working TLS", not "Python 2.7 with broken TLS" (my only-slightly tongue-in-cheek proposed names for the releases). My concern with having Chris' proposal of an opt-in SEPython release is about discovery. I suspect most people will never find out about the SEPython release, meaning that most people will remain on insecure Python 2.7 distributions. If that happens, requests is essentially required to continue to support our current TLS model for 2.7 for as long as we support 2.7, and this PEP would give us no benefit at all. We're obviously not the only stakeholder here, but it's worth noting. Cory

On 24 March 2014 23:49, Skip Montanaro wrote:

So what's the big deal? Why can't we be pragmatic and call this thing (whatever it turns out to be) 2.8? Is this pledge and its rationale written down in a PEP somewhere, so I can study the reasons behind what appears at this point to be blind adherence?

There's a ton of work involved in creating a new feature release, and there's no way we're going to go through that much drudgery for the legacy Python 2 series without someone paying for it. Even the more limited network security infrastructure update proposal in this PEP is conditional on corporate contributors agreeing to do the heavy lifting.

Did someone administer a blood oath at a recent PyCon?

As discussed in the PEP, a Python 2.8 release wouldn't actually solve this problem, so there's no reason for commercial redistributors to contribute to making it happen. For example, RHEL7 and derivatives are already locked in to 2.7 until 2024, RHEL6 and derivatives are locked in to 2.6 until 2020. The only way to keep those combination of RHEL and the Python 2 standard library from holding back the evolution of internet security standards is to find a way solve the problem *within* the 2.7 line in such a way that I can then make the case for also backporting it to 2.6 in a RHEL6 point release. I think the proposal currently in round 3 of the PEP is something I can sell to the Platform Engineering team as a necessary update to make in a relatively timely fashion (I don't think the situation is critical yet, but give it another year or two and I'll be a lot more concerned). By contrast, trying to handle this via a Python 2.8 release would mean I would still have to advocate for us to adopt the policy in the PEP independently of upstream, and that's not a good outcome for anyone. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 23 Mar 2014 21:31:12 -0400, Barry Warsaw wrote:

On Mar 24, 2014, at 11:38 AM, Chris Angelico wrote:

...

Easy. Just set PYTHONPATH to import the SEPython [1] lib ahead of the standard lib. Then you can go back to the standard 2.7 (if you want to) by unsetting PYTHONPATH.

It'd be nice if SEPython defined a modified sys.version for clarity, but otherwise, it'd be a vanilla Python 2.7.

That's certainly more in the direction of what I think is an appropriate upstream solution.

The thing is, there isn't one single "what's best for users" resolution. There are many, many competing requirements and I think it will be difficult to satisfy everyone. I'm particularly sensitive to complaints of unexpected changes between micro releases.

In the context of that last sentence, I think it is worth noting the stance that 3.4 is taking[*] about security backward compatibility, since many people may not be aware of it (we only just finished making the documentation clear). If you use create_default_context() to get your context object, you get a "best practices" level of security *that may change between maintenance releases*. If you want things to not change between maintenance releases, you create your own context object and set its controls appropriately. In other words, the programmer opts in to maintenance release security improvements by using create_default_context. I presume that whatever comes out of this PEP will use the same approach. Note: thanks again to Christian Heimes and Antoine Pitrou for this work. Without Christian's work, I think we wouldn't even be having this conversation. Antoine's earlier work laid essential groundwork, but by itself I'm not sure that would have been enough to result in calls for a backport. It took both of them, with some help from others as well. --David [*] I actually don't know if this was discussed on python-dev previously because I've got a backlog of messages I'm not caught up on. If it hasn't been, then doubly good to mention it now, since the first 3.4 maintenance release hasn't happened yet :)

On 25 March 2014 00:36, Nick Coghlan wrote:

On 25 March 2014 00:18, Skip Montanaro wrote: The PEP does not permit backwards compatibility breaks in maintenance releases

Well, ssl.create_default_context() will use the same "this is a dynamic best practices API" policy as it does in 3.4. But everything that already exists in 2.7.6 will remain compatible with that behaviour by default. The policy change in the PEP won't automagically make networked software secure - it will just make it a heck of a lot *easier* to write in Python 2 (or the common subset of Python 2 and Python 3) by backporting the Python 3.4 enhancements to a 2.7 maintenance release, and, depending on the relative timing of the releases, likely the Python 3.5 enhancements to a later maintenance release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Mar 24, 2014 at 8:28 PM, Cory Benfield wrote:

On 24 March 2014 08:44, Nick Coghlan wrote:

...

The position I am coming to is that the "enhanced security" release should be the default one that we publish binary installers for, but we should also ensure that downstream redistributors can easily do "Python 2.7 with legacy SSL" releases if they so choose. I'm happier forcing end users to rely on a redistributor to opt in to a lower security option than I am to knowingly publish too many additional releases with network security infrastructure that is (at best) rapidly approaching its use by date.

I'm with you here Nick, the default should really be "Python 2.7 with working TLS", not "Python 2.7 with broken TLS" (my only-slightly tongue-in-cheek proposed names for the releases). My concern with having Chris' proposal of an opt-in SEPython release is about discovery. I suspect most people will never find out about the SEPython release, meaning that most people will remain on insecure Python 2.7 distributions. If that happens, requests is essentially required to continue to support our current TLS model for 2.7 for as long as we support 2.7, and this PEP would give us no benefit at all. We're obviously not the only stakeholder here, but it's worth noting.

The opting in could be done at the distro level. Red Hat could ship a thing called /usr/bin/python that runs SEPython, and that package could be identified and numbered in such a way that it's clearly a drop-in replacement for vanilla Python. If backward compatibility is done carefully (which, from everything I'm seeing here, is the way it's to be done), there should be absolutely no downside to using SEPython, except for portability (because now you're depending on it for security), and corner cases like testing. Alternatively, it can be done as a separate package: install "python" to get Python, and install "python-security-patches" or "python-tls" to get the tweaks - once that's installed, /usr/bin/python becomes SEPython. Programs can depend on that package to ensure security. That does have a partial discoverability problem (possibly even worse, as you could test on a computer that has x, y, z installed, and then deploy on a brand new computer, and nothing there ever depended on SEPython, so you're vulnerable without knowing why), but that could be ameliorated by simply advising people to type "yum install python-tls" if they want to run any Python programs that use TLS. But I'd be inclined to the first option; its biggest downside is that installing "python" installs something other than the "standard stdlib", which IMO isn't a big problem if (a) it's 100% backward compatible, and (b) it's semi-official anyway. ChrisA

On 2014-03-25 01:29, Ben Darnell wrote:

On Mon, Mar 24, 2014 at 4:44 AM, Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

On 24 Mar 2014 15:25, "Chris Angelico" mailto:rosuav@gmail.com> wrote:

> As has already been pointed out, this can already happen, but in an > ad-hoc way. Making it official or semi-official would mean that a > script written for Debian's "Python 2.7.10" would run on Red Hat's > "Python 2.7.10", which would surely be an advantage.

And having it break on the official Windows and Mac OS X binaries would benefit end users, how?

The position I am coming to is that the "enhanced security" release should be the default one that we publish binary installers for, but we should also ensure that downstream redistributors can easily do "Python 2.7 with legacy SSL" releases if they so choose. I'm happier forcing end users to rely on a redistributor to opt in to a lower security option than I am to knowingly publish too many additional releases with network security infrastructure that is (at best) rapidly approaching its use by date.

I am strongly against allowing downstream distributors that choice. Unless the security-enhanced variant of Python 2.7 quickly and completely overtakes all previous versions, we will be creating a compatibility problem between the two variants of Python 2.7. I believe that the changes motivating this PEP can be made with minimal backwards-incompatibility risk and (if the PEP is accepted) we should use all the leverage at our disposal to drive adoption. The risk is not backwards incompatibility, it is ambiguity of what Python 2.7 means. If changes under this PEP would result in any distributors rationally remaining at Python 2.7.6, then the result of any such changes should be labelled Python 2.8.

I think that calling it Python 2.8 would be a bad idea for the reasons that have already been stated. Perhaps it should just be called Python 2.7 Enhanced Security ("Python 2.7 ES").

On 24 March 2014 19:37, Chris Angelico wrote:

The opting in could be done at the distro level. Red Hat could ship a thing called /usr/bin/python that runs SEPython, and that package could be identified and numbered in such a way that it's clearly a drop-in replacement for vanilla Python. If backward compatibility is done carefully (which, from everything I'm seeing here, is the way it's to be done), there should be absolutely no downside to using SEPython, except for portability (because now you're depending on it for security), and corner cases like testing.

What's your solution for OS X, Windows et al? My concern is that if you have a release called 'Python' and a release called 'Python with security stuff', a surprisingly large number of people will download the first, especially if the notes for the security release say 'this may cause some minor compatibility problems'. IMHO, I'd rather have good security be the default for everyone, and require an explicit opt-out to get the bad security release.

On Tue, Mar 25, 2014 at 7:12 PM, Cory Benfield wrote:

On 24 March 2014 19:37, Chris Angelico wrote:

...

The opting in could be done at the distro level. Red Hat could ship a thing called /usr/bin/python that runs SEPython, and that package could be identified and numbered in such a way that it's clearly a drop-in replacement for vanilla Python. If backward compatibility is done carefully (which, from everything I'm seeing here, is the way it's to be done), there should be absolutely no downside to using SEPython, except for portability (because now you're depending on it for security), and corner cases like testing.

What's your solution for OS X, Windows et al? My concern is that if you have a release called 'Python' and a release called 'Python with security stuff', a surprisingly large number of people will download the first, especially if the notes for the security release say 'this may cause some minor compatibility problems'. IMHO, I'd rather have good security be the default for everyone, and require an explicit opt-out to get the bad security release.

Exactly the same. If someone wants to distribute SEPython (that someone might be python.org itself, or ActiveState, or anyone else who has an interest in it), they're welcome to do so; and it could be done either as an all-in-one that packages all of CPython, or as an add-on; either way would work just as well, but the former would be cleaner. ChrisA

On Tue, Mar 25, 2014 at 7:37 PM, Cory Benfield wrote:

On 25 March 2014 08:26, Chris Angelico wrote:

...

Exactly the same. If someone wants to distribute SEPython (that someone might be python.org itself, or ActiveState, or anyone else who has an interest in it), they're welcome to do so; and it could be done either as an all-in-one that packages all of CPython, or as an add-on; either way would work just as well, but the former would be cleaner.

Reading this I suspect we're mostly in agreement but having trouble communicating. My understanding of your point is simply that you don't want python-dev to 'bless' either of the 2.7 releases proposed as _the_ 2.7 release, instead pushing that choice on to people distributing Python. I can get behind that plan so long as the source code releases are named in such a way that people are either a) forced to make a choice; or b) defaulted to secure 2.7.

I'd like python.org / python-dev to bless, if not some specific version, at least some specific structure. I think that's something like what has been in the PEP at some point, though I haven't dug into the current version deeply enough to be sure. But if you take current 2.7 as a baseline, every new feature would be implemented by creating a new attribute of either the ssl module or some class in it; if the attribute is there, you can use it (eg a constant/enum value that's a parameter to something else), and if it's not, you can't. As long as the names are consistent, it'd be easy for a program to either probe and use what it can get, or just use what it wants and bomb if you don't give it a sufficiently-secure Python. So by that model, current 2.7 is fully compliant, and anything that doesn't actively conflict with that is also compliant. Any script that is written for the current 2.7 is guaranteed also to run on any compliant SEPython; and anything written for SEPython has to gracefully handle (which might mean cleanly bombing) anything down to and including current 2.7. Does that make sense? ChrisA

On Tue Mar 25 2014 at 4:21:51 AM, Georg Brandl wrote:

Am 25.03.2014 08:51, schrieb Nick Coghlan:

...

...

I think that calling it Python 2.8 would be a bad idea for the reasons that have already been stated.

Perhaps it should just be called Python 2.7 Enhanced Security ("Python 2.7 ES").

The PEP currently calls the proposed unmodified fork of 2.7 "Python 2.7 with Legacy SSL". I suspect we could potentially ask the PSF to enforce that from a trademark perspective (that is, redistributors wouldn't be allowed to call versions with the legacy infrastructure "Python 2.7", they'd have to include the "with Legacy SSL" qualifier - that would also encompass all redistributions of 2.7.6 and below).

I don't know. It still feels like a source of confusion all round to have two different (C)Pythons not distinguished by version number.

I haven't followed all of this thread, so forgive me if this suggestion has come up already:

Since we know the EOL of 2.7, can't we say there won't be any more "non-secure" bugfix releases than up to 2.7.9, and the namespace 2.7.10 (yeah I know, but still way better than 2.8) and above is free for the "new SSL" versions.

This also works from a version requirement point of view: if you require Python

...

= 2.7.10 you know you'll get the new features. If you don't, you shouldn't be using (or carefully checking) the new opt-in features.

Or if this is such a big deal we start with 2.7.6 and not postpone until 2.7.10 (which I guess could happen immediately after 2.7.9 and have nothing more than the upgraded modules). People have been making grandiose statements about how the security of the internet is hampered by Python 2.7 in this discussion. If these statements are actually not over-stated then we should do the fix sooner *and* add the incentive people to switch over by getting more bug fixes. It's not like Python 2.7 is getting a ton of fixes at this point anyway.

...

I'm actually personally OK with just making vendors do all the work if they're really so worried about a slightly increased chance of undetected regressions that they prefer to keep using older SSL infrastructure. I think persisting with the old SSL infrastructure for too much longer would be a fundamentally bad idea, so I don't mind at all making it more difficult for downstream redistributors to do so.

I agree, if no other solution can be found we should err on the secure side (as opposed to the safe side).

As long as we make it clear we have chosen to change our backwards-compatibility guarantees in the name of security and have a link to the last backwards-compatible release then I agree as well.

Am 25.03.2014 23:15, schrieb Nick Coghlan:

...

As long as we make it clear we have chosen to change our backwards-compatibility guarantees in the name of security and have a link to

On 26 Mar 2014 01:19, "Brett Cannon" mailto:bcannon@gmail.com> wrote: the last backwards-compatible release then I agree as well.

I am not sure how this meme got started, but let me be clear: the proposed policy DOES NOT provide blanket permission to break backwards compatibility in the affected modules. It only allows ADDING new features to bring these modules into line with their Python 3 counterparts, making it easier for third party packages like requests to do the right thing in a cross-version compatible way.

We know. That's what we mean by that. Georg

On 3/25/2014 6:15 PM, Nick Coghlan wrote:

I am not sure how this meme got started, but let me be clear: the proposed policy DOES NOT provide blanket permission to break backwards compatibility in the affected modules. It only allows ADDING new features to bring these modules into line with their Python 3 counterparts, making it easier for third party packages like requests to do the right thing in a cross-version compatible way.

I think the problem is that 'no new features' covers two quite different types of 'new' lumped together: change-new and new-new. New-new adds something completely new and does not break any sensible behavior that we worry about. We freely add new-new features with each new version. We do not add them to maintenance releases because then maintenance releases would be new versions. There are also considerations of incubation time, and the increased possibility of regressions with candidate-release versus alpha-beta-candidate-release schedules. Change-new additions changing existing behavior (like changing a default) and can therefore break code that we care about. Even for new versions, we are wary them and may require deprecation. One of the ironies of change-new is that it is much more similar to change-fix, which is allowed as maintenance, than new-new is. The tracker controversies are about issues on the borderline between change-fix and change-new. The original version of the PEP reference 434 as precedent. While partly valid, the reference could have fed the confusion because 434 does not draw the distinction given above. Indeed, part of the point of the PEP is that separating change-fix from change-new is difficult. As far as regressions go, a big new-new patch touching several untested modules worries me more than a small change-??? patch tweaking something in only one module, especially after writing tests for the module. -- Terry Jan Reedy

On Wed Mar 26 2014 at 2:15:39 AM, Georg Brandl wrote:

Am 26.03.2014 00:06, schrieb Nick Coghlan:

...

On 26 Mar 2014 08:32, "Georg Brandl" mailto:g.brandl@gmx.net> wrote:

...

Am 25.03.2014 23:15, schrieb Nick Coghlan:

...

On 26 Mar 2014 01:19, "Brett Cannon"

mailto:bcannon@gmail.com

...

...

mailto:bcannon@gmail.com>> wrote:

...

As long as we make it clear we have chosen to change our backwards-compatibility guarantees in the name of security and have a

...

...

...

the last backwards-compatible release then I agree as well.

I am not sure how this meme got started, but let me be clear: the

...

...

...

policy DOES NOT provide blanket permission to break backwards compatibility in the affected modules. It only allows ADDING new features to bring

...

...

...

into line with their Python 3 counterparts, making it easier for

link to proposed these modules third party

...

...

...

packages like requests to do the right thing in a cross-version compatible way.

We know. That's what we mean by that.

That's not what Brett said - he called 2.7.6 the "last backwards compatible release". That's not correct, as even under my proposal, 2.7.7+ will still be backwards compatible.

Yeah, I took "backwards-compatibility guarantees" to also mean the "no new features" guarantee, but you're right that the two can be separated.

That's also the way I took it; not that code would be inherently broken from 2.7.0 -> 2.7.x, but that the "let's stay conservative and add *nothing*" would stop being followed since something in 2.7.x wouldn't work in 2.7.0. IOW I expect (except maybe in extreme edge cases) that unmodified code would work across 2.7 as long as it works in 2.7.0 no matter what happens with this PEP.