Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
Hi folks, I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence. As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet. Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/ There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first. Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe. Regards, Nick. P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's. ================================ PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour. Rationale ========= PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure. PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase. These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). Requirements for capability detection ===================================== As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour. Recommendation for an environment variable based security downgrade =================================================================== Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself. In these cases, a configuration mechanism is needed that provides: * an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _https_verify_envvar = 'PYTHONHTTPSVERIFY' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to: * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Recommendation for backporting to earlier Python versions ========================================================= Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. Required marker attribute ------------------------- The required marker attribute on the ``ssl`` module when implementing this recommendation is:: _cert_verification_config = '<path to configuration file>' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- This approach is currently only defined for \*nix system Python installations. The recommended configuration file name is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Combining the recommendations ============================= If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Copyright ========= This document has been placed into the public domain. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So I dropped the ball on this too -- I was going to either have a look or tell you quickly to find a BDFL-delegate, but ended up doing neither. Skimming it now I really don't think I'm the right person to review this. Maybe you could ask Alex Gaynor? On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
Regards, Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015
Abstract ========
PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour.
Rationale =========
PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor).
Requirements for capability detection =====================================
As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade ===================================================================
Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions =========================================================
Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting.
Recommended file location -------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is ``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format -----------------------
The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present.
Example implementation ----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
* run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch.
Combining the recommendations =============================
If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled.
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context
# Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright =========
This document has been placed into the public domain.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 17 November 2015 at 09:07, Guido van Rossum <guido@python.org> wrote:
So I dropped the ball on this too -- I was going to either have a look or tell you quickly to find a BDFL-delegate, but ended up doing neither. Skimming it now I really don't think I'm the right person to review this. Maybe you could ask Alex Gaynor?
If Alex is interested, that could definitely work. I've also asked Christian Heimes if he might be interested in handling it, as he worked on several of the SSL module changes that were backported, and he also works on FreeIPA [1] now, which will potentially be affected by the changes to RHEL & CentOS 7. Regards, Nick. [1] FreeIPA is an open source Identity Management & Authentication system: http://www.freeipa.org/
On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
Regards, Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015
Abstract ========
PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour.
Rationale =========
PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor).
Requirements for capability detection =====================================
As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade ===================================================================
Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions =========================================================
Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting.
Recommended file location -------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is ``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format -----------------------
The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present.
Example implementation ----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
* run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch.
Combining the recommendations =============================
If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled.
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context
# Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright =========
This document has been placed into the public domain.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-). So, Alex, do you want to approve this PEP? On Mon, Nov 16, 2015 at 3:54 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 17 November 2015 at 09:07, Guido van Rossum <guido@python.org> wrote:
So I dropped the ball on this too -- I was going to either have a look or tell you quickly to find a BDFL-delegate, but ended up doing neither. Skimming it now I really don't think I'm the right person to review this. Maybe you could ask Alex Gaynor?
If Alex is interested, that could definitely work. I've also asked Christian Heimes if he might be interested in handling it, as he worked on several of the SSL module changes that were backported, and he also works on FreeIPA [1] now, which will potentially be affected by the changes to RHEL & CentOS 7.
Regards, Nick.
[1] FreeIPA is an open source Identity Management & Authentication system: http://www.freeipa.org/
On Tue, Nov 10, 2015 at 4:47 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
Regards, Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015
Abstract ========
PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour.
Rationale =========
PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor).
Requirements for capability detection =====================================
As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade ===================================================================
Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions =========================================================
Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting.
Recommended file location -------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is ``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format -----------------------
The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present.
Example implementation ----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
* run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch.
Combining the recommendations =============================
If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled.
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context
# Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright =========
This document has been placed into the public domain.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
2015-11-17 1:00 GMT+01:00 Guido van Rossum <guido@python.org>:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
Can we try to get a vote from maintainers of the Python2/3 packages of other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.? Victor
On 17 November 2015 at 20:33, Victor Stinner <victor.stinner@gmail.com> wrote:
2015-11-17 1:00 GMT+01:00 Guido van Rossum <guido@python.org>:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
Can we try to get a vote from maintainers of the Python2/3 packages of other Linux distributions? Debian, Ubuntu, OpenSUSE, etc.?
I know Oracle were interested based on a discussion between them and a member of Red Hat's product security team about it on oss-security, but their devs never followed up on it upstream (even after an explicit suggestion that they do so), so I'm interpreting that as willingness to go along with whatever happens in RHEL. For Debian, Ubuntu and SUSE, their original determinations for the relevant CVE were "too intrusive to backport", so folks currently need to upgrade to newer versions of those distros to get the improved default behaviour: * http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html * https://security-tracker.debian.org/tracker/CVE-2014-9365 * https://www.suse.com/security/cve/CVE-2014-9365.html If having an opt-in backwards-compatible-by-default approach available (albeit as a PEP 466+476+493 patch set in the RHEL/CentOS system Python 2.7.5 package) prompts other distro security teams to reconsider those initial assessments, that would be a nice outcome, but it isn't my own main priority (so Guido makes a good point in favouring finding a non-Red-Hatter willing to act as BDFL-Delegate) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11 November 2015 at 10:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
RHEL 7.2 is out today, together with details of what we've now committed to supporting for certification verification configuration in the RHEL 7 system Python: https://access.redhat.com/articles/2039753 That article also describes the PEP 476 status across the different Python builds Red Hat supports, which I've summarised below. Versions with PEP 476 implemented: * Python 3.4 SCL: cert verification always enabled by default * system Python in RHEL 7.2+: still off by default due to the compatibility break, but with PEP 493's file based configuration setting to enable it by default Versions without PEP 476 implemented: * Python 2.7 SCL * Python 3.3 SCL * system Python in RHEL 7.1 and earlier * system Python in all versions of RHEL 6 I assume that status of the Python 2.7 SCL will change at some point, but don't have an ETA or any technical details to share at this point. The system Python versions in RHEL 5 and earlier didn't include the ssl module at all (since that was only added to the standard library in Python 2.6), so they're not affected by the concerns raised in PEP 476 in the first place. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2015-11-17 01:00, Guido van Rossum wrote:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
So, Alex, do you want to approve this PEP?
I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat. Alex is a great choice. In the same mail I sent Nick a quick review of the latest PEP version in private. 1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag. 2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows? 3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too. 4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later. 5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs. It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS. Christian
Hi all, While I appreciate the vote of confidence from everyone, I'm not interested in being the BDFL-delegate for this. I don't think it's a good idea, and I'm not willing to put further time into. If he's interested, Donald Stufft would make a good choice for delegate. Really do appreciate everyone's confidence. Cheers, Alex On Mon, Nov 23, 2015 at 2:35 PM, Christian Heimes <christian@python.org> wrote:
On 2015-11-17 01:00, Guido van Rossum wrote:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
So, Alex, do you want to approve this PEP?
I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat. Alex is a great choice.
In the same mail I sent Nick a quick review of the latest PEP version in private.
1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag.
2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows?
3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too.
4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later.
5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs. It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS.
Christian
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
1. Does this affect easy_install? 2. If/because this affects easy_install, should the guidance / suggested package installation tool be [pip]; because pip install_requires backports.ssl_match_hostname https://pypi.python.org/pypi/backports.ssl_match_hostname On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
Regards, Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015
Abstract ========
PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour.
Rationale =========
PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor).
Requirements for capability detection =====================================
As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade ===================================================================
Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions =========================================================
Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting.
Recommended file location -------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is ``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format -----------------------
The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present.
Example implementation ----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
* run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch.
Combining the recommendations =============================
If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled.
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context
# Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright =========
This document has been placed into the public domain.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
... Just had this discussion in regards to easy_install, Ubuntu 14.04 LTS, and the ReadTheDocs Docker images (as well as: ~why should I have to wget/curl get-pip.py) https://github.com/rtfd/readthedocs-docker-images/pull/3 On Nov 23, 2015 2:47 PM, "Wes Turner" <wes.turner@gmail.com> wrote:
1. Does this affect easy_install? 2. If/because this affects easy_install, should the guidance / suggested package installation tool be [pip]; because pip install_requires backports.ssl_match_hostname
https://pypi.python.org/pypi/backports.ssl_match_hostname On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS verification backport proposals in PEP 493, and let the upstream and downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates the HTTPS verification feature backport based on the current PEP 493 draft, even though that hasn't formally been pronounced as an Active recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now: https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either didn't care about the question (because they're using unmodified upstream versions so the PEP didn't affect them), or else thought the approach described in the PEP was reasonable, so I'm hoping the consequences of my mistake won't be too severe.
Regards, Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a point where it's too late to realistically say "No", but the truth is that preparation for the Python in Education miniconf at PyCon Australia ramped up immediately after the July discussion, and then I personally got confused as to the scope of what was being included in 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493 being deferred to a later release, but it's actually the whole package of 466+476+493). That's my fault for trying to keep track of too many things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493 Title: HTTPS verification recommendations for Python 2.7 redistributors Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Informational Content-Type: text/x-rst Created: 10-May-2015 Post-History: 06-Jul-2015
Abstract ========
PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour.
Rationale =========
PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings.
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the new settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case of interest to the authors of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes two alternative approaches that redistributors may take when addressing these problems. Redistributors may choose to implement one, both, or neither of these approaches based on their assessment of the needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor).
Requirements for capability detection =====================================
As these recommendations are intended to cover backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, the recommendations are defined to allow the presence or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade ===================================================================
Some redistributors may wish to provide a per-application option to disable certificate verification in selected applications that run on or embed CPython without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively turned off for particular applications after upgrading to a version of Python that verifies certificates by default * the ability for all users to configure this setting on a per-application basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7, including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to providing a per-application configuration setting for HTTPS certificate verification that doesn't require modifications to the application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory(): config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the vulnerable configuration used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions =========================================================
Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default
This approach should not be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default.
Required marker attribute -------------------------
The required marker attribute on the ``ssl`` module when implementing this recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name.
Recommended modifications to the Python standard library --------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting.
Recommended file location -------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is ``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format -----------------------
The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present.
Example implementation ----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations -----------------------
The specific recommendations for the backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration:
* run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being)
Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch.
Combining the recommendations =============================
If a redistributor chooses to implement both recommendations, then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user, virtual environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled.
Example implementation ----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory(): # Check for am environmental override of the default behaviour config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context
# Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright =========
This document has been placed into the public domain.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf On 24 November 2015 at 05:35, Christian Heimes <christian@python.org> wrote:
1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag.
I guarded the relevant sections in the examples with "if not sys.flags.ignore_environment:"
2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows?
The environment variable section should be generic, and the configuration file section is largely specific to vendors offering long term commercial support options for Linux distros. There was already a qualification in the "Recommended file location" section limiting the scope to *nix systems, so rather than trying to cover non-Linux systems, I've instead updated that qualification to limit the recommendation even further to specifically Linux systems. If *BSD folks pipe up saying they'd like to be included, then adding another filename recommendation wouldn't be difficult.
3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too.
This is a system level configuration setting to preserve backwards compatible behaviour in the system Python, so scoping it at the interpreter level was a deliberate design decision. However, I added a new section to both recommendations describing the lack of interaction with virtual environments.
4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later.
For future releases, I think a different filename makes sense, as we don't really want a global "turn off HTTPS certificate verification" flag. Perhaps something like "/etc/python/network-security.cfg", for example.
5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs.
I added a note to clarify that the section name comes from the "https" URL schema passed to the relevant client APIs.
It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS.
That changed in PEP 476 - there's a separate private context for HTTPS standard library clients now, which allows HTTPS to verify hostnames by default, while other clients are still permissive. This is noted in PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols The reason for this is that browsers have provided a pretty good forcing function in getting folks to use properly signed certificates, at least on the public internet, but self-signed and improperly signed certificates are still significantly more common for other protocols. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 November 2015 at 06:47, Wes Turner <wes.turner@gmail.com> wrote:
1. Does this affect easy_install?
easy_install has validated certificates since distribute was merged back into the project as part of setuptools 0.7 [1], and aside from one issue with HTTPS tunnelling [2], the certificate verification code has been stable since setuptools 1.3 [3].
2. If/because this affects easy_install, should the guidance / suggested package installation tool be [pip]; because pip install_requires backports.ssl_match_hostname
setuptools/easy_install uses backports.ssl_match_hostname if it's available, and otherwise has its own implementation. Cheers, Nick. [1] https://pythonhosted.org/setuptools/history.html#id159 [2] https://pythonhosted.org/setuptools/history.html#id80 [3] https://pythonhosted.org/setuptools/history.html#id123 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 November 2015 at 05:35, Christian Heimes <christian@python.org> wrote:
On 2015-11-17 01:00, Guido van Rossum wrote:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
So, Alex, do you want to approve this PEP?
I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat.
Likewise, but the intersection between "wants to get PEP 476 into the hands of as many system operators as possible as soon as possible", "is a CPython core developer", and "doesn't work for Red Hat" is looking to be a rather select group :) Since we already know Red Hat are OK with the draft recommendations, and I missed the RHEL 7.2 release date anyway, perhaps Barry or Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable release update windmill? I know there was previously a decision from Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the stability risks [1], but the configuration file based approach recommended in PEP 493 is backwards compatible by default, with the decision to opt in to the improved settings after upgrading current systems being made by system administrators rather than the distro vendor. With around 3 1/2 years still to run on 14.04's support lifecycle, that has the potential to reach quite a few systems that otherwise wouldn't benefit from the change until well after Ubuntu 16.04 is released next year. Regards, Nick. [1] http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-9365.html -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Nov 23, 2015 at 5:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 24 November 2015 at 06:47, Wes Turner <wes.turner@gmail.com> wrote:
1. Does this affect easy_install?
easy_install has validated certificates since distribute was merged back into the project as part of setuptools 0.7 [1], and aside from one issue with HTTPS tunnelling [2], the certificate verification code has been stable since setuptools 1.3 [3].
Got it; thanks. * https://bitbucket.org/pypa/setuptools/src/tip/setuptools/ssl_support.py
2. If/because this affects easy_install, should the guidance / suggested package installation tool be [pip]; because pip install_requires backports.ssl_match_hostname
setuptools/easy_install uses backports.ssl_match_hostname if it's available, and otherwise has its own implementation.
* setuptools 0.7 [src] -- 2013-06-?? -- https://bitbucket.org/pypa/setuptools/src/7f2c08e9/?at=0.7 * setuptools 0.7 [distro] -- 201X-XX-XX [see: whohas, ] Setuptools latest is now at version 18.5 (2015-11-02).
Cheers, Nick.
[1] https://pythonhosted.org/setuptools/history.html#id159 [2] https://pythonhosted.org/setuptools/history.html#id80 [3] https://pythonhosted.org/setuptools/history.html#id123
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote:
Since we already know Red Hat are OK with the draft recommendations, and I missed the RHEL 7.2 release date anyway, perhaps Barry or Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable release update windmill? I know there was previously a decision from Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the stability risks [1], but the configuration file based approach recommended in PEP 493 is backwards compatible by default
Right, but this isn't a patch we'd particularly want to carry ourselves. Maybe if it were available upstream, tried and tested, it could be considered for backporting, but it still wouldn't be zero cost. We'd have to also handle migration paths to newer Ubuntu releases, which probably means removing the config file on future upgrades. There's also the possibility of implementing different defaults on new installs of 14.04 versus upgrades to 14.04. And even if a system administrator enabled it for one particular application, it could break other applications on the same machine, so it just punts a difficult decision down the line. We're also not seeing much (any?) demand from our users, and the initial attempt at turning this on by default *did* get a strong negative reaction because of the compatibility break. I'm concerned about accepting PEP 493 making a strong recommendation to downstreams. Yes, in an ideal world we all want security by default, but I think the backward compatibility concerns of the PEP are understated, especially as they relate to a maintenance release of a stable long term support version of the OS. I don't want PEP 493 to be a cudgel that people beat us up with instead of having an honest discussion of the difficult trade-offs involved. Having said all that, I think informing people of the issue, and letting any future reconsideration be demand driven is the right approach for now. $0.02-ly y'rs, -Barry
On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote:
For Debian, Ubuntu and SUSE, their original determinations for the relevant CVE were "too intrusive to backport", so folks currently need to upgrade to newer versions of those distros to get the improved default behaviour:
This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing personal!). "Improved default behavior"... for whom? It's not improved for the folks whose applications are broken by changing the default. Cheers, -Barry
On 24 November 2015 at 12:05, Barry Warsaw <barry@python.org> wrote:
On Nov 17, 2015, at 11:44 PM, Nick Coghlan wrote:
For Debian, Ubuntu and SUSE, their original determinations for the relevant CVE were "too intrusive to backport", so folks currently need to upgrade to newer versions of those distros to get the improved default behaviour:
This is an example of my problem with the tone of PEP 493 (sorry Nick, nothing personal!). "Improved default behavior"... for whom? It's not improved for the folks whose applications are broken by changing the default.
I think there are three relevant categories here: 1. Folks who assume that "https" means the same thing in Python that it means in web browsers, and are currently experiencing a silent security failure 2. Folks who already know it doesn't, and are relying on that to keep their infrastructure working 3. Folks currently in group 2 who would like to improve their infrastructure to default to verifying certificates The upstream change in PEP 476 was driven by the interests of folks in group 1. This makes a lot of sense, given that a popular way of introducing folks to programming now is by writing programs that interact with web APIs over HTTPS. Rebasing is fine for reaching desktop audiences, so this group should already have been covered by the updated python.org binaries for Windows and Mac OS X, new binary releases from the cross-platform redistributors, and updates to non-LTS Linux distros. For folks in group 2, the main goal is "Don't break their stuff as a side effect of updating a supported version", which is why PEP 493 recommends leaving certificate verification disabled by default if backporting the PEP 476 changes. The target audience for PEP 493 is then the folks in group 3: infrastructure operators that *want* to turn failures to validate certificates in Python applications into a noisy failure, but *don't* want to have to switch to a new major version of their entire base operating system to do it. With the PEP, opting in to PEP 476 for Python 2 applications and services is just a configuration file setting, readily managed with any configuration management tool. If just the configuration file approach is supported (as is the case for RHEL 7.2), then that decision needs to be made on a system-wide basis, and you need to use a chroot (or a full Linux container) in order to opt out (or in) for a single application. If the environment variable downgrade is also supported, then the environments for individual applications can be tweaked, even while the overall environment is upgraded to complain about missing security checks by default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 November 2015 at 11:59, Barry Warsaw <barry@python.org> wrote:
On Nov 24, 2015, at 10:18 AM, Nick Coghlan wrote:
Since we already know Red Hat are OK with the draft recommendations, and I missed the RHEL 7.2 release date anyway, perhaps Barry or Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable release update windmill? I know there was previously a decision from Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the stability risks [1], but the configuration file based approach recommended in PEP 493 is backwards compatible by default
Right, but this isn't a patch we'd particularly want to carry ourselves. Maybe if it were available upstream, tried and tested, it could be considered for backporting, but it still wouldn't be zero cost.
If anyone wants to look through the patches implementing the backport, they're available through the CentOS git repositories: https://git.centos.org/commit/rpms!python.git/refs!heads!c7
We'd have to also handle migration paths to newer Ubuntu releases, which probably means removing the config file on future upgrades.
You'll need to figure out those migration paths anyway, as the certificate verification compatibility break will exist between 14.04 and 16.04 regardless of what happens with PEP 493.
There's also the possibility of implementing different defaults on new installs of 14.04 versus upgrades to 14.04.
For those aspects, we left the defaults in RHEL 7.2 as implementing the 2.7.5 behaviour (i.e. no cert verification), but used the "platform default" mechanism in PEP 493 to ensure we had the ability to change our mind if circumstances change. For the immediately foreseeable future though, it will be up to operations teams to decide how they want the Python 2.7 standard library to behave with respect to HTTPS client certificate verification on RHEL & CentOS 7.
And even if a system administrator enabled it for one particular application, it could break other applications on the same machine, so it just punts a difficult decision down the line.
We're also not seeing much (any?) demand from our users, and the initial attempt at turning this on by default *did* get a strong negative reaction because of the compatibility break.
From a change management perspective, there's also a very big difference between "We made this decision for you, and you'll need to update your systems accordingly in order to upgrade", and "We made it
My own main concern is providing an additional layer of defence against certificate verification related security bugs in network services and clients written in Python like OpenStack, Ansible, FreeIPA, Pulp, etc, which is why I'm happy with the approach of giving system operators the tools they need to make their own decision about whether or not they want that additional layer of protection on a given host. Folks building Docker images can also easily opt in or out by including a suitable configuration file. possible for you to make your own decision about this new configuration setting, based on what you think is best for your particular situation". There's still some risk in the latter case (since you may accidentally break things), but the upstream releases flushed out a lot of those problems for this particular update (e.g. the gevent compatibility issue that affected the upstream PEP 466 backport). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 November 2015 at 03:46, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think there are three relevant categories here:
1. Folks who assume that "https" means the same thing in Python that it means in web browsers, and are currently experiencing a silent security failure 2. Folks who already know it doesn't, and are relying on that to keep their infrastructure working 3. Folks currently in group 2 who would like to improve their infrastructure to default to verifying certificates
I'm not directly interested in this PEP (any code I write that is relevant will run on Windows) but there's another group: 4. People who use https because it's the published API, but are only looking at internal systems (where the threat window is minimised) and the internal infrastructure is out of their control but insecure. Browser users and other apps have to go through pain to work (accepting self-signed certs, repeatedly responding "Yes" to security warnings). Python code written for internal use just works at the moment. Maybe it "shouldn't", in some abstract sense, but it *does* and that is important to users. This group may know that they are taking risks, and may want to improve the infrastructure, but they *can't*. And this change breaks their apps (possibly in ways they can't fix easily - not all client environments offer ways to add trust, there's a reason tools like wget have "ignore certificate checks" flags even though everyone knows they are a bad idea). The push for "secure by default" is important and needed, but needs to be handled in a more pragmatic way, recognising that Python is used in situations like the above, where people know they aren't doing the right thing, but have no better options. Shutting down their least bad option on a point of principle isn't helpful. Paul
On 2015-11-24 01:18, Nick Coghlan wrote:
On 24 November 2015 at 05:35, Christian Heimes <christian@python.org> wrote:
On 2015-11-17 01:00, Guido van Rossum wrote:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
So, Alex, do you want to approve this PEP?
I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat.
Likewise, but the intersection between "wants to get PEP 476 into the hands of as many system operators as possible as soon as possible", "is a CPython core developer", and "doesn't work for Red Hat" is looking to be a rather select group :)
Right, with Antoine and Alex out of scope and you, Victor and me working for Red Hat, the air is getting thin. Benjamin is familiar with the ssl module. Or we can follow Alex's advice and ask somebody from the PyCA group (Donald, Paul, lvh) or requests (Cory) to get some outside perspective.
Since we already know Red Hat are OK with the draft recommendations, and I missed the RHEL 7.2 release date anyway, perhaps Barry or Matthias might be interested in tilting at the Ubuntu 14.04 LTS stable release update windmill? I know there was previously a decision from Ubuntu Security not to backport PEPs 466 & 476 to 2.7.5 due to the stability risks [1], but the configuration file based approach recommended in PEP 493 is backwards compatible by default, with the decision to opt in to the improved settings after upgrading current systems being made by system administrators rather than the distro vendor. With around 3 1/2 years still to run on 14.04's support lifecycle, that has the potential to reach quite a few systems that otherwise wouldn't benefit from the change until well after Ubuntu 16.04 is released next year.
Yes, that makes a lot of sense. Christian
On Nov 24, 2015, at 7:53 AM, Christian Heimes <christian@python.org> wrote:
On 2015-11-24 01:18, Nick Coghlan wrote:
On 24 November 2015 at 05:35, Christian Heimes <christian@python.org> wrote:
On 2015-11-17 01:00, Guido van Rossum wrote:
Hm, making Christian the BDFL-delegate would mean two out of three authors *and* the BDFL-delegate all working for Red Hat, which clearly has a stake (and IIUC has already committed to this approach ahead of PEP approval). SO then it would look like this is just rubber-stamping Red Hat's internal decision process (if it's a process -- sounds more like an accident :-).
So, Alex, do you want to approve this PEP?
I haven't read this thread until now. Independently from your objection I have raised the same concern with Nick today. I'd be willing to BDFL the PEP but I'd rather have somebody outside of Red Hat.
Likewise, but the intersection between "wants to get PEP 476 into the hands of as many system operators as possible as soon as possible", "is a CPython core developer", and "doesn't work for Red Hat" is looking to be a rather select group :)
Right, with Antoine and Alex out of scope and you, Victor and me working for Red Hat, the air is getting thin. Benjamin is familiar with the ssl module. Or we can follow Alex's advice and ask somebody from the PyCA group (Donald, Paul, lvh) or requests (Cory) to get some outside perspective.
Under normal circumstances I'd probably be willing to do it even though I have doubts about the value of it. However, I'm less than two weeks away from closing on a house and will be moving into it after that, so my time is very limited right now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 24 Nov 2015 8:12 pm, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 24 November 2015 at 03:46, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think there are three relevant categories here:
1. Folks who assume that "https" means the same thing in Python that it means in web browsers, and are currently experiencing a silent security failure 2. Folks who already know it doesn't, and are relying on that to keep their infrastructure working 3. Folks currently in group 2 who would like to improve their infrastructure to default to verifying certificates
I'm not directly interested in this PEP (any code I write that is relevant will run on Windows) but there's another group:
4. People who use https because it's the published API, but are only looking at internal systems (where the threat window is minimised) and the internal infrastructure is out of their control but insecure. Browser users and other apps have to go through pain to work (accepting self-signed certs, repeatedly responding "Yes" to security warnings). Python code written for internal use just works at the moment. Maybe it "shouldn't", in some abstract sense, but it *does* and that is important to users.
This group may know that they are taking risks, and may want to improve the infrastructure, but they *can't*. And this change breaks their apps (possibly in ways they can't fix easily - not all client environments offer ways to add trust, there's a reason tools like wget have "ignore certificate checks" flags even though everyone knows they are a bad idea).
I believe you're referring mainly to the original PEP 476 change there. In the context of PEP 493, this is another group that would potentially benefit from the suggested "security downgrade" environment variable (if any redistributors decide to implement that - RHEL doesn't as yet), since it would provide a way to restore the old behaviour without changing their client code or monkeypatching the SSL module as described in PEP 476. Regards, Nick.
On 24 November 2015 at 13:20, Nick Coghlan <ncoghlan@gmail.com> wrote:
I believe you're referring mainly to the original PEP 476 change there. In the context of PEP 493, this is another group that would potentially benefit from the suggested "security downgrade" environment variable (if any redistributors decide to implement that - RHEL doesn't as yet), since it would provide a way to restore the old behaviour without changing their client code or monkeypatching the SSL module as described in PEP 476.
I'm actually referring to the fact that your classification didn't seem to include people who have no control over their infrastructure (except in class 1 which implies ignorance rather than powerlessness...). PEP 493 is of benefit to such people, so there's now downside in explicitly noting this. My concern is that *because* people consistently forget about the class of people who have to put up with bad infrastructure but can't do anything about it, we risk promoting a sense of "security as the enemy" - which is the direct opposite of what we're trying to do. I have no interest or opinion regarding this PEP itself, but I would like to see "people who have to put up with whatever infrastructure they are dumped with, and use Python to ease that burden" recognised as an important class of user. They are very under-represented in discussions, as it's usually big business closed source and similar environments that are in that situation. Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-)) Apologies, this is a bit of a hobby horse of mine, I'll pipe down now. Paul.
In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-))
But does it help them? Or does it increase the power of those who hand out certificates and who are intensely security conscious over those who would like to get some work done this afternoon? Laura
On 25 November 2015 at 00:27, Laura Creighton <lac@openend.se> wrote:
In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-))
But does it help them? Or does it increase the power of those who hand out certificates and who are intensely security conscious over those who would like to get some work done this afternoon?
In situations where IT are still the "Department of No", rather than focusing on facilitating folks in getting their work done, I think the most likely outcome of the configuration file recommendation in PEP 493 is preservation of the status quo: admins simply won't change the config setting, even if they deploy a version of Linux that adopts the approach suggested in the PEP. If they do enable full certificate verification (or upgrade the environments they provide to a version of Python that has it enabled by default) without doing appropriate compatibility testing first, then they're going to hit the compatibility problems Paul is talking about. The aspect of the PEP that has the potential to help in the case of poor infrastructure management is providing the ability to globally turn off certificate verification on a per-process basis. It's the networking equivalent of monkeypatching - you know there are risks with doing it, but also judge the near term benefits to outweigh those longer term risks in your particular situation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I think the PEP is a good step forward to compromise between the crypto purists (use whatever technologies makes us more secure even if it breaks things) and those who cannot upgrade their Python 2.7 because of the PEP 476 change, since it causes their applications to fail (e.g. because the embedded devices they want to interface to only support self signed certs). I would still find having built-in support for the recommendations in the Python stdlib a better approach, but PEP 493 is good enough in at least solving real problems people are having. PS: Would be great to have a PyPI package which implements these recommendations so that you can simply add it as dependency - and then please for Python 3 as well, since people with embedded devices will want to be able to use Python 3 as well ;-) On 24.11.2015 15:27, Laura Creighton wrote:
In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-))
But does it help them? Or does it increase the power of those who hand out certificates and who are intensely security conscious over those who would like to get some work done this afternoon?
Laura _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 24 2015)
Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
On 2015-11-24 00:47, Nick Coghlan wrote:
Updated version of the PEP posted: https://hg.python.org/peps/rev/8decac213ebf
On 24 November 2015 at 05:35, Christian Heimes <christian@python.org> wrote:
1) The example implementation of the function doesn't check the sys.flags.ignore_environment. Internally CPython has specialized getenv function that ignores env vars with PYTHON prefix when the flag is set. PYTHON* env vars aren't removed from os.environ. Modules have to check the flag.
I guarded the relevant sections in the examples with "if not sys.flags.ignore_environment:"
Thanks!
2) The PEP is rather Linux-centric. What's the recommended path to the config file on other platforms like BDS (/usr/local/etc/ is preferred for additional dependencies on FreeBSD), OSX and Windows?
The environment variable section should be generic, and the configuration file section is largely specific to vendors offering long term commercial support options for Linux distros.
There was already a qualification in the "Recommended file location" section limiting the scope to *nix systems, so rather than trying to cover non-Linux systems, I've instead updated that qualification to limit the recommendation even further to specifically Linux systems. If *BSD folks pipe up saying they'd like to be included, then adding another filename recommendation wouldn't be difficult.
3) What's the interaction between the location of the config file and virtual envs? Would it make sense to search for the file in a venv's etc/ first and then dispatch to global /etc/? That way venvs can influence the setting, too.
This is a system level configuration setting to preserve backwards compatible behaviour in the system Python, so scoping it at the interpreter level was a deliberate design decision. However, I added a new section to both recommendations describing the lack of interaction with virtual environments.
I mentioned 2) and 3) because I suspect that some people will want to use the new setting to configure applications. You know how people think. They get a new shiny tool and then they abuse it as a hammer to drive in all their problems. :)
4) It makes sense to make the cert-verification.cfg file future-proof and reserve it for other cert-related configuration in the future. For example it could be used to define new contexts, set protocols, ciphers or hashes for cert pinning. It should be enough to say that CPython reserves the right to add more sections and keys later.
For future releases, I think a different filename makes sense, as we don't really want a global "turn off HTTPS certificate verification" flag. Perhaps something like "/etc/python/network-security.cfg", for example.
Are you planning to remove the "disable verification flag" in the future?
5) I'm not particular fond of the section name [https]. For one It is ambiguous because it doesn't distinguish between server certs and client certs.
I added a note to clarify that the section name comes from the "https" URL schema passed to the relevant client APIs.
It's also not correct. The default context is used for other protocols like imap, smtp etc. over TLS.
That changed in PEP 476 - there's a separate private context for HTTPS standard library clients now, which allows HTTPS to verify hostnames by default, while other clients are still permissive. This is noted in PEP 476: https://www.python.org/dev/peps/pep-0476/#other-protocols
The reason for this is that browsers have provided a pretty good forcing function in getting folks to use properly signed certificates, at least on the public internet, but self-signed and improperly signed certificates are still significantly more common for other protocols.
I'm sorry, I forgot about PEP 476 and the separate context function for HTTPS. Somehow I expected the setting to influence all TLS/SSL connection, not just HTTPS. I should have read the topic line. In the light of the new information the setting name makes sense. Christian
On 24 November 2015 at 14:27, Laura Creighton <lac@openend.se> wrote:
In a message of Tue, 24 Nov 2015 14:05:53 +0000, Paul Moore writes:
Simply adding "people who have no control over their broken infrastructure" with a note that this PEP helps them, would be sufficient here (and actually helps the case for the PEP, so why not? ;-))
But does it help them? Or does it increase the power of those who hand out certificates and who are intensely security conscious over those who would like to get some work done this afternoon?
My reading is that if fully implemented (and Nick has already confirmed that Red Hat didn't do this) it would add an environment variable that would allow the user to (in effect) say "I can't fix my security infrastructure, so just leave me alone and let me take the risk". So in theory this PEP would give back some of the ability to ignore the problem that previous PEPs took away. (And by "ignore the problem", here I mean "just try to get some work done in spite of a security and infrastructure group that don't understand how to implement security and infrastructure, dammit!") Like it or not, in many organisations, security and development are a huge "us and them" battle. For me, it's important that Python doesn't take sides in that battle, while still offering education to anyone willing to listen. (All I've learned about security is as a result of working with Python - sadly, that knowledge has not made my job one iota easier, it's just increased my stress levels :-() Paul
On Mon, Nov 23, 2015 at 5:59 PM, Barry Warsaw <barry@python.org> wrote:
I'm concerned about accepting PEP 493 making a strong recommendation to downstreams. Yes, in an ideal world we all want security by default, but I think the backward compatibility concerns of the PEP are understated, especially as they relate to a maintenance release of a stable long term support version of the OS. I don't want PEP 493 to be a cudgel that people beat us up with instead of having an honest discussion of the difficult trade-offs involved.
It sounds like the implementation sections of the PEP are acceptable but that the PEP's general tone seems to assume that distributors are champing at the bit to backport and that the recommendations here make it so that backporting is a no-brainer -- which does not seem to reflect the real-world? I think the tone could be changed to address that as it doesn't seem like forcing distros to backport is a real goal of the PEP. The main purposes of the PEP seem to be: * Enumerate several ways that distributors can backport these 2.7.9 features to older releases * Allow programmers to detect the presence of the features from their code * Give end-users the ability to choose between backwards compatibility and enhanced security Here's some ideas for changing the tone: Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates to be appropriate for communication over the public internet. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. + Change to "PEP 476 updated Python's default handling of HTTPS certificates to validate that the certs belonged to the server". This way we're saying what the change is rather than making a value judgement of whether people who don't choose to backport are "appropriate" or not. Appropriate-ness is probably best left as an argument in the text of PEP 476. This PEP provides recommendations to downstream redistributors wishing to provide a smoother migration experience when helping their users to manage this change in Python's default behaviour. + Change to "downstream redistributors wishing to backport the enhancements in a way that allows users to choose between backwards compatible behaviour or more secure certificate handling." As barry noted, this PEP doesn't change the amount of work needed to migrate. It does, however, give users some choice in when they are going to perform that work. Additionally, this isn't simply about distributors who want to make the transition smoother... (there's no downstreams that want to make it "more painful" are there? ;-) It's really about making backporting of the enhancements less painful for users. Rationale ========= PEP 476 changed Python's default behaviour to better match the needs and expectations of developers operating over the public internet, a category which appears to include most new Python developers. It is the position of the authors of this PEP that this was a correct decision. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. + per barry's mesage, it would be good to either devote a paragraph to the backwards compatibility implications here or link to https://www.python.org/dev/peps/pep-0476/#backwards-compatibility The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure. + The wording here seems to reflect a different scope than merely backporting by distros. Perhaps we should change it to: "[...]set by the public internet. Distributions may wish to help these sites transition by backporting the PEP 476 changes to earlier versions of python in a way that does not require the administrators to upgrade their certificate management infrastructure immediately. This would allow sites to choose to use the distribution suppiied python in a backwards compatible fashion until their certificate management infrastructure was updated and then toggle their site to utilize the more secure features provided by PEP 476." [...] These designs are being proposed as a recommendation for redistributors, rather than as new upstream features, as they are needed purely to support legacy environments migrating from older versions of Python 2.7. Neither approach is being proposed as an upstream Python 2.7 feature, nor as a feature in any version of Python 3 (whether published directly by the Python Software Foundation or by a redistributor). + This paragraph seems a little out of place here as it is not about why this PEP is useful so much as who will implement it. It also seems to tie into Abstract closely as it reinforces and expands on where this is going to make a difference. Perhaps move it to Abstract or create a "Scope" section that can follow abstract and contain this information. wording can be changed a bit to show that it isn't expected that all redistributors are expected to follow this but only those who are already backporting: "These recommendations are being proposed to redistributors who wish to backport PEP 476 features rather than as a new upstream feature as they are needed purely to support migrating legacy environments from older versions of python 2.7" -Toshio
On 24 November 2015 at 17:16, Toshio Kuratomi <a.badger@gmail.com> wrote:
The long term answer for such environments is to update their internal certificate management to at least match the standards set by the public internet, but in the meantime, it is desirable to offer these administrators a way to continue receiving maintenance updates to the Python 2.7 series, without having to gate that on upgrades to their certificate management infrastructure.
+ The wording here seems to reflect a different scope than merely backporting by distros. Perhaps we should change it to: "[...]set by the public internet. Distributions may wish to help these sites transition by backporting the PEP 476 changes to earlier versions of python in a way that does not require the administrators to upgrade their certificate management infrastructure immediately. This would allow sites to choose to use the distribution suppiied python in a backwards compatible fashion until their certificate management infrastructure was updated and then toggle their site to utilize the more secure features provided by PEP 476."
I'm not actually sure that it's the place of this PEP to even comment on what the long term answer for such environments should be (or indeed, any answer, long term or not). I've argued the position that in some organisations it feels like security don't actually understand the issues of carefully balancing secure operation against flexible development practices, but conversely it's certainly true that in many organisations, they *have* weighed the various arguments and made an informed decision on how to set up their internal network. It's entirely possible that self-signed certificates are entirely the right decision for their circumstances. Why would a Python PEP be qualified to comment on that decision? In my opinion, we should take all of the value judgements out of this paragraph, and just state the facts. How about: """ In order to provide additional flexibility to allow infrastructure administrators to provide the appropriate solution for their environment, this PEP offers a way for administrators to upgrade to later versions of the Python 2.7 series without being forced to update their existing security certificate management infrastructure as a prerequisite. """ Paul
On Tue, Nov 24, 2015 at 10:08 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'm not actually sure that it's the place of this PEP to even comment on what the long term answer for such environments should be (or indeed, any answer, long term or not). I've argued the position that in some organisations it feels like security don't actually understand the issues of carefully balancing secure operation against flexible development practices,
I agree with this.
but conversely it's certainly true that in many organisations, they *have* weighed the various arguments and made an informed decision on how to set up their internal network. It's entirely possible that self-signed certificates are entirely the right decision for their circumstances. Why would a Python PEP be qualified to comment on that decision?
I don't quite agree with this but it's probably moot in the face of the previous and certain cornercases. Self-signed certs work just fine with the backports to python-2.7.9+ but you have to add the ca to the clients. An organization that has weighed the arguments and made an informed decision to use self-signed certs should either do this (to prevent MITM) or they should switch to using http instead of https (because MITM isn't a feasible attack here). The cornercases come into play because you don't always control all of the devices and services on your network. The site could evaluate and decide that MITM isn't a threat to their switch's configuration interface but that interface might be served over https using a certificate signed by their network vendor who doesn't give out their ca certificate (simply stated: your security team knows what they're doing but your vendor's does not).
In my opinion, we should take all of the value judgements out of this paragraph, and just state the facts. How about:
""" In order to provide additional flexibility to allow infrastructure administrators to provide the appropriate solution for their environment, this PEP offers a way for administrators to upgrade to later versions of the Python 2.7 series without being forced to update their existing security certificate management infrastructure as a prerequisite. """
Two notes: * python-2.7.9+ doesn't give you flexibility in this regard so organizations do have to update their certificate management infrastructure. The cornercase described above becomes something that has to be addressed at the code level. Environments that are simply misconfigured have to be fixed. So in that regard, a value judgement does seem appropriate here. the judgement is "Listen guys, this PEP advises redistributors on how they might provide a migration path for you but it *does not bandaid the problem indefinitely*. So long term, you have to change your practices or you'll be out in the cold when your redistributor upgrades to python-2.7.9+" * Your proposed text actually removes the fix that I was adding -- this version of the paragraph implies that if your environment is compatible with the redistributors' python-2.7.8 (or less) then it will also be compatible with the redistributors' python-2.7.9+. That is not true. Whether or not we take out any value judgement as to the user's present environment this paragraph needs to be fixed to make it clear that this only affects redistributor's packages which have backported pep 476 to python-2.7.8 or older. Once the redistributor updates to a newer python sites which relied on this crutch will break. -Toshio
On 24 November 2015 at 18:37, Toshio Kuratomi <a.badger@gmail.com> wrote:
I don't quite agree with this but it's probably moot in the face of the previous and certain cornercases. Self-signed certs work just fine with the backports to python-2.7.9+ but you have to add the ca to the clients. An organization that has weighed the arguments and made an informed decision to use self-signed certs should either do this (to prevent MITM) or they should switch to using http instead of https (because MITM isn't a feasible attack here).
There's an assumption (which is untrue for the big organisations I'm familiar with) that security are involved with the full stack - typically in my experience "corporate security" set rules that are filtered down through many layers, and development teams have no influence on those decisions - conversely, the people setting security policies aren't aware of the details of development day to day needs. It's easy to say that users just click on the "allow" button in their browser if they want to use self-signed certificates, but not have any viable solution for automation code that doesn't use a browser - and such automation code is "not corporately supported", so policy won't get updated to address the disconnect.
The cornercases come into play because you don't always control all of the devices and services on your network. The site could evaluate and decide that MITM isn't a threat to their switch's configuration interface but that interface might be served over https using a certificate signed by their network vendor who doesn't give out their ca certificate (simply stated: your security team knows what they're doing but your vendor's does not).
This sounds like a similar situation to what I described above. I'm not sure I'd see these as corner cases, though - they are pretty much day to day business in my experience :-(
In my opinion, we should take all of the value judgements out of this paragraph, and just state the facts. How about:
""" In order to provide additional flexibility to allow infrastructure administrators to provide the appropriate solution for their environment, this PEP offers a way for administrators to upgrade to later versions of the Python 2.7 series without being forced to update their existing security certificate management infrastructure as a prerequisite. """
Two notes:
* python-2.7.9+ doesn't give you flexibility in this regard so organizations do have to update their certificate management infrastructure. The cornercase described above becomes something that has to be addressed at the code level. Environments that are simply misconfigured have to be fixed. So in that regard, a value judgement does seem appropriate here. the judgement is "Listen guys, this PEP advises redistributors on how they might provide a migration path for you but it *does not bandaid the problem indefinitely*. So long term, you have to change your practices or you'll be out in the cold when your redistributor upgrades to python-2.7.9+"
Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is of limited interest to me). I thought that the environment variable gave developers a "get out" clause. Maybe it's not what we want them to do (for some value of "we") but isn't that the point of the PEP? Admittedly if distributions don't *implement* that part of the PEP (and I understand Red Hat haven't) then people are still stuck. But "this PEP offers a way" is not incompatible with "your vendor didn't implement the PEP so you're still stuck, sorry"...
* Your proposed text actually removes the fix that I was adding -- this version of the paragraph implies that if your environment is compatible with the redistributors' python-2.7.8 (or less) then it will also be compatible with the redistributors' python-2.7.9+. That is not true. Whether or not we take out any value judgement as to the user's present environment this paragraph needs to be fixed to make it clear that this only affects redistributor's packages which have backported pep 476 to python-2.7.8 or older. Once the redistributor updates to a newer python sites which relied on this crutch will break.
Sorry for that. Certainly getting the facts right is crucial, and it looks like my suggestion didn't do that. But hopefully someone can fix it up (if people think it's a good way to go). Paul
On Tue, Nov 24, 2015 at 10:56 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 24 November 2015 at 18:37, Toshio Kuratomi <a.badger@gmail.com> wrote:
The cornercases come into play because you don't always control all of the devices and services on your network. The site could evaluate and decide that MITM isn't a threat to their switch's configuration interface but that interface might be served over https using a certificate signed by their network vendor who doesn't give out their ca certificate (simply stated: your security team knows what they're doing but your vendor's does not).
This sounds like a similar situation to what I described above. I'm not sure I'd see these as corner cases, though - they are pretty much day to day business in my experience :-(
It sounds like you're coming from a Windows background and I'm coming from a Linux background which might be a small disconnect here -- we do seem to be in agreement that what's "right to do" isn't always easy or possible for the client to accomplish so I think we should probably leave it at that.
In my opinion, we should take all of the value judgements out of this paragraph, and just state the facts. How about:
""" In order to provide additional flexibility to allow infrastructure administrators to provide the appropriate solution for their environment, this PEP offers a way for administrators to upgrade to later versions of the Python 2.7 series without being forced to update their existing security certificate management infrastructure as a prerequisite. """
Two notes:
* python-2.7.9+ doesn't give you flexibility in this regard so organizations do have to update their certificate management infrastructure. The cornercase described above becomes something that has to be addressed at the code level. Environments that are simply misconfigured have to be fixed. So in that regard, a value judgement does seem appropriate here. the judgement is "Listen guys, this PEP advises redistributors on how they might provide a migration path for you but it *does not bandaid the problem indefinitely*. So long term, you have to change your practices or you'll be out in the cold when your redistributor upgrades to python-2.7.9+"
Hmm, maybe I misread the PEP (I only skimmed it - as I say, Linux is of limited interest to me). I thought that the environment variable gave developers a "get out" clause. Maybe it's not what we want them to do (for some value of "we") but isn't that the point of the PEP?
Admittedly if distributions don't *implement* that part of the PEP (and I understand Red Hat haven't) then people are still stuck. But "this PEP offers a way" is not incompatible with "your vendor didn't implement the PEP so you're still stuck, sorry"...
Yeah, I think you are correct in your understanding of what actual changes to the python distrribution are being proposed for redistributors in the PEP. Reading through the PEP again, I'm not sure if I'm correct in thinking that this only applies to backporting... it may be that the environment section of the PEP applies to any python-2 while the config file section only applies to backporting. Nick, could you clarify? The PEP is clear that it doesn't apply to python-3 or cross-distro. So that means that sites still can't rely on this long-term (but their long term would extend to the lifetime of their vendor supporting python2 rather than when their vendor updated to 2.7.9+) and also that developers can't depend on this if they're developing portable code.
* Your proposed text actually removes the fix that I was adding -- this version of the paragraph implies that if your environment is compatible with the redistributors' python-2.7.8 (or less) then it will also be compatible with the redistributors' python-2.7.9+. That is not true. Whether or not we take out any value judgement as to the user's present environment this paragraph needs to be fixed to make it clear that this only affects redistributor's packages which have backported pep 476 to python-2.7.8 or older. Once the redistributor updates to a newer python sites which relied on this crutch will break.
Sorry for that. Certainly getting the facts right is crucial, and it looks like my suggestion didn't do that. But hopefully someone can fix it up (if people think it's a good way to go).
Could be that I'm wrong -- will wait for Nick to clarify before I think about what could be done to make this wording better. -Toshio
On 25 Nov 2015 05:42, "Toshio Kuratomi" <a.badger@gmail.com> wrote:
Yeah, I think you are correct in your understanding of what actual changes to the python distrribution are being proposed for redistributors in the PEP. Reading through the PEP again, I'm not sure if I'm correct in thinking that this only applies to backporting... it may be that the environment section of the PEP applies to any python-2 while the config file section only applies to backporting. Nick, could you clarify?
Yep, the environment variable section is the part MAL persuaded me was a good idea during the original discussion of this PEP, based on the interests of eGenix's customers. It's a cross-platform suggestion for distributors wanting to provide a smoother upgrade path, applicable to both backports and rebasing on 2.7.9+. (I originally didn't like it, but became amenable once it was pointed out that anyone with the ability to set process environment variables can already alter OpenSSL's settings to use a different CA cert file or directory) The config file recommendation is specific to Linux distro backports, with the aim of saying "IF you backport PEP 476 to an older Python 2.7 point release, THEN you should do it like this for cross-distro consistency". As the current discussion shows, I'm *personally* an advocate for backporting, but I don't think the PEP itself should reflect that - the suggestions to adjust the tone are good ones, so I'll update the text accordingly. Cheers, Nick.
On 24 Nov 2015, at 12:53, Christian Heimes <christian@python.org> wrote: Right, with Antoine and Alex out of scope and you, Victor and me working for Red Hat, the air is getting thin. Benjamin is familiar with the ssl module. Or we can follow Alex's advice and ask somebody from the PyCA group (Donald, Paul, lvh) or requests (Cory) to get some outside perspective.
I’m not a CPython core developer, so I think that excludes me. For what it’s worth, I think that this is a good PEP, and I’m strongly in favour of it being approved in some form.
There's a lot to process in this thread, but as I see it, the issue breaks down to these questions: * How should PEP 493 be implemented? * What should the default be? * How should PEP 493 be worded to express the right tone to redistributors? Let me take on the implementation details here. On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
I would still find having built-in support for the recommendations in the Python stdlib a better approach
As would I. Currently the PEP contains some sample code for reading a configuration file and an environment variable, and assigning the default behavior based on these values. The first improvement to the stdlib I would have made would have been to add a convenience function to ssl to enable and disable the verification. Something like this: def ssl.verify_certificates(enable) Then at the simplest level, an application that wants to enable higher security could do this early in its main entry point: import ssl if hasattr(ssl, 'verify_certificates'): ssl.verify_certificates(True) or ... ssl.verify_certificates(False) and that would work regardless of the platform default, and it would work properly on older versions of Python 2.7 that didn't have the backport. This seems so straightforward, I'm wondering why PEP 493 doesn't consider it. I would like to see the sample code in PEP 493 implemented in the stdlib. It would then be a matter of cherry picking that should a distro decide to backport it. It means one less non-trivial delta from upstream which has to be managed separately. The only difference to the combined approach I'd make is to handle the case where the _cert_verification_config file doesn't exist. (If you included the convenience method above, then the sample code could be simplified down to a default for the `enable` parameter and a call to ssl.verify_certificates().) Is it too late to include both of those suggestions in PEP 493 and Python 2.7.11? Cheers, -Barry
On 26 November 2015 at 08:57, Barry Warsaw <barry@python.org> wrote:
There's a lot to process in this thread, but as I see it, the issue breaks down to these questions:
* How should PEP 493 be implemented?
* What should the default be?
* How should PEP 493 be worded to express the right tone to redistributors?
Let me take on the implementation details here.
On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
I would still find having built-in support for the recommendations in the Python stdlib a better approach
As would I.
For what its worth: a PEP telling distributors to patch the standard library is really distasteful to me. We've spent a long time trying to build close relations such that when something doesn't work distributors can share their needs with us and we can make Python out of the box be a good fit. This seems to fly in the exact opposite direction: we're explicitly making it so that Python builds on these vendor's platforms will not be the same as you get by checking out the Python source code. Ugh. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Converged Cloud
On Thu, 26 Nov 2015 09:17:02 +1300, Robert Collins <robertc@robertcollins.net> wrote:
On 26 November 2015 at 08:57, Barry Warsaw <barry@python.org> wrote:
There's a lot to process in this thread, but as I see it, the issue breaks down to these questions:
* How should PEP 493 be implemented?
* What should the default be?
* How should PEP 493 be worded to express the right tone to redistributors?
Let me take on the implementation details here.
On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
I would still find having built-in support for the recommendations in the Python stdlib a better approach
As would I.
For what its worth: a PEP telling distributors to patch the standard library is really distasteful to me.
We've spent a long time trying to build close relations such that when something doesn't work distributors can share their needs with us and we can make Python out of the box be a good fit. This seems to fly in the exact opposite direction: we're explicitly making it so that Python builds on these vendor's platforms will not be the same as you get by checking out the Python source code.
I think we should include the environment variable support in CPython and be done with it (nuke the PEP otherwise). Which is what I've thought from the beginning :) --David
On Nov 24, 2015, at 09:16 AM, Toshio Kuratomi wrote:
It sounds like the implementation sections of the PEP are acceptable
Well, see my follow up to MAL's email. From Rob's and RDM's responses, it seems we're not alone. :) I like all your other proposed changes, although I'll note the preference to minimize the unique-to-downstream changes. Cheers, -Barry
On Nov 24, 2015, at 02:12 PM, Nick Coghlan wrote:
On 24 November 2015 at 11:59, Barry Warsaw <barry@python.org> wrote:
We'd have to also handle migration paths to newer Ubuntu releases, which probably means removing the config file on future upgrades.
You'll need to figure out those migration paths anyway, as the certificate verification compatibility break will exist between 14.04 and 16.04 regardless of what happens with PEP 493.
Sure, but that's a very different situation. If someone upgraded from 14.04.3 to 14.04.4, they definitely wouldn't expect it to break working code. If it did, it would be a pretty serious regression. Upgrading to LTS maintenance releases should be a no-brainer. Upgrading from one LTS to a newer one, you *know* things are going to change, so you'll review the release notes and do some very careful analysis of your mission critical code. Cheers, -Barry
On Nov 25, 2015, at 03:39 PM, R. David Murray wrote:
I think we should include the environment variable support in CPython and be done with it (nuke the PEP otherwise). Which is what I've thought from the beginning :)
+1. I'd like to see my proposed convenience function added too. I'm -0 on the config file though since while I think it might be useful, I'm not sure we'd actually install that on Ubuntu. Cheers, -Barry
On 26 November 2015 at 05:57, Barry Warsaw <barry@python.org> wrote:
There's a lot to process in this thread, but as I see it, the issue breaks down to these questions:
* How should PEP 493 be implemented?
* What should the default be?
* How should PEP 493 be worded to express the right tone to redistributors?
Let me take on the implementation details here.
On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
I would still find having built-in support for the recommendations in the Python stdlib a better approach
As would I.
Currently the PEP contains some sample code for reading a configuration file and an environment variable, and assigning the default behavior based on these values.
The first improvement to the stdlib I would have made would have been to add a convenience function to ssl to enable and disable the verification. Something like this:
def ssl.verify_certificates(enable)
Then at the simplest level, an application that wants to enable higher security could do this early in its main entry point:
import ssl if hasattr(ssl, 'verify_certificates'): ssl.verify_certificates(True)
or
... ssl.verify_certificates(False)
and that would work regardless of the platform default, and it would work properly on older versions of Python 2.7 that didn't have the backport. This seems so straightforward, I'm wondering why PEP 493 doesn't consider it.
PEP 476 rejected providing a public indefinitely maintained API for this, so PEP 493 is specifically about helping commercial redistributors offer a smoother transition plan to customers without affecting the public Python level API, and without encouraging a plethora of mutually incompatible transition schemes.
I would like to see the sample code in PEP 493 implemented in the stdlib. It would then be a matter of cherry picking that should a distro decide to backport it. It means one less non-trivial delta from upstream which has to be managed separately. The only difference to the combined approach I'd make is to handle the case where the _cert_verification_config file doesn't exist.
(If you included the convenience method above, then the sample code could be simplified down to a default for the `enable` parameter and a call to ssl.verify_certificates().)
Is it too late to include both of those suggestions in PEP 493 and Python 2.7.11?
PEP 493 isn't about attempting to rehash the PEP 476 discussions in search of a different conclusion, so this would need to be a different PEP, preferably one that targets Python 3.6 first and covers more than just HTTPS. Such an API could then be considered as a backport candidate for 2.7.12. (2.7.11rc1 has already been published, so there isn't any scope for updates there). I actually think that's a nicer approach, but also think the recommendations in PEP 493 are much simpler to implement and test, while still meeting the bar of "good enough to solve the problem at hand". If anyone does decide to write that PEP, then the fact that SELinux offers "setenforce 0" and AppArmor can be turned off with a kernel boot option can be used as examples of platform developers letting end users make their own security decisions, even if the platform provider recommends leaving the more opinionated default behaviour enabled. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 26 November 2015 at 06:17, Robert Collins <robertc@robertcollins.net> wrote:
On 26 November 2015 at 08:57, Barry Warsaw <barry@python.org> wrote:
There's a lot to process in this thread, but as I see it, the issue breaks down to these questions:
* How should PEP 493 be implemented?
* What should the default be?
* How should PEP 493 be worded to express the right tone to redistributors?
Let me take on the implementation details here.
On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
I would still find having built-in support for the recommendations in the Python stdlib a better approach
As would I.
For what its worth: a PEP telling distributors to patch the standard library is really distasteful to me.
I'm not a big fan of it either, but it's the way sustainable commercial open source distribution works in practice: customers want selective upgrades, so customers get selective upgrades; customers want greater platform integration, so customers get greater platform integration; etc, etc. Large Python shops like Google will typically maintain their own custom Python variants with their own patches applied that are suitable and sufficient for their use cases, but may not be suitable for pushing upstream (e.g. they may be Linux specific patches, without support for other platforms). That's why I've focused on providing guidance to redistributors to keep their patches within reasonably consistent bounds for PEPs like bundling pip, backporting the network security enhancements to Python 2.7, and changing the default behaviour for HTTPS verification. Different redistributors have different customer bases, which also differ from the audience for upstream python.org releases, so saying "don't patch Python" is denying the reality of the obligations commercial vendors have to their customers, while "when you patch Python, please ensure you abide by these guidelines" is something redistributors can realistically do.
We've spent a long time trying to build close relations such that when something doesn't work distributors can share their needs with us and we can make Python out of the box be a good fit. This seems to fly in the exact opposite direction: we're explicitly making it so that Python builds on these vendor's platforms will not be the same as you get by checking out the Python source code.
This has always been true for the RHEL system Python - it gets backported changes, just like the kernel does (albeit nowhere near as many). That's one of the main things long term support customers are paying for - developers continuing to care about a code base after the volunteers working on the project upstream have (entirely reasonably) shifted their attentions to newer versions. We try to keep the patches to a minimum, and avoid having to carry large patches indefinitely by only backporting changes that have already been accepted in later versions, but "never apply patches to upstream projects as a downstream redistributor" isn't a realistic goal. In this particular case, the migration problems were already raised in the PEP 476 discussions, and the decision was made to *not* provide a public API specifically for globally turning off HTTPS certificate verification, since that would either: 1. Need to be added to Python 3 and maintained indefinitely; or 2. Be implemented in Python 2.7 only, and thus create another barrier to 2->3 migration A HTTPS specific flag wasn't considered a suitable long term API design for Python 3, while a multi-protocol configuration design that also handled setting the defaults for other protocols like secure SMTP, IMAP, POP3, SFTP, etc seemed generally desirable (but also doesn't exist anywhere, and poses signficant challenges in designing and implementing a good test matrix). The only concrete things that have changed since PEP 476 was approved is that: * we figured out the monkeypatching based migration recommendations in that PEP didn't actually work for the redistributor case * Robert Kuska came up with a way to do a backwards compatible backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to migrating from Python 2 to Python 3 PEP 493 originally only described the latter idea - I added MAL's design for the environment variable based security downgrade based on the original PEP 493 discussions in July. Splitting the latter out to its own PEP that proposes it as a forwards compatible configuration setting akin to "curl --insecure" and "wget --no-check-certificate" for Python 2.7.12, 3.4.4, 3.5.2 and 3.6 upstream would be fine by me, since that would also significantly clarify the fact that the two ideas are entirely orthogonal to each other (except for the need to define a relative priority when you combine the two). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 26, 2015, at 02:13 PM, Nick Coghlan wrote:
PEP 476 rejected providing a public indefinitely maintained API for this, so PEP 493 is specifically about helping commercial redistributors offer a smoother transition plan to customers without affecting the public Python level API, and without encouraging a plethora of mutually incompatible transition schemes.
Of course, the API would only have to be support for the life of 2.7; it would never go in 3.x so the burden is minimal.
PEP 493 isn't about attempting to rehash the PEP 476 discussions in search of a different conclusion, so this would need to be a different PEP, preferably one that targets Python 3.6 first and covers more than just HTTPS.
That seems like overkill. PEP 493 is specifically about Python 2.7 and providing ways for downstreams to facilitate more choice for end-users and end-administrators. Although I think it could safely sneak in after rc1, that would be for the RM to decide. Even if it were deferred to 2.7.12, it would still provide a better, more consistent experience if implemented upstream. Cheers, -Barry
On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:
I'm not a big fan of it either, but it's the way sustainable commercial open source distribution works in practice:
While it's inevitable that redistributors have to deviate from upstream, in Debian and Ubuntu, we really try to keep that at a minimum. Debian, of course is an all volunteer organization so our "customers" are just normal users who I think want an experience as close to installing from source as possible, consistent with the principles and policies of the Debian project. Debian at least tries to document the handful of deviations from upstream. Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and document them when we must deviate. Ubuntu is a community driven distro so while Canonical itself has customers, it's much more likely that feedback about the Python stack comes from ordinary users. Again, my personal goal is to make Python on Ubuntu a pleasant and comfortable environment, as close to installing from source as possible, consistent with the principles and policies of the project. All this to say that IMHO, solutions to problems should go as far upstream as possible. Guidelines certainly are useful (e.g. PEP 394) to maintain consistency across downstream providers where the decisions are primarily in the sphere of influence of the downstreams, but IMHO there's still more that upstream can do to promote PEP 493.
Different redistributors have different customer bases, which also differ from the audience for upstream python.org releases, so saying "don't patch Python" is denying the reality of the obligations commercial vendors have to their customers, while "when you patch Python, please ensure you abide by these guidelines" is something redistributors can realistically do.
ISTM that the same forces are in play regardless of whether the change is in code or in an informational PEP. Best to get consensus where possible, and manifest those decisions in code, but if competing goals are the outcome of a code change or informational PEP, downstream consumers will still make what they judge to be the best decision in the interest of their users, balanced against their own competing constraints.
In this particular case, the migration problems were already raised in the PEP 476 discussions, and the decision was made to *not* provide a public API specifically for globally turning off HTTPS certificate verification, since that would either:
1. Need to be added to Python 3 and maintained indefinitely; or 2. Be implemented in Python 2.7 only, and thus create another barrier to 2->3 migration
A minor one, for sure. It wouldn't significantly bother me if the API were underscore-prefixed as a big honkin' clue about the guarantees being made.
* Robert Kuska came up with a way to do a backwards compatible backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to migrating from Python 2 to Python 3
I'm not sure I agree with that. If you have to set an environment variable to get your application to work in Python 2.7, it's *still* a migration issue when you move to Python 3. You're just moving the change out of the code and into the environment, where it's arguably more difficult to re-discover down the road. Cheers, -Barry
On 27 November 2015 at 03:15, Barry Warsaw <barry@python.org> wrote:
On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:
I'm not a big fan of it either, but it's the way sustainable commercial open source distribution works in practice:
While it's inevitable that redistributors have to deviate from upstream, in Debian and Ubuntu, we really try to keep that at a minimum. Debian, of course is an all volunteer organization so our "customers" are just normal users who I think want an experience as close to installing from source as possible, consistent with the principles and policies of the Debian project. Debian at least tries to document the handful of deviations from upstream.
Yeah, we need to be better about that - while https://access.redhat.com/articles/2039753 covers the details for the certificate verification changes, there's currently no easy way to get a description of the changes backported to the RHEL/CentOS system Python without trawling through old Errata announcements :(
Likewise in Ubuntu, we try to keep deviations from Debian at a minimum, and document them when we must deviate. Ubuntu is a community driven distro so while Canonical itself has customers, it's much more likely that feedback about the Python stack comes from ordinary users. Again, my personal goal is to make Python on Ubuntu a pleasant and comfortable environment, as close to installing from source as possible, consistent with the principles and policies of the project.
I'd strongly agree with that description for Fedora and softwarecollections.org, but for the RHEL/CentOS system Python I think the situation is slightly different: there, the goal is to meet the long term support commitments involved in being a base RHEL package. As the nominal base version of the package (2.7.5 in the case of RHEL 7) doesn't change, there is naturally going to be increasing divergence from the nominal version.
All this to say that IMHO, solutions to problems should go as far upstream as possible. Guidelines certainly are useful (e.g. PEP 394) to maintain consistency across downstream providers where the decisions are primarily in the sphere of influence of the downstreams, but IMHO there's still more that upstream can do to promote PEP 493.
I tried to go down the "upstream first" path with a properly supported "off switch" in PEP 476, and didn't succeed (hence the monkeypatch compromise). It sounds like several folks would like to see us revisit that decision, though.
Different redistributors have different customer bases, which also differ from the audience for upstream python.org releases, so saying "don't patch Python" is denying the reality of the obligations commercial vendors have to their customers, while "when you patch Python, please ensure you abide by these guidelines" is something redistributors can realistically do.
ISTM that the same forces are in play regardless of whether the change is in code or in an informational PEP. Best to get consensus where possible, and manifest those decisions in code, but if competing goals are the outcome of a code change or informational PEP, downstream consumers will still make what they judge to be the best decision in the interest of their users, balanced against their own competing constraints.
In this particular case, the migration problems were already raised in the PEP 476 discussions, and the decision was made to *not* provide a public API specifically for globally turning off HTTPS certificate verification, since that would either:
1. Need to be added to Python 3 and maintained indefinitely; or 2. Be implemented in Python 2.7 only, and thus create another barrier to 2->3 migration
A minor one, for sure. It wouldn't significantly bother me if the API were underscore-prefixed as a big honkin' clue about the guarantees being made.
OK, you've persuaded me to recast the PEP from an Informational one to a Standards track one. The section about backporting to versions prior to 2.7.9 won't change much (except in tone), but the ability to opt out on a process-wide basis will be pitched as a language level feature.
* Robert Kuska came up with a way to do a backwards compatible backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to migrating from Python 2 to Python 3
I'm not sure I agree with that. If you have to set an environment variable to get your application to work in Python 2.7, it's *still* a migration issue when you move to Python 3. You're just moving the change out of the code and into the environment, where it's arguably more difficult to re-discover down the road.
By that I meant that setting the environment variable won't *break* anything in Python 3, whereas if we add a new Python level API and people call it without a hasattr() guard, then even if their infrastructure has been upgraded to use verifiable certificates, the client applications will break on Python 3. However, with an underscore prefix and an admonition to use a hasattr() check, I agree a 2.7.x-only Python level API isn't really any more of a barrier to migration than only offering the environment variable. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 26, 2015 4:53 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 27 November 2015 at 03:15, Barry Warsaw <barry@python.org> wrote:
Likewise in Ubuntu, we try to keep deviations from Debian at a minimum,
and
document them when we must deviate. Ubuntu is a community driven distro so while Canonical itself has customers, it's much more likely that feedback about the Python stack comes from ordinary users. Again, my personal goal is to make Python on Ubuntu a pleasant and comfortable environment, as close to installing from source as possible, consistent with the principles and policies of the project.
I'd strongly agree with that description for Fedora and softwarecollections.org, but for the RHEL/CentOS system Python I think the situation is slightly different: there, the goal is to meet the long term support commitments involved in being a base RHEL package. As the nominal base version of the package (2.7.5 in the case of RHEL 7) doesn't change, there is naturally going to be increasing divergence from the nominal version.
I think the goal in rhel/centos is similar, actually. The maintenance burden for non upstream changes has been acknowledged as a problem to be avoided by rhel maintainers before. The caveat for those distributions is that they accumulate more *backports*. However, backports are easier to maintain than non upstream changes. The test of the upstream community helps to find and fix bugs in the code; the downstream maintainer just needs to stay aware of whether fixes are going into the code they've backported.
I tried to go down the "upstream first" path with a properly supported "off switch" in PEP 476, and didn't succeed (hence the monkeypatch compromise). It sounds like several folks would like to see us revisit that decision, though.
That's the rub. If there's now enough support to push this upstream I think everyone downstream will be happier. If it turns out there's still enough resistance to keep it from upstream then I suppose you cross that bridge if it becomes necessary. -Toshio
On 27 November 2015 at 10:52, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 27 November 2015 at 03:15, Barry Warsaw <barry@python.org> wrote:
On Nov 26, 2015, at 03:06 PM, Nick Coghlan wrote:
In this particular case, the migration problems were already raised in the PEP 476 discussions, and the decision was made to *not* provide a public API specifically for globally turning off HTTPS certificate verification, since that would either:
1. Need to be added to Python 3 and maintained indefinitely; or 2. Be implemented in Python 2.7 only, and thus create another barrier to 2->3 migration
A minor one, for sure. It wouldn't significantly bother me if the API were underscore-prefixed as a big honkin' clue about the guarantees being made.
OK, you've persuaded me to recast the PEP from an Informational one to a Standards track one. The section about backporting to versions prior to 2.7.9 won't change much (except in tone), but the ability to opt out on a process-wide basis will be pitched as a language level feature.
New draft pushed: https://hg.python.org/peps/rev/f602a47ea795 This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead). There are then 3 backport recommendation sections: * one for backporting the ability to enable HTTPS verification by default for the entire Python installation * one for backporting the new features described in this PEP * one for combining both backports in a single implementation I tried to trim some of the resulting duplication between the sections, but I suspect the overall text is still longer than it needs to be. I've also added links describing the origin of the configuration file based backport in RHEL development, and where to find the source code for the patches in Fedora. Cheers, Nick. =================================== PEP: 493 Title: HTTPS verification migration tools for Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André Lemburg <mal@lemburg.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015 Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015 Abstract ======== PEP 476 updated Python's default handling of HTTPS certificates in client modules to align with certificate handling in web browsers, by validating that the certificates received belonged to the server the client was attempting to contact. The Python 2.7 long term maintenance series was judged to be in scope for this change, with the new behaviour introduced in the Python 2.7.9 maintenance release. This has created a non-trivial barrier to adoption for affected Python 2.7 maintenance releases, so this PEP proposes additional Python 2.7 specific features that allow system administrators and other users to more easily decouple the decision to verify server certificates in HTTPS client modules from the decision to update to newer Python 2.7 maintenance releases. Rationale ========= PEP 476 changed Python's default behaviour to align with expectations established by web browsers in regards to the semantics of HTTPS URLs: starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library validate server certificates by default. However, it is also the case that this change *does* cause problems for infrastructure administrators operating private intranets that rely on self-signed certificates, or otherwise encounter problems with the new default certificate verification settings. To manage these kinds of situations, web browsers provide users with "click through" warnings that allow the user to add the server's certificate to the browser's certificate store. Network client tools like ``curl`` and ``wget`` offer options to switch off certificate checking entirely (by way of ``curl --insecure`` and ``wget --no-check-certificate``, respectively). At a different layer of the technology stack, Linux security modules like `SELinux` and `AppArmor`, while enabled by default by distribution vendors, offer relatively straightforward mechanisms for turning them off. At the moment, no such convenient mechanisms exist to disable Python's default certificate checking for a whole process. PEP 476 did attempt to address this question, by covering how to revert to the old settings process wide by monkeypatching the ``ssl`` module to restore the old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed to allow system administrators to disable the feature by default in their Standard Operating Environment definition has been determined to be insufficient in at least some cases. The specific case that led to the initial creation of this PEP is the one where a Linux distributor aims to provide their users with a `smoother migration path <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__ than the standard one provided by consuming upstream CPython 2.7 releases directly, but other potential challenges have also been pointed out with updating embedded Python runtimes and other user level installations of Python. Rather than allowing a plethora of mutually incompatibile migration techniques to bloom, this PEP proposes an additional feature to be added to Python 2.7.12 to make it easier to revert a process to the past behaviour of skipping certificate validation in HTTPS client modules. It also provides additional recommendations to redistributors backporting these features to versions of Python prior to Python 2.7.9. These designs are being proposed purely as tools for helping to manage the transition to the new default certificate handling behaviour in the context of Python 2.7. They are not being proposed as new features for Python 3, as it is expected that the vast majority of client applications affected by this problem without the ability to update the application itself will be Python 2 applications. It would likely be desirable for a future version of Python 3 to allow default certificate handling for secure protocols to be configurable on a per-protocol basis, but that question is beyond the scope of this PEP. Alternatives ------------ In the absence of clear upstream guidance and recommendations, commercial redistributors will still make their own design decisions in the interests of their customers. The main approaches available are: * Continuing to rebase on new Python 2.7.x releases, while providing no additional assistance beyond the mechanisms defined in PEP 476 in migrating from unchecked to checked hostnames in standard library HTTPS clients * Gating availability of the changes in default handling of HTTPS connections on upgrading from Python 2 to Python 3 * For Linux distribution vendors, gating availability of the changes in default handling of HTTPS connections on upgrading to a new operating system version * Implementing one or both of the backport suggestions described in this PEP, regardless of the formal status of the PEP Requirements for capability detection ===================================== As the proposals in this PEP aim to facilitate backports to earlier Python versions, the Python version number cannot be used as a reliable means for detecting them. Instead, they are designed to allow the presence or absence of the feature to be determined using the following technique:: python -c "import ssl; ssl._relevant_attribute" This will fail with `AttributeError` (and hence a non-zero return code) if the relevant capability is not available. The marker attributes are prefixed with an underscore to indicate the implementation dependent nature of these capabilities - not all Python distributions will offer them, only those that are providing a multi-stage migration process from the original Python 2.7 HTTPS handling to the new default behaviour. Feature: Configuration API ========================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``ssl._verify_https_certificates()`` to specify the default handling of HTTPS certificates in standard library client libraries. It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is the ``ssl._verify_https_certificates`` function itself. Specification ------------- The ``ssl._verify_https_certificates`` function will work as follows:: def _verify_https_certificates(enable=True): """Verify server HTTPS certificates by default?""" global _create_default_https_context if enable: _create_default_https_context = create_default_context else: _create_default_https_context = _create_unverified_context If called without arguments, or with ``enable`` set to a true value, then standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification. If called with ``enable`` set to a false value, then standard library client modules will subsequently skip verifying HTTPS certificates by default. Security Considerations ----------------------- The inclusion of this feature will allow security sensitive applications to include the following forward-compatible snippet in their code:: if hasattr(ssl, "_verify_https_certificates"): ssl._verify_https_certificates() Some developers may also choose to opt out of certificate checking using ``ssl._verify_https_certificates(enable=False)``. This doesn't introduce any major new security concerns, as monkeypatching the affected internal APIs was already possible. Feature: environment based configuration ======================================== This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that allows the default verification to be disabled without modifying the application source code (which may not even be available in cases of bytecode-only application distribution) It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is: * the ``ssl._https_verify_envvar`` attribute, giving the name of environment variable affecting the default behaviour This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant environment variable name. Specification ------------- Rather than always defaulting to the use of ``ssl.create_default_context``, the ``ssl`` module will be modified to: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` if this environment variable is present and set to ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` as usual Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting == '0': return _create_unverified_context return create_default_context _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this approach does introduce a new downgrade attack against the default security settings that potentially allows a sufficiently determined attacker to revert Python to the default behaviour used in CPython 2.7.8 and earlier releases. However, such an attack requires the ability to modify the execution environment of a Python process prior to the import of the ``ssl`` module, and any attacker with such access would already be able to modify the behaviour of the underlying OpenSSL implementation. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Backporting PEP 476 to earlier Python versions ============================================== Some redistributors, most notably Linux distributions, may choose to backport the PEP 476 HTTPS verification changes to modified Python versions based on earlier Python 2 maintenance releases. In these cases, a configuration mechanism is needed that provides: * an opt-in model that allows the decision to enable HTTPS certificate verification to be made independently of the decision to upgrade to the Python version where the feature was first backported * the ability for system administrators to set the default behaviour of Python applications and scripts run directly in the system Python installation * the ability for the redistributor to consider changing the default behaviour of *new* installations at some point in the future without impacting existing installations that have been explicitly configured to skip verifying HTTPS certificates by default As it only affects backports to earlier releases of Python 2.7, this change is not proposed for inclusion in upstream CPython, but rather is offered as guidance to redistributors to reduce the likelihood of multiple mutually incompatible approaches to backporting being adopted. This approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will validate HTTPS certificates by default. Feature detection ----------------- The marker attribute on the ``ssl`` module related to this feature is:: _cert_verification_config = '<path to configuration file>' This not only makes it straightforward to detect the presence (or absence) of the capability, it also makes it possible to programmatically determine the relevant configuration file name. Recommended modifications to the Python standard library -------------------------------------------------------- The recommended approach to backporting the PEP 476 modifications to an earlier point release is to implement the following changes relative to the default PEP 476 behaviour implemented in Python 2.7.9+: * modify the ``ssl`` module to read a system wide configuration file when the module is first imported into a Python process * define a platform default behaviour (either verifying or not verifying HTTPS certificates) to be used if this configuration file is not present * support selection between the following three modes of operation: * ensure HTTPS certificate verification is enabled * ensure HTTPS certificate verification is disabled * delegate the decision to the redistributor providing this Python version * set the ``ssl._create_default_https_context`` function to be an alias for either ``ssl.create_default_context`` or ``ssl._create_unverified_context`` based on the given configuration setting. Recommended file location ------------------------- As the PEP authors are not aware of any vendors providing long-term support releases targeting Windows, Mac OS X or \*BSD systems, this approach is currently only specifically defined for Linux system Python installations. The recommended configuration file name on Linux systems is ``/etc/python/cert-verification.cfg``. The ``.cfg`` filename extension is recommended for consistency with the ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library. Recommended file format ----------------------- The configuration file should use a ConfigParser ini-style format with a single section named ``[https]`` containing one required setting ``verify``. The suggested section name is taken from the "https" URL schema passed to affected client APIs. Permitted values for ``verify`` are: * ``enable``: ensure HTTPS certificate verification is enabled by default * ``disable``: ensure HTTPS certificate verification is disabled by default * ``platform_default``: delegate the decision to the redistributor providing this particular Python version If the ``[https]`` section or the ``verify`` setting are missing, or if the ``verify`` setting is set to an unknown value, it should be treated as if the configuration file is not present. Example implementation ---------------------- :: _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() Security Considerations ----------------------- The specific recommendations for this backporting case are designed to work for privileged, security sensitive processes, even those being run in the following locked down configuration: * run from a locked down administrator controlled directory rather than a normal user directory (preventing ``sys.path[0]`` based privilege escalation attacks) * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based privilege escalation attacks) * run using the ``-s`` switch (preventing user site directory based privilege escalation attacks) * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege escalation attacks) The intent is that the *only* reason HTTPS verification should be getting turned off system wide when using this approach is because: * an end user is running a redistributor provided version of CPython rather than running upstream CPython directly * that redistributor has decided to provide a smoother migration path to verifying HTTPS certificates by default than that being provided by the upstream project * either the redistributor or the local infrastructure administrator has determined that it is appropriate to override the default upstream behaviour (at least for the time being) Using an administrator controlled configuration file rather than an environment variable has the essential feature of providing a smoother migration path, even for applications being run with the ``-E`` switch. Interaction with Python virtual environments -------------------------------------------- This setting is scoped by the interpreter installation and affects all Python processes using that interpreter, regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Origins of this recommendation ------------------------------ This recommendation is based on the backporting approach adopted for Red Hat Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP and described in detail in `this KnowledgeBase article <https://access.redhat.com/articles/2039753>`__. Red Hat's patches implementing this backport for Python 2.7.5 can be found in the `CentOS git repository <https://git.centos.org/commit/rpms!python.git/refs!heads!c7>`__. Backporting this PEP to earlier Python versions =============================================== The configuration file based backport described above is designed to cover backporting the PEP 476 changes to default certificate handling without the additional configuration mechanisms defined in this PEP. If this PEP is accepted, then an additional backporting option becomes available, which is to backport the per-process configuration mechanisms defined in this PEP, without backporting the ability to change the default behaviour of the overall Python installation. Such a backport would differ from the mechanism proposed in this PEP solely in the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would continue to default to skipping certificate validation. In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and set to anything *other* than ``'0'``, then HTTPS certificate verification should be enabled. Feature detection ----------------- There's no specific attribute indicating that this situation applies. Rather, it is indicated by the ``ssl._verify_https_certificates`` and ``ssl._https_verify_envvar`` attributes being present in a Python version that is nominally older than Python 2.7.9. Specification ------------- Implementing this backport involves backporting the changes in PEP 466, 476 and this PEP, with the following change to the handling of the ``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module: * read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first imported into a Python process * set the ``ssl._create_default_https_context`` function to be an alias for ``ssl.create_default_context`` if this environment variable is present and set to any value other than ``'0'`` * otherwise, set the ``ssl._create_default_https_context`` function to be an alias for ``ssl._create_unverified_context`` Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' def _get_https_context_factory(): if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting != '0': return create_default_context return _create_unverified_context _create_default_https_context = _get_https_context_factory() def _disable_https_default_verification(): """Skip verification of HTTPS certificates by default""" global _create_default_https_context _create_default_https_context = _create_unverified_context Security Considerations ----------------------- This change would be a strict security upgrade for any Python version that currently defaults to skipping certificate validation in standard library HTTPS clients. The technical trade-offs to be taken into account relate largely to the magnitude of the PEP 466 backport also required rather than to anything security related. Interaction with Python virtual environments -------------------------------------------- The default setting is read directly from the process environment, and hence works the same way regardless of whether or not the interpreter is being run inside an activated Python virtual environment. Recommendation for combined feature backports ============================================= If a redistributor chooses to backport the environment variable based configuration setting from this PEP to a modified Python version that also implements the configuration file based PEP 476 , then the environment variable should take precedence over the system-wide configuration setting. This allows the setting to be changed for a given user or application, regardless of the installation-wide default behaviour. Example implementation ---------------------- :: _https_verify_envvar = 'PYTHONHTTPSVERIFY' _cert_verification_config = '/etc/python/cert-verification.cfg' def _get_https_context_factory(): # Check for an environmental override of the default behaviour if not sys.flags.ignore_environment: config_setting = os.environ.get(_https_verify_envvar) if config_setting is not None: if config_setting == '0': return _create_unverified_context return create_default_context # Check for a system-wide override of the default behaviour context_factories = { 'enable': create_default_context, 'disable': _create_unverified_context, 'platform_default': _create_unverified_context, # For now :) } import ConfigParser config = ConfigParser.RawConfigParser() config.read(_cert_verification_config) try: verify_mode = config.get('https', 'verify') except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): verify_mode = 'platform_default' default_factory = context_factories.get('platform_default') return context_factories.get(verify_mode, default_factory) _create_default_https_context = _get_https_context_factory() -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan writes:
This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead).
+1 on the same order of "words" in the function name and the evnironment variable. I tend to prefer "ssl._https_verify_certificates" since these aren't "HTTPS certificates".
Nick Coghlan writes:
This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead).
+1 on the same order of "words" in the function name and the evnironment variable. I tend to prefer "ssl._https_verify_certificates" since these aren't "HTTPS certificates".
On 27 November 2015 at 17:42, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Nick Coghlan writes:
This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead).
+1 on the same order of "words" in the function name and the evnironment variable. I tend to prefer "ssl._https_verify_certificates" since these aren't "HTTPS certificates".
Ah, yes, that would make sense, so I've just pushed an update that switches the function name order to match the environment variable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 27 Nov 2015, at 06:04, Nick Coghlan <ncoghlan@gmail.com> wrote:
Feature: Configuration API ==========================
This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x releases. It consists of a new ``ssl._verify_https_certificates()`` to specify the default handling of HTTPS certificates in standard library client libraries.
It is not proposed to forward port this change to Python 3, so Python 3 applications that need to support skipping certificate verification will still need to define their own suitable security context.
Feature detection -----------------
The marker attribute on the ``ssl`` module related to this feature is the ``ssl._verify_https_certificates`` function itself.
Specification -------------
The ``ssl._verify_https_certificates`` function will work as follows::
def _verify_https_certificates(enable=True): """Verify server HTTPS certificates by default?""" global _create_default_https_context if enable: _create_default_https_context = create_default_context else: _create_default_https_context = _create_unverified_context
If called without arguments, or with ``enable`` set to a true value, then standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification.
Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it…last import wins?
On 27 November 2015 at 18:47, Cory Benfield <cory@lukasa.co.uk> wrote:
Perhaps I missed this, Nick, but what happens if multiple third party libraries apply updates to call this function in incompatible ways? For example, if you depend on libfoo which calls ssl._verify_https_certificates(False) and libbar which calls ssl._verify_https_certificates(True)? Is it…last import wins?
Last import wins, but libaries shouldn't be mutating process global state as a side effect of import - like the sys module, the ssl module configuration should only be modified (directly or indirectly) from __main__. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Nov 27, 2015, at 04:04 PM, Nick Coghlan wrote:
New draft pushed: https://hg.python.org/peps/rev/f602a47ea795
This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead).
Thanks for this, and +1 on Stephen's suggested name change (which you've already pushed). Two comments: the PEP still describes the configuration file implementation. Is this slated for 2.7.12 also? If not, should it just be dropped from the PEP? I'd mildly prefer no default value for `enable` in _https_verify_certificates(). I'd have preferred a keyword-only argument, but of course this is Python 2. Instead, I'd like to force passing True or False (and document using `enable=True` or `enable=False`) and not rely on a default argument. But I'm only +0 on that detail. Cheers, -Barry
(Oops, I had a version of this reply sitting in my Drafts folder for a week, and only noticed after pushing the most recent PEP update that it had never been sent) On 1 December 2015 at 06:32, Barry Warsaw <barry@python.org> wrote:
On Nov 27, 2015, at 04:04 PM, Nick Coghlan wrote:
New draft pushed: https://hg.python.org/peps/rev/f602a47ea795
This is a significant rewrite that switches the PEP to a Standards Track PEP proposing two new features for 2.7.12+: an "ssl._verify_https_certificates()" configuration function, and a "PYTHONHTTPSVERIFY" environment variable (although writing them together like that makes me wonder if the latter should now be "PYTHONVERIFYHTTPS" instead).
Thanks for this, and +1 on Stephen's suggested name change (which you've already pushed).
Two comments: the PEP still describes the configuration file implementation. Is this slated for 2.7.12 also? If not, should it just be dropped from the PEP?
That recommendation is still needed to backport PEP 476 itself to versions prior to 2.7.9 - otherwise there's no way to flip the default from "don't verify" to "verify" for the entire Python installation. It may be that the system Python in RHEL/CentOS 7 and derivatives ends up being the only case where that happens (since that's the only instance I'm aware of with a 2024 support deadline, rather than 2019 or earlier), but if anyone else does do it, it would be preferable if they adopted the same approach to configuring it. However, I just pushed an update that reverses the presentation order of the two main backporting sections: https://hg.python.org/peps/rev/17e0e36cbc19 The original order came from the point where this was just an Informational PEP suggesting some backporting techniques, but now that it suggests some actual upstream changes for 2.7.12+, it makes more sense to cover those first, and then be more explicit that it's acceptable to skip implementing the rest of the PEP entirely. Accordingly, that change also includes the following new paragraph in the section on the PEP 476 backport: ================ This PEP doesn't take a position on whether or not this particular change is a good idea - rather, it suggests that *if* a redistributor chooses to go down the path of making the default behaviour configurable in a version of Python older than Python 2.7.9, then maintaining a consistent approach across redistributors would be beneficial for users. ================
I'd mildly prefer no default value for `enable` in _https_verify_certificates(). I'd have preferred a keyword-only argument, but of course this is Python 2. Instead, I'd like to force passing True or False (and document using `enable=True` or `enable=False`) and not rely on a default argument. But I'm only +0 on that detail.
My rationale for giving it a default is to make it marginally more straightforward to turn verification on than it is to turn it off. That's going to be most relevant in the pre-2.7.9 backport case, since in 2.7.9+ the HTTPS certificate verification will already be on by default. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (16)
-
Alex Gaynor -
Barry Warsaw -
Christian Heimes -
Cory Benfield -
Donald Stufft -
Guido van Rossum -
Laura Creighton -
M.-A. Lemburg -
Nick Coghlan -
Paul Moore -
R. David Murray -
Robert Collins -
Stephen J. Turnbull -
Toshio Kuratomi -
Victor Stinner -
Wes Turner