<div dir="ltr"><div><div><div><div>Hi folks,<br><br>Since the last discussion back in November (just after the RHEL 7.2 release), I've rewritten PEP 493 to be a standards track PEP targeting Python 2.7.12. Barry also kindly volunteered to serve as BDFL-Delegate, so we have a clear path to pronouncement if nobody notices any new problems or concerns that didn't come up in previous discussions :)<br><br></div><div>The PEP now focuses on adding two new configuration mechanisms to the PEP 476 backport in Python 2.7:<br><br></div><div>* turning off default certificate verification through a Python API<br></div><div>* turning off default certificate verification through an environment variable<br><br></div><div>Both of these are defined in such a way that if backported to a version where default verification is still off by default, they can be used to turn it *on*.<br></div><br></div>The original file based configuration proposal to change the default behaviour of older versions is also still covered, but moved to a clearly optional section. (The gist of that section is now "If you backport this capability, aim to stay consistent with already existing backports of it")<br><br></div>Regards,<br></div>Nick.<br><div><div><div><div><div><br>==========================================<br>PEP: 493<br>Title: HTTPS verification migration tools for Python 2.7<br>Version: $Revision$<br>Last-Modified: $Date$<br>Author: Nick Coghlan <<a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a>>,<br> Robert Kuska <<a href="mailto:rkuska@redhat.com">rkuska@redhat.com</a>>,<br> Marc-André Lemburg <<a href="mailto:mal@lemburg.com">mal@lemburg.com</a>><br>BDFL-Delegate: Barry Warsaw<br>Status: Draft<br>Type: Standards Track<br>Content-Type: text/x-rst<br>Created: 10-May-2015<br>Python-Version: 2.7.12<br>Post-History: 06-Jul-2015, 11-Nov-2015, 24-Nov-2015, 24-Feb-2016<br><br><br>Abstract<br>========<br><br>PEP 476 updated Python's default handling of HTTPS certificates in client<br>modules to align with certificate handling in web browsers, by validating<br>that the certificates received belonged to the server the client was attempting<br>to contact. The Python 2.7 long term maintenance series was judged to be in<br>scope for this change, with the new behaviour introduced in the Python 2.7.9<br>maintenance release.<br><br>This has created a non-trivial barrier to adoption for affected Python 2.7<br>maintenance releases, so this PEP proposes additional Python 2.7 specific<br>features that allow system administrators and other users to more easily<br>decouple the decision to verify server certificates in HTTPS client modules<br>from the decision to update to newer Python 2.7 maintenance releases.<br><br><br>Rationale<br>=========<br><br>PEP 476 changed Python's default behaviour to align with expectations<br>established by web browsers in regards to the semantics of HTTPS URLs:<br>starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard library<br>validate server certificates by default.<br><br>However, it is also the case that this change *does* cause problems for<br>infrastructure administrators operating private intranets that rely on<br>self-signed certificates, or otherwise encounter problems with the new default<br>certificate verification settings.<br><br>To manage these kinds of situations, web browsers provide users with "click<br>through" warnings that allow the user to add the server's certificate to the<br>browser's certificate store. Network client tools like ``curl`` and ``wget``<br>offer options to switch off certificate checking entirely (by way of<br>``curl --insecure`` and ``wget --no-check-certificate``, respectively).<br><br>At a different layer of the technology stack, Linux security modules like<br>`SELinux` and `AppArmor`, while enabled by default by distribution vendors,<br>offer relatively straightforward mechanisms for turning them off.<br><br>At the moment, no such convenient mechanisms exist to disable Python's<br>default certificate checking for a whole process.<br><br>PEP 476 did attempt to address this question, by covering how to revert to the<br>old settings process wide by monkeypatching the ``ssl`` module to restore the<br>old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed<br>to allow system administrators to disable the feature by default in their<br>Standard Operating Environment definition has been determined to be<br>insufficient in at least some cases. The specific case that led to the<br>initial creation of this PEP is the one where a Linux distributor aims to<br>provide their users with a<br>`smoother migration path <<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1173041">https://bugzilla.redhat.com/show_bug.cgi?id=1173041</a>>`__<br>than the standard one provided by consuming upstream CPython 2.7 releases<br>directly, but other potential challenges have also been pointed out with<br>updating embedded Python runtimes and other user level installations of Python.<br><br>Rather than allowing a plethora of mutually incompatibile migration techniques<br>to bloom, this PEP proposes an additional feature to be added to Python 2.7.12<br>to make it easier to revert a process to the past behaviour of skipping<br>certificate validation in HTTPS client modules. It also provides additional<br>recommendations to redistributors backporting these features to versions of<br>Python prior to Python 2.7.9.<br><br>Alternatives<br>------------<br><br>In the absence of clear upstream guidance and recommendations, commercial<br>redistributors will still make their own design decisions in the interests of<br>their customers. The main approaches available are:<br><br>* Continuing to rebase on new Python 2.7.x releases, while providing no<br> additional assistance beyond the mechanisms defined in PEP 476 in migrating<br> from unchecked to checked hostnames in standard library HTTPS clients<br>* Gating availability of the changes in default handling of HTTPS connections<br> on upgrading from Python 2 to Python 3<br>* For Linux distribution vendors, gating availability of the changes in default<br> handling of HTTPS connections on upgrading to a new operating system version<br>* Implementing one or both of the backport suggestions described in this PEP,<br> regardless of the formal status of the PEP<br><br><br>Scope Limitations<br>=================<br><br>These changes are being proposed purely as tools for helping to manage the<br>transition to the new default certificate handling behaviour in the context<br>of Python 2.7. They are not being proposed as new features for Python 3, as<br>it is expected that the vast majority of client applications affected by this<br>problem without the ability to update the application itself will be Python 2<br>applications.<br><br>It would likely be desirable for a future version of Python 3 to allow the<br>default certificate handling for secure protocols to be configurable on a<br>per-protocol basis, but that question is beyond the scope of this PEP.<br><br><br>Requirements for capability detection<br>=====================================<br><br>As the proposals in this PEP aim to facilitate backports to earlier Python<br>versions, the Python version number cannot be used as a reliable means for<br>detecting them. Instead, they are designed to allow the presence<br>or absence of the feature to be determined using the following technique::<br><br> python -c "import ssl; ssl.<_relevant_attribute>"<br><br>This will fail with `AttributeError` (and hence a non-zero return code) if the<br>relevant capability is not available.<br><br>The feature detection attributes defined by this PEP are:<br><br>* ``ssl._https_verify_certificates``: runtime configuration API<br>* ``ssl._https_verify_envvar``: environment based configuration<br>* ``ssl._cert_verification_config``: file based configuration (PEP 476 opt-in)<br><br>The marker attributes are prefixed with an underscore to indicate the<br>implementation dependent and security sensitive nature of these capabilities.<br><br><br>Feature: Configuration API<br>==========================<br><br>This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x<br>releases. It consists of a new ``ssl._https_verify_certificates()`` to specify<br>the default handling of HTTPS certificates in standard library client libraries.<br><br>It is not proposed to forward port this change to Python 3, so Python 3<br>applications that need to support skipping certificate verification will still<br>need to define their own suitable security context.<br><br>Feature detection<br>-----------------<br><br>The marker attribute on the ``ssl`` module related to this feature is the<br>``ssl._https_verify_certificates`` function itself.<br><br>Specification<br>-------------<br><br>The ``ssl._https_verify_certificates`` function will work as follows::<br><br> def _https_verify_certificates(enable=True):<br> """Verify server HTTPS certificates by default?"""<br> global _create_default_https_context<br> if enable:<br> _create_default_https_context = create_default_context<br> else:<br> _create_default_https_context = _create_unverified_context<br><br>If called without arguments, or with ``enable`` set to a true value, then<br>standard library client modules will subsequently verify HTTPS certificates by default, otherwise they will skip verification.<br><br>If called with ``enable`` set to a false value, then standard library client<br>modules will subsequently skip verifying HTTPS certificates by default.<br><br>Security Considerations<br>-----------------------<br><br>The inclusion of this feature will allow security sensitive applications to<br>include the following forward-compatible snippet in their code::<br><br> if hasattr(ssl, "_https_verify_certificates"):<br> ssl._https_verify_certificates()<br><br>Some developers may also choose to opt out of certificate checking using<br>``ssl._https_verify_certificates(enable=False)``. This doesn't introduce any<br>major new security concerns, as monkeypatching the affected internal APIs was<br>already possible.<br><br><br>Feature: environment based configuration<br>========================================<br><br>This change is proposed for inclusion in CPython 2.7.12 and later CPython 2.7.x<br>releases. It consists of a new ``PYTHONHTTPSVERIFY`` environment variable that<br>can be set to ``'0'`` to disable the default verification without modifying the<br>application source code (which may not even be available in cases of<br>bytecode-only application distribution)<br><br>It is not proposed to forward port this change to Python 3, so Python 3<br>applications that need to support skipping certificate verification will still<br>need to define their own suitable security context.<br><br>Feature detection<br>-----------------<br><br>The marker attribute on the ``ssl`` module related to this feature is:<br><br>* the ``ssl._https_verify_envvar`` attribute, giving the name of environment<br> variable affecting the default behaviour<br><br>This not only makes it straightforward to detect the presence (or absence) of<br>the capability, it also makes it possible to programmatically determine the<br>relevant environment variable name.<br><br>Specification<br>-------------<br><br>Rather than always defaulting to the use of ``ssl.create_default_context``,<br>the ``ssl`` module will be modified to:<br><br>* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first<br> imported into a Python process<br>* set the ``ssl._create_default_https_context`` function to be an alias for<br> ``ssl._create_unverified_context`` if this environment variable is present<br> and set to ``'0'``<br>* otherwise, set the ``ssl._create_default_https_context`` function to be an<br> alias for ``ssl.create_default_context`` as usual<br><br>Example implementation<br>----------------------<br><br>::<br><br> _https_verify_envvar = 'PYTHONHTTPSVERIFY'<br><br> def _get_https_context_factory():<br> if not sys.flags.ignore_environment:<br> config_setting = os.environ.get(_https_verify_envvar)<br> if config_setting == '0':<br> return _create_unverified_context<br> return create_default_context<br><br> _create_default_https_context = _get_https_context_factory()<br><br>Security Considerations<br>-----------------------<br><br>Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11, this<br>approach does introduce a new downgrade attack against the default security<br>settings that potentially allows a sufficiently determined attacker to revert<br>Python to the default behaviour used in CPython 2.7.8 and earlier releases.<br>However, such an attack requires the ability to modify the execution<br>environment of a Python process prior to the import of the ``ssl`` module,<br>and any attacker with such access would already be able to modify the<br>behaviour of the underlying OpenSSL implementation.<br><br>Interaction with Python virtual environments<br>--------------------------------------------<br><br>The default setting is read directly from the process environment, and hence<br>works the same way regardless of whether or not the interpreter is being run<br>inside an activated Python virtual environment.<br><br><br>Reference Implementation<br>========================<br><br>A patch for Python 2.7 implementing the above two features is attached to<br>the `relevant tracker issue <<a href="http://bugs.python.org/issue23857">http://bugs.python.org/issue23857</a>>`__.<br><br><br>Backporting this PEP to earlier Python versions<br>===============================================<br><br>If this PEP is accepted, then commercial Python redistributors may choose to<br>backport the per-process configuration mechanisms defined in this PEP to base<br>versions older than Python 2.7.9, *without* also backporting PEP 476's change<br>to the default behaviour of the overall Python installation.<br><br>Such a backport would differ from the mechanism proposed in this PEP solely in<br>the default behaviour when ``PYTHONHTTPSVERIFY`` was not set at all: it would<br>continue to default to skipping certificate validation.<br><br>In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and<br>set to anything *other* than ``'0'``, then HTTPS certificate verification<br>should be enabled.<br><br>Feature detection<br>-----------------<br><br>There's no specific attribute indicating that this situation applies. Rather,<br>it is indicated by the ``ssl._https_verify_certificates`` and<br>``ssl._https_verify_envvar`` attributes being present in a Python version that<br>is nominally older than Python 2.7.12.<br><br>Specification<br>-------------<br><br>Implementing this backport involves backporting the changes in PEP 466, 476 and<br>this PEP, with the following change to the handling of the<br>``PYTHONHTTPSVERIFY`` environment variable in the ``ssl`` module:<br><br>* read the ``PYTHONHTTPSVERIFY`` environment variable when the module is first<br> imported into a Python process<br>* set the ``ssl._create_default_https_context`` function to be an alias for<br> ``ssl.create_default_context`` if this environment variable is present<br> and set to any value other than ``'0'``<br>* otherwise, set the ``ssl._create_default_https_context`` function to be an<br> alias for ``ssl._create_unverified_context``<br><br>Example implementation<br>----------------------<br><br>::<br><br> _https_verify_envvar = 'PYTHONHTTPSVERIFY'<br><br> def _get_https_context_factory():<br> if not sys.flags.ignore_environment:<br> config_setting = os.environ.get(_https_verify_envvar)<br> if config_setting != '0':<br> return create_default_context<br> return _create_unverified_context<br><br> _create_default_https_context = _get_https_context_factory()<br><br> def _disable_https_default_verification():<br> """Skip verification of HTTPS certificates by default"""<br> global _create_default_https_context<br> _create_default_https_context = _create_unverified_context<br><br>Security Considerations<br>-----------------------<br><br>This change would be a strict security upgrade for any Python version that<br>currently defaults to skipping certificate validation in standard library<br>HTTPS clients. The technical trade-offs to be taken into account relate largely<br>to the magnitude of the PEP 466 backport also required rather than to anything<br>security related.<br><br>Interaction with Python virtual environments<br>--------------------------------------------<br><br>The default setting is read directly from the process environment, and hence<br>works the same way regardless of whether or not the interpreter is being run<br>inside an activated Python virtual environment.<br><br><br>Backporting PEP 476 to earlier Python versions<br>==============================================<br><br>The backporting approach described above leaves the default HTTPS certificate<br>verification behaviour of a Python 2.7 installation unmodified: verifying<br>certificates still needs to be opted into on a per-connection or per-process<br>basis.<br><br>To allow the default behaviour of the entire installation to be modified<br>without breaking backwards compatibility, Red Hat designed a configuration<br>mechanism for the system Python 2.7 installation in Red Hat Enterprise Linux<br>7.2+ that provides:<br><br>* an opt-in model that allows the decision to enable HTTPS certificate<br> verification to be made independently of the decision to upgrade to the<br> operating system version where the feature was first backported<br>* the ability for system administrators to set the default behaviour of Python<br> applications and scripts run directly in the system Python installation<br>* the ability for the redistributor to consider changing the default behaviour<br> of *new* installations at some point in the future without impacting existing<br> installations that have been explicitly configured to skip verifying HTTPS<br> certificates by default<br><br>As it only affects backports to earlier releases of Python 2.7, this change is<br>not proposed for inclusion in upstream CPython, but rather is offered as<br>a recommendation to other redistributors that choose to offer a similar feature<br>to their users.<br><br>This PEP doesn't take a position on whether or not this particular change is a<br>good idea - rather, it suggests that *if* a redistributor chooses to go down<br>the path of making the default behaviour configurable in a version of Python<br>older than Python 2.7.9, then maintaining a consistent approach across<br>redistributors would be beneficial for users.<br><br>However, this approach SHOULD NOT be used for any Python installation that<br>advertises itself as providing Python 2.7.9 or later, as most Python users<br>will have the reasonable expectation that all such environments will verify<br>HTTPS certificates by default.<br><br><br>Feature detection<br>-----------------<br><br>The marker attribute on the ``ssl`` module related to this feature is::<br><br> _cert_verification_config = '<path to configuration file>'<br><br>This not only makes it straightforward to detect the presence (or absence) of<br>the capability, it also makes it possible to programmatically determine the<br>relevant configuration file name.<br><br><br>Recommended modifications to the Python standard library<br>--------------------------------------------------------<br><br>The recommended approach to backporting the PEP 476 modifications to an earlier<br>point release is to implement the following changes relative to the default<br>PEP 476 behaviour implemented in Python 2.7.9+:<br><br>* modify the ``ssl`` module to read a system wide configuration file when the<br> module is first imported into a Python process<br>* define a platform default behaviour (either verifying or not verifying HTTPS<br> certificates) to be used if this configuration file is not present<br>* support selection between the following three modes of operation:<br><br> * ensure HTTPS certificate verification is enabled<br> * ensure HTTPS certificate verification is disabled<br> * delegate the decision to the redistributor providing this Python version<br><br>* set the ``ssl._create_default_https_context`` function to be an alias for<br> either ``ssl.create_default_context`` or ``ssl._create_unverified_context``<br> based on the given configuration setting.<br><br><br>Recommended file location<br>-------------------------<br><br>As the PEP authors are not aware of any vendors providing long-term support<br>releases targeting Windows, Mac OS X or \*BSD systems, this approach is<br>currently only specifically defined for Linux system Python installations.<br><br>The recommended configuration file name on Linux systems is<br>``/etc/python/cert-verification.cfg``.<br><br>The ``.cfg`` filename extension is recommended for consistency with the<br>``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.<br><br><br>Recommended file format<br>-----------------------<br><br>The configuration file should use a ConfigParser ini-style format with a<br>single section named ``[https]`` containing one required setting ``verify``.<br><br>The suggested section name is taken from the "https" URL schema passed to<br>affected client APIs.<br><br>Permitted values for ``verify`` are:<br><br>* ``enable``: ensure HTTPS certificate verification is enabled by default<br>* ``disable``: ensure HTTPS certificate verification is disabled by default<br>* ``platform_default``: delegate the decision to the redistributor providing<br> this particular Python version<br><br>If the ``[https]`` section or the ``verify`` setting are missing, or if the<br>``verify`` setting is set to an unknown value, it should be treated as if the<br>configuration file is not present.<br><br><br>Example implementation<br>----------------------<br><br>::<br><br> _cert_verification_config = '/etc/python/cert-verification.cfg'<br><br> def _get_https_context_factory():<br> # Check for a system-wide override of the default behaviour<br> context_factories = {<br> 'enable': create_default_context,<br> 'disable': _create_unverified_context,<br> 'platform_default': _create_unverified_context, # For now :)<br> }<br> import ConfigParser<br> config = ConfigParser.RawConfigParser()<br> config.read(_cert_verification_config)<br> try:<br> verify_mode = config.get('https', 'verify')<br> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):<br> verify_mode = 'platform_default'<br> default_factory = context_factories.get('platform_default')<br> return context_factories.get(verify_mode, default_factory)<br><br> _create_default_https_context = _get_https_context_factory()<br><br><br>Security Considerations<br>-----------------------<br><br>The specific recommendations for this backporting case are designed to work for<br>privileged, security sensitive processes, even those being run in the following<br>locked down configuration:<br><br>* run from a locked down administrator controlled directory rather than a normal<br> user directory (preventing ``sys.path[0]`` based privilege escalation attacks)<br>* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based<br> privilege escalation attacks)<br>* run using the ``-s`` switch (preventing user site directory based privilege<br> escalation attacks)<br>* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege<br> escalation attacks)<br><br>The intent is that the *only* reason HTTPS verification should be getting<br>turned off installation wide when using this approach is because:<br><br>* an end user is running a redistributor provided version of CPython rather<br> than running upstream CPython directly<br>* that redistributor has decided to provide a smoother migration path to<br> verifying HTTPS certificates by default than that being provided by the<br> upstream project<br>* either the redistributor or the local infrastructure administrator has<br> determined that it is appropriate to retain the default pre-2.7.9 behaviour<br> (at least for the time being)<br><br>Using an administrator controlled configuration file rather than an environment<br>variable has the essential feature of providing a smoother migration path, even<br>for applications being run with the ``-E`` switch.<br><br>Interaction with Python virtual environments<br>--------------------------------------------<br><br>This setting is scoped by the interpreter installation and affects all Python<br>processes using that interpreter, regardless of whether or not the interpreter<br>is being run inside an activated Python virtual environment.<br><br>Origins of this recommendation<br>------------------------------<br><br>This recommendation is based on the backporting approach adopted for Red Hat<br>Enterprise Linux 7.2, as published in the original July 2015 draft of this PEP<br>and described in detail in `this KnowledgeBase article<br><<a href="https://access.redhat.com/articles/2039753">https://access.redhat.com/articles/2039753</a>>`__. Red Hat's patches implementing<br>this backport for Python 2.7.5 can be found in the `CentOS git repository<br><<a href="https://git.centos.org/commit/rpms!python.git/refs!heads!c7">https://git.centos.org/commit/rpms!python.git/refs!heads!c7</a>>`__.<br><br><br>Recommendation for combined feature backports<br>=============================================<br><br>If a redistributor chooses to backport the environment variable based<br>configuration setting from this PEP to a modified Python version that also<br>implements the configuration file based PEP 476 backport, then the environment<br>variable should take precedence over the system-wide configuration setting.<br>This allows the setting to be changed for a given user or application,<br>regardless of the installation-wide default behaviour.<br><br>Example implementation<br>----------------------<br><br>::<br><br> _https_verify_envvar = 'PYTHONHTTPSVERIFY'<br> _cert_verification_config = '/etc/python/cert-verification.cfg'<br><br> def _get_https_context_factory():<br> # Check for an environmental override of the default behaviour<br> if not sys.flags.ignore_environment:<br> config_setting = os.environ.get(_https_verify_envvar)<br> if config_setting is not None:<br> if config_setting == '0':<br> return _create_unverified_context<br> return create_default_context<br><br> # Check for a system-wide override of the default behaviour<br> context_factories = {<br> 'enable': create_default_context,<br> 'disable': _create_unverified_context,<br> 'platform_default': _create_unverified_context, # For now :)<br> }<br> import ConfigParser<br> config = ConfigParser.RawConfigParser()<br> config.read(_cert_verification_config)<br> try:<br> verify_mode = config.get('https', 'verify')<br> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):<br> verify_mode = 'platform_default'<br> default_factory = context_factories.get('platform_default')<br> return context_factories.get(verify_mode, default_factory)<br><br> _create_default_https_context = _get_https_context_factory()<br><br><br>Copyright<br>=========<br><br>This document has been placed into the public domain.<br><br>-- <br><div class="gmail_signature">Nick Coghlan | <a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a> | Brisbane, Australia</div>
</div></div></div></div></div></div>