[Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
Wes Turner
wes.turner at gmail.com
Mon Nov 23 15:47:52 EST 2015
1. Does this affect easy_install?
2. If/because this affects easy_install,
should the guidance / suggested package installation tool be [pip];
because pip install_requires backports.ssl_match_hostname
https://pypi.python.org/pypi/backports.ssl_match_hostname
On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> I have a confession to make - I dropped the ball on the HTTPS
> verification backport proposals in PEP 493, and let the upstream and
> downstream approval processes get out of sequence.
>
> As a result, the RHEL 7.2 beta released back in September incorporates
> the HTTPS verification feature backport based on the current PEP 493
> draft, even though that hasn't formally been pronounced as an Active
> recommendation by python-dev yet.
>
> Accordingly, I'm belatedly submitting it for pronouncement now:
> https://www.python.org/dev/peps/pep-0493/
>
> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
> to handle it, we'll need to address that question first.
>
> Our last discussion back in July seemed to show that folks either
> didn't care about the question (because they're using unmodified
> upstream versions so the PEP didn't affect them), or else thought the
> approach described in the PEP was reasonable, so I'm hoping the
> consequences of my mistake won't be too severe.
>
> Regards,
> Nick.
>
> P.S. I'm aware that this looks like presenting a fait accompli at a
> point where it's too late to realistically say "No", but the truth is
> that preparation for the Python in Education miniconf at PyCon
> Australia ramped up immediately after the July discussion, and then I
> personally got confused as to the scope of what was being included in
> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
> being deferred to a later release, but it's actually the whole package
> of 466+476+493). That's my fault for trying to keep track of too many
> things at once (and thus failing at some of them), not anyone else's.
>
> ================================
>
> PEP: 493
> Title: HTTPS verification recommendations for Python 2.7 redistributors
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>,
> Robert Kuska <rkuska at redhat.com>,
> Marc-André Lemburg <mal at lemburg.com>
> Status: Draft
> Type: Informational
> Content-Type: text/x-rst
> Created: 10-May-2015
> Post-History: 06-Jul-2015
>
>
> Abstract
> ========
>
> PEP 476 updated Python's default handling of HTTPS certificates to be
> appropriate for communication over the public internet. The Python 2.7 long
> term maintenance series was judged to be in scope for this change, with the
> new behaviour introduced in the Python 2.7.9 maintenance release.
>
> This PEP provides recommendations to downstream redistributors wishing to
> provide a smoother migration experience when helping their users to manage
> this change in Python's default behaviour.
>
>
> Rationale
> =========
>
> PEP 476 changed Python's default behaviour to better match the needs and
> expectations of developers operating over the public internet, a category
> which appears to include most new Python developers. It is the position of
> the authors of this PEP that this was a correct decision.
>
> However, it is also the case that this change *does* cause problems for
> infrastructure administrators operating private intranets that rely on
> self-signed certificates, or otherwise encounter problems with the new
> default
> certificate verification settings.
>
> The long term answer for such environments is to update their internal
> certificate management to at least match the standards set by the public
> internet, but in the meantime, it is desirable to offer these
> administrators
> a way to continue receiving maintenance updates to the Python 2.7 series,
> without having to gate that on upgrades to their certificate management
> infrastructure.
>
> PEP 476 did attempt to address this question, by covering how to revert the
> new settings process wide by monkeypatching the ``ssl`` module to restore
> the
> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique
> proposed
> to allow system administrators to disable the feature by default in their
> Standard Operating Environment definition has been determined to be
> insufficient in at least some cases. The specific case of interest to the
> authors of this PEP is the one where a Linux distributor aims to provide
> their users with a
> `smoother migration path
> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
> than the standard one provided by consuming upstream CPython 2.7 releases
> directly, but other potential challenges have also been pointed out with
> updating embedded Python runtimes and other user level installations of
> Python.
>
> Rather than allowing a plethora of mutually incompatibile migration
> techniques
> to bloom, this PEP proposes two alternative approaches that redistributors
> may take when addressing these problems. Redistributors may choose to
> implement
> one, both, or neither of these approaches based on their assessment of the
> needs of their particular userbase.
>
> These designs are being proposed as a recommendation for redistributors,
> rather
> than as new upstream features, as they are needed purely to support legacy
> environments migrating from older versions of Python 2.7. Neither approach
> is being proposed as an upstream Python 2.7 feature, nor as a feature in
> any
> version of Python 3 (whether published directly by the Python Software
> Foundation or by a redistributor).
>
>
> Requirements for capability detection
> =====================================
>
> As these recommendations are intended to cover backports to earlier Python
> versions, the Python version number cannot be used as a reliable means for
> detecting them. Instead, the recommendations are defined to allow the
> presence
> or absence of the feature to be determined using the following technique::
>
> python -c "import ssl; ssl._relevant_attribute"
>
> This will fail with `AttributeError` (and hence a non-zero return code) if
> the
> relevant capability is not available.
>
> The marker attributes are prefixed with an underscore to indicate the
> implementation dependent nature of these capabilities - not all Python
> distributions will offer them, only those that are providing a multi-stage
> migration process from the legacy HTTPS handling to the new default
> behaviour.
>
>
> Recommendation for an environment variable based security downgrade
> ===================================================================
>
> Some redistributors may wish to provide a per-application option to disable
> certificate verification in selected applications that run on or embed
> CPython
> without needing to modify the application itself.
>
> In these cases, a configuration mechanism is needed that provides:
>
> * an opt-out model that allows certificate verification to be selectively
> turned off for particular applications after upgrading to a version of
> Python that verifies certificates by default
> * the ability for all users to configure this setting on a per-application
> basis, rather than on a per-system, or per-Python-installation basis
>
> This approach may be used for any redistributor provided version of Python
> 2.7,
> including those that advertise themselves as providing Python 2.7.9 or
> later.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
> This not only makes it straightforward to detect the presence (or absence)
> of
> the capability, it also makes it possible to programmatically determine the
> relevant environment variable name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to providing a per-application configuration
> setting
> for HTTPS certificate verification that doesn't require modifications to
> the
> application itself is to:
>
> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
> variable when the module is first imported into a Python process
> * set the ``ssl._create_default_https_context`` function to be an alias for
> ``ssl._create_unverified_context`` if this environment variable is
> present
> and set to ``'0'``
> * otherwise, set the ``ssl._create_default_https_context`` function to be
> an
> alias for ``ssl.create_default_context`` as usual
>
>
> Example implementation
> ----------------------
>
> ::
>
> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>
> def _get_https_context_factory():
> config_setting = os.environ.get(_https_verify_envvar)
> if config_setting == '0':
> return _create_unverified_context
> return create_default_context
>
> _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> Relative to an unmodified version of CPython 2.7.9 or later, this approach
> does introduce a new downgrade attack against the default security settings
> that potentially allows a sufficiently determined attacker to revert Python
> to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
> However, such an attack requires the ability to modify the execution
> environment of a Python process prior to the import of the ``ssl`` module,
> and any attacker with such access would already be able to modify the
> behaviour of the underlying OpenSSL implementation.
>
>
> Recommendation for backporting to earlier Python versions
> =========================================================
>
> Some redistributors, most notably Linux distributions, may choose to
> backport
> the PEP 476 HTTPS verification changes to modified Python versions based on
> earlier Python 2 maintenance releases. In these cases, a configuration
> mechanism is needed that provides:
>
> * an opt-in model that allows the decision to enable HTTPS certificate
> verification to be made independently of the decision to upgrade to the
> Python version where the feature was first backported
> * the ability for system administrators to set the default behaviour of
> Python
> applications and scripts run directly in the system Python installation
> * the ability for the redistributor to consider changing the default
> behaviour
> of *new* installations at some point in the future without impacting
> existing
> installations that have been explicitly configured to skip verifying
> HTTPS
> certificates by default
>
> This approach should not be used for any Python installation that
> advertises
> itself as providing Python 2.7.9 or later, as most Python users will have
> the
> reasonable expectation that all such environments will validate HTTPS
> certificates by default.
>
>
> Required marker attribute
> -------------------------
>
> The required marker attribute on the ``ssl`` module when implementing this
> recommendation is::
>
> _cert_verification_config = '<path to configuration file>'
>
> This not only makes it straightforward to detect the presence (or absence)
> of
> the capability, it also makes it possible to programmatically determine the
> relevant configuration file name.
>
>
> Recommended modifications to the Python standard library
> --------------------------------------------------------
>
> The recommended approach to backporting the PEP 476 modifications to an
> earlier
> point release is to implement the following changes relative to the default
> PEP 476 behaviour implemented in Python 2.7.9+:
>
> * modify the ``ssl`` module to read a system wide configuration file when
> the
> module is first imported into a Python process
> * define a platform default behaviour (either verifying or not verifying
> HTTPS
> certificates) to be used if this configuration file is not present
> * support selection between the following three modes of operation:
>
> * ensure HTTPS certificate verification is enabled
> * ensure HTTPS certificate verification is disabled
> * delegate the decision to the redistributor providing this Python
> version
>
> * set the ``ssl._create_default_https_context`` function to be an alias for
> either ``ssl.create_default_context`` or
> ``ssl._create_unverified_context``
> based on the given configuration setting.
>
>
> Recommended file location
> -------------------------
>
> This approach is currently only defined for \*nix system Python
> installations.
>
> The recommended configuration file name is
> ``/etc/python/cert-verification.cfg``.
>
> The ``.cfg`` filename extension is recommended for consistency with the
> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>
>
> Recommended file format
> -----------------------
>
> The configuration file should use a ConfigParser ini-style format with a
> single section named ``[https]`` containing one required setting
> ``verify``.
>
> Permitted values for ``verify`` are:
>
> * ``enable``: ensure HTTPS certificate verification is enabled by default
> * ``disable``: ensure HTTPS certificate verification is disabled by default
> * ``platform_default``: delegate the decision to the redistributor
> providing
> this particular Python version
>
> If the ``[https]`` section or the ``verify`` setting are missing, or if the
> ``verify`` setting is set to an unknown value, it should be treated as if
> the
> configuration file is not present.
>
>
> Example implementation
> ----------------------
>
> ::
>
> _cert_verification_config = '/etc/python/cert-verification.cfg'
>
> def _get_https_context_factory():
> # Check for a system-wide override of the default behaviour
> context_factories = {
> 'enable': create_default_context,
> 'disable': _create_unverified_context,
> 'platform_default': _create_unverified_context, # For now :)
> }
> import ConfigParser
> config = ConfigParser.RawConfigParser()
> config.read(_cert_verification_config)
> try:
> verify_mode = config.get('https', 'verify')
> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
> verify_mode = 'platform_default'
> default_factory = context_factories.get('platform_default')
> return context_factories.get(verify_mode, default_factory)
>
> _create_default_https_context = _get_https_context_factory()
>
>
> Security Considerations
> -----------------------
>
> The specific recommendations for the backporting case are designed to work
> for
> privileged, security sensitive processes, even those being run in the
> following
> locked down configuration:
>
> * run from a locked down administrator controlled directory rather than a
> normal
> user directory (preventing ``sys.path[0]`` based privilege escalation
> attacks)
> * run using the ``-E`` switch (preventing ``PYTHON*`` environment variable
> based
> privilege escalation attacks)
> * run using the ``-s`` switch (preventing user site directory based
> privilege
> escalation attacks)
> * run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
> escalation attacks)
>
> The intent is that the *only* reason HTTPS verification should be getting
> turned off system wide when using this approach is because:
>
> * an end user is running a redistributor provided version of CPython rather
> than running upstream CPython directly
> * that redistributor has decided to provide a smoother migration path to
> verifying HTTPS certificates by default than that being provided by the
> upstream project
> * either the redistributor or the local infrastructure administrator has
> determined that it is appropriate to override the default upstream
> behaviour
> (at least for the time being)
>
> Using an administrator controlled configuration file rather than an
> environment
> variable has the essential feature of providing a smoother migration path,
> even
> for applications being run with the ``-E`` switch.
>
>
> Combining the recommendations
> =============================
>
> If a redistributor chooses to implement both recommendations, then the
> environment variable should take precedence over the system-wide
> configuration
> setting. This allows the setting to be changed for a given user, virtual
> environment or application, regardless of the system-wide default
> behaviour.
>
> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is
> defined, and
> set to anything *other* than ``'0'``, then HTTPS certificate verification
> should be enabled.
>
> Example implementation
> ----------------------
>
> ::
>
> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
> _cert_verification_config = '/etc/python/cert-verification.cfg'
>
> def _get_https_context_factory():
> # Check for am environmental override of the default behaviour
> config_setting = os.environ.get(_https_verify_envvar)
> if config_setting is not None:
> if config_setting == '0':
> return _create_unverified_context
> return create_default_context
>
> # Check for a system-wide override of the default behaviour
> context_factories = {
> 'enable': create_default_context,
> 'disable': _create_unverified_context,
> 'platform_default': _create_unverified_context, # For now :)
> }
> import ConfigParser
> config = ConfigParser.RawConfigParser()
> config.read(_cert_verification_config)
> try:
> verify_mode = config.get('https', 'verify')
> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
> verify_mode = 'platform_default'
> default_factory = context_factories.get('platform_default')
> return context_factories.get(verify_mode, default_factory)
>
> _create_default_https_context = _get_https_context_factory()
>
>
> Copyright
> =========
>
> This document has been placed into the public domain.
>
>
> --
> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/c7da2713/attachment-0001.html>
More information about the Python-Dev
mailing list