[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Nick Coghlan ncoghlan at gmail.com
Mon Jun 5 09:59:18 EDT 2017

On 5 June 2017 at 21:44, Donald Stufft <donald at stufft.io> wrote:
> Is pip allowed to use the hypothetical _ensurepip_ssl outside of ensurepip?

Yes, so something like _tls_bootstrap would probably be a better name
for the helper module (assuming we go down the "private API to
bootstrap 3rd party SSL modules" path).

The leading underscore would be akin to the one on _thread - it isn't
that the API would be undocumented or inaccessible, it's that it
wouldn't have any backwards compatibility guarantees for general use,
so you shouldn't use it without a really good reason.

> Thinking about this reminded me about the *other* reason pip avoids
> dependencies— avoiding making assertions about what versions of software can
> actually be installed. IOW if pip depends on pyOpenSSL >= X.Y, then you
> essentially can’t install any other version of pyOpenSSL and you’d be forced
> to upgrade.

Indeed, and I think that's sufficient justification for at least
adding the private TLS bootstrapping API.

> This isn’t end of the world and pyOpenSSL is generally stable enough we
> could *probably* get away with a unversioned dependency on a non Windows
> platform. On Windows we’d have to uh, I guess use a SChannel c-types
> backend? Not having our TLS library in the stdlib makes it a bit more
> difficult, but from pip’s own POV if there’s a MemoryBio that we’re allowed
> to use and then requests uses pyOpenSSL normally (we’d apply a small patch
> to pip’s copy of requests to make it use it, but that’s easy to do with our
> setup) I think that would be workable.
> I think that’s a less optimal solution than just accepting PEP 546 which I
> think is beneficial to the Python ecosystem as a whole, making it easier to
> port more code onto 3.x (and allowing unlocking 3.x’s capabilities) and will
> make it easier to maintain the ssl library on 2.7, given it will have less
> of a divergence from 3.x again.

Right, I'm just trying to make sure we clearly separate the two
arguments: pip bootstrapping (which can potentially be addressed
through a private alternate SSL/TLS API) and ecosystem advancement
(making it easier for SSL/TLS capable applications to retain Python
2.7.x compatibility at least until 2020, while still supporting the
improvements in asynchronous IO capabilities in the 3.x series).

Separating the two main options like that gives us two future scenarios:

1. Private TLS bootstrapping API

- requests & any other libraries that want this functionality would
need a dependency on PyOpenSSL for Python versions prior to 3.5
- the base SSL/TLS support in Python 2.7 remains focused on
synchronous IO, with only limited support for sans-io style protocol
library implementations
- some time in 2017 or 2018, pip starts requiring that environments
provide either an externally managed _tls_bootstrap module, *or* a
sufficiently recent PyOpenSSL
- the first CPython maintenance release to include that pip update
would also provide the required _tls_bootstrap module by default
- redistributors will be able to backport _tls_bootstrap as an
independent module (e.g. as part of their system pip packages) rather
than necessarily including it directly in their Python runtime
- this distribution model should be resilient over time, allowing
_tls_bootstrap to be rebased freely without risking breaking end user
applications that weren't expecting it (this approach would likely
even be useful in LTS 3.x environments as they age - e.g. Ubuntu 14.04
and 16.04)
- the private _tls_bootstrap API would either be a straight backport
of the 3.6 ssl module, or else a shim around the corresponding
standard library's ssl module to add the required features, whichever
was judged easiest to implement and maintain

2. Public standard library ssl module API update

- libraries needing the functionality *either* depend on PyOpenSSL
*or* set their minimum Python version to 2.7.14+
- if libraries add a Python version check to their installation
metadata, LTS distributions are out of luck, since we typically don't
rebase Python (e.g. the RHEL 7 system Python still advertises itself
as 2.7.5, even though assorted backports have been applied on top of
- even if libraries rely on API feature detection rather than explicit
version checks, redistributors still need to actually patch the
standard library - we can't just add a guaranteed-side-effect-free
module as a separate post-installation step
- while some system administrators may be willing and able to deploy
their own variant a _tls_boostrap module in advance of their
redistributors providing one, relatively few are going to be willing
to apply custom patches to Python before deploying it

So honestly, I don't think the current "ecosystem advancement"
argument in PEP 546 actually works, as the enhancement is
*significantly* easier to roll out to older 2.7.x releases if the
adoption model is "Supporting SSL/TLS with sans-io style protocol
implementations on Python versions prior to Python 3.5 requires the
use of PyOpenSSL or a suitable PEP 543 compatible library (unless
you're a package management tool, in which case you should check for a
separate platform provided Python version independent _tls_bootstrap
module before falling back to PyOpenSSL)".

As long as we actually document the _tls_bootstrap API (even if that's
just a cross-reference to the Python 3.6 ssl module documentation),
then folks wanting to share asynchronous IO code between Python 2 & 3
will have the following options:

- use wrap_socket() (as Tornado does today)
- depend on PyOpenSSL (as Twisted does today)
- use _tls_bootstrap if available, fall back to PyOpenSSL otherwise
<-- new piece added by PEP 546 due to the problems around pip
attempting to upgrade its own dependencies


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Python-Dev mailing list