[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Cory Benfield cory at lukasa.co.uk
Thu Jun 1 09:12:41 EDT 2017


> On 1 Jun 2017, at 12:20, Antoine Pitrou <solipsis at pitrou.net> wrote:
> 
> On Thu, 1 Jun 2017 12:01:41 +0100
> Cory Benfield <cory at lukasa.co.uk> wrote:
>> In principle, sure. In practice, that means most of our users don’t use those features and so we don’t get any feedback on whether they’re good solutions to the problem.
> 
> On bugs.python.org we get plenty of feedback from people using Python
> 3's features, and we have been for years.
> 
> Your concern would have been very valid in the Python 3.2 timeframe,
> but I don't think it is anymore.

Ok? I guess?

I don’t know what to do with that answer, really. I gave you some data (80%+ of requests downloads over the last month were Python 2), and you responded with “it doesn’t cause us problems”. That’s good for you, I suppose, and well done, but it doesn’t seem immediately applicable to the concern I have.

>> All of this is related. I wrote a very, very long email initially and deleted it all because it was just too long to expect any normal human being to read it, but the TL;DR here is that we also want to support async/await, and doing so requires a memory BIO object.
> 
> async/await doesn't require a memory BIO object.  For example, Tornado
> supports async/await (*) even though it doesn't use a memory BIO object
> for its SSL layer.  And asyncio started with a non-memory BIO SSL
> implementation while still using "yield from".
> 
> (*) Despite the fact that Tornado's own coroutines are yield-based
> generators.

You are right, sorry. I should not have used the word “require”. Allow me to rephrase.

MemoryBIO objects are vastly, vastly more predictable and tractable than wrapped sockets when combined with non-blocking I/O. Using wrapped sockets and select/poll/epoll/kqueue, while possible, requires extremely subtle code that is easy to get wrong, and can nonetheless still have awkward bugs in it. I would be extremely loathe to use such an implementation, but you are correct, such an implementation can exist.

>> As to Tornado, the biggest concern there is that there is no support for composing the TLS over non-TCP sockets as far as I am aware. The wrapped socket approach is not suitable for some kinds of stream-based I/O that users really should be able to use with Requests (e.g. UNIX pipes).
> 
> Hmm, why would you use TLS on UNIX pipes except as an academic
> experiment?  Tornado is far from a full-fledged networking package like
> Twisted, but its HTTP(S) support should be very sufficient
> (understandably, since it is the core use case for it).

Let me be clear that there is no intention to use either Tornado or Twisted’s HTTP/1.1 parsers or engines. With all due respect to both projects, I have concerns about both their client implementations. Tornado’s default is definitely not suitable for use in Requests, and the curl backend is but, surprise surprise, requires a C extension and oh god we’re back here again. I have similar concerns about Twisted’s default HTTP/1.1 client. Tornado’s HTTP/1.1 server is certainly sufficient, but also not of much use to Requests. Requests very much intends to use our own HTTP logic, not least because we’re sick of relying on someone else’s.

Literally what we want is to have an event loop backing us that we can integrate with async/await and that requires us to reinvent as few wheels as possible while giving an overall better end-user experience. If I were to use Tornado, because I would want to integrate PEP 543 support into Tornado I’d ultimately have to rewrite Tornado’s TLS implementation *anyway* to replace it with a PEP 543 version. If I’m doing that, I’d much rather do it with MemoryBIO than wrapped sockets, for all of the reasons above.

As a final note, because I think we’re getting into the weeds here: this is not *necessary*. None of this is *necessary*. Requests exists, and works today. We’ll get Windows TLS support regardless of anything that’s done here, because I’ll just shim it into urllib3 like we did for macOS. What I am pushing for with PEP 543 is an improvement that would benefit the whole ecosystem: all I want to do is to make it possible for me to actually use it and ship it to users in the tools I maintain.

It is reasonable and coherent for python-dev to say “well, good luck, but no backports to help you out”. The result of that is that I put PEP 543 on the backburner (because it doesn’t solve Requests/urllib3’s problems, and ultimately my day job is about resolving those issues), and probably that we shutter the async discussion for Requests until we drop Python 2 support. That’s fine, Python is your project, not mine. But I don’t see that there’s any reason for us not to ask for this backport. After all, the worst you can do is say no, and my problems remain the same.

Cory


More information about the Python-Dev mailing list