[Python-ideas] PEP 3156 - Asynchronous IO Support Rebooted

[Antoine Pitrou]

On Sun, 6 Jan 2013 16:45:52 +0100
Markus <nepenthesdev at gmail.com> wrote:
> >> Transports:
> >> I think SSL should be a Protocol not a transport - implemented using BIO pairs.
> >> If you can chain protocols, like Transport / ProtocolA / ProtocolB you can have
> >> TCP / SSL / HTTP as https or  TCP / SSL / SOCKS /  HTTP as https via
> >> ssl enabled socks proxy without having to much problems. Another
> >> example, shaping a connection TCP / RATELIMIT / HTTP.
> >
> > Interesting idea. This may be up to the implementation -- not every
> > implementation may have BIO wrappers available (AFAIK the stdlib
> > doesn't),
> 
> Right, for ssl bios pyopenssl is required - or ctypes.

Or a patch to Python 3.4.
See http://docs.python.org/devguide/

By the way, how does "SSL as a protocol" deal with SNI? How does the
HTTP layer tell the SSL layer which servername to indicate?
Or, on the server-side, how would the SSL layer invoke the HTTP layer's
servername callback?

> > (I'm trying to follow Twisted's guidance here, they seem to have the
> > longest experience doing these kinds of things. When I talked to Glyph
> > IIRC he was skeptical about reconnecting in general.)
> 
> Point is - connections don't last forever, even if we want them to.
> If the transport supports "reconnect" - it is still upto the protocol
> to either support it or not.
> If a Protocol gets disconnected and wants to reconnect -without the
> Transport supporting .reconnect()- the protocol has to know it's
> factory.

+1 to this.

>  + connection_established(self, transport)
> the connection is established - in your proposal it is connection_made
> which I disagree with due to the lack of context in the Futures,
> returns None
> 
>  + timeout_dns(self)
> Resolving the domain name failed - Protocol can .reconnect() for
> another try. returns None
> 
>  + timeout_idle(self)
> connection was idle for some time - send a high layer keep alive or
> close the connection - returns None
> 
>  + timeout_connecting(self)
> connection timed out connection - Protocol can .reconnect() for
> another try, returns None

I would rather have connection_failed(self, exc).
(where exc can be a OSError or a socket.timeout)

> >>  * data_received(data) - if it was possible to return the number of
> >> bytes consumed by the protocol, and have the Transport buffer the rest
> >> for the next io in call, one would avoid having to do this in every
> >> Protocol on it's own - learned from experience.
> >
> > Twisted has a whole slew of protocol implementation subclasses that
> > implement various strategies like line-buffering (including a really
> > complex version where you can turn the line buffering on and off) and
> > "netstrings". I am trying to limit the PEP's size by not including
> > these, but I fully expect that in practice a set of useful protocol
> > implementations will be created that handles common cases. I'm not
> > convinced that putting this in the transport/protocol interface will
> > make user code less buggy: it seems easy for the user code to miscount
> > the bytes or not return a count at all in a rarely taken code branch.
> 
> Please don't drop this.
> 
> You never know how much data you'll receive, you never know how much
> data you need for a message, so the Protocol needs a buffer.
> Having this io in buffer in the Transports allows every Protocol to
> benefit, they try to read a message from the data passed to
> data_received(), if the data received is not sufficient to create a
> full message, they need to buffer it and wait for more data.

Another solution for every Protocol to benefit is to provide a bunch of
base Protocol implementations, as Twisted does: LineReceiver, etc.

Your proposed solution (returning the number of consumed bytes) implies
a lot of slicing and concatenation of immutable bytes objects inside
the Transport, which may be quite inefficient.

Regards

Antoine.