[Twisted-Python] suggestions for improvement in Twisted APIs
Howdy, we investigated the networking APIs of Twisted (10.2) and published our findings in Ottawa Linux Symposium 2012: http://nw.dreamhosters.com/ols/ols2012/ols2012-komu.pdf We propose two types of improvements to the framework described in full detail in section 4.3.1 of the publication. First, we suggest security bug fixes. Second, we suggest improvements that can improve either the user experience (i.e. by improving latency in IPv6 environments) or developers (after all, frameworks are a conveniency for the developers). The suggestions for improvements are briefly summarized below. Security issues --------------- * R1.3: IPv6-mapped addresses should not be present on the wire for security reasons. * R3.2: Server-side multihoming for UDP does not work properly. The framework should use SO_BINDTODEVICE option or sendmsg()/recvmsg() interfaces in a proper way. * (R5.3): The framework does not initialize the SSL/TLS implementation automatically Suggested improvements (for better end-user or developer experience) -------------------------------------------------------------------- * (R1.1): The framework does not support symbolic host names in its APIs, i.e., does not hide the details of hostname-to-address resolution from the application * R1.2: The details of IPv6 are not abstracted away from the application * R2.1: The framework does not implement DNS look ups with getaddrinfo(). This is important for IPv6 source address selection * R2.2: The framework does not support parallel DNS look ups over IPv4 and IPv6 to optimize latency * R3.3: The framework does not support parallel connect() over IPv4 and IPv6 to minimize the latency for connection set-up * (R4.1): TCP and UDP are not easily interchangeable Please refer to section 4.3.4 in the publication for a more elaborate discussion of the improvements.
Hi, On 04/05/13 14:28, Miika Komu wrote:
Security issues --------------- * R1.3: IPv6-mapped addresses should not be present on the wire for security reasons. * R3.2: Server-side multihoming for UDP does not work properly. The framework should use SO_BINDTODEVICE option or sendmsg()/recvmsg() interfaces in a proper way. * (R5.3): The framework does not initialize the SSL/TLS implementation automatically
clarification... strictly speaking, R3.2 could be categorized as a non-security bug (it results in failed connectivity).
On 05/04/2013 07:28 AM, Miika Komu wrote:
Howdy,
we investigated the networking APIs of Twisted (10.2) and published our findings in Ottawa Linux Symposium 2012:
http://nw.dreamhosters.com/ols/ols2012/ols2012-komu.pdf Thanks for sending us your suggestions for improvement. I'm wondering about your choice of version, though: Twisted 10.2 was released in 2010. Why did you look at an old version of Twisted for a conference in 2012?
Suggested improvements (for better end-user or developer experience) -------------------------------------------------------------------- * (R1.1): The framework does not support symbolic host names in its APIs, i.e., does not hide the details of hostname-to-address resolution from the application * R1.2: The details of IPv6 are not abstracted away from the application * R2.1: The framework does not implement DNS look ups with getaddrinfo(). This is important for IPv6 source address selection * R2.2: The framework does not support parallel DNS look ups over IPv4 and IPv6 to optimize latency * R3.3: The framework does not support parallel connect() over IPv4 and IPv6 to minimize the latency for connection set-up
I believe pretty much of all of these have been fixed in Twisted over the past year, or are written and in the process of addressing review comments. Except for this: * (R4.1): TCP and UDP are not easily interchangeable Which I don't understand at all. UDP and TCP are completely different than each other, there's no way to make them interchangeable.
Hi Itamar, On 05/04/2013 05:00 PM, Itamar Turner-Trauring wrote:
On 05/04/2013 07:28 AM, Miika Komu wrote:
Howdy,
we investigated the networking APIs of Twisted (10.2) and published our findings in Ottawa Linux Symposium 2012:
http://nw.dreamhosters.com/ols/ols2012/ols2012-komu.pdf Thanks for sending us your suggestions for improvement. I'm wondering about your choice of version, though: Twisted 10.2 was released in 2010. Why did you look at an old version of Twisted for a conference in 2012?
I'll have to check this from my co-author who did the analysis.
Suggested improvements (for better end-user or developer experience) -------------------------------------------------------------------- * (R1.1): The framework does not support symbolic host names in its APIs, i.e., does not hide the details of hostname-to-address resolution from the application * R1.2: The details of IPv6 are not abstracted away from the application * R2.1: The framework does not implement DNS look ups with getaddrinfo(). This is important for IPv6 source address selection * R2.2: The framework does not support parallel DNS look ups over IPv4 and IPv6 to optimize latency * R3.3: The framework does not support parallel connect() over IPv4 and IPv6 to minimize the latency for connection set-up
I believe pretty much of all of these have been fixed in Twisted over the past year, or are written and in the process of addressing review comments.
This is good news!
Except for this:
* (R4.1): TCP and UDP are not easily interchangeable
Which I don't understand at all. UDP and TCP are completely different than each other, there's no way to make them interchangeable.
with connected sockets, (almost) the same API calls could be used and the details could be possibly abstracted away (some kind of a flag still needed to choose between TCP and UDP). A number of applications appear to support both TCP and UDP-based connectivity: 4.2.4 A Summary of the Sockets API Findings and Their Implications: Finding 2 statistics: TCP-UDP hybrids 26.3% Finding 2. Hybrid applications using both TCP and UDP occur as frequently as TCP-only applications. Application developers seem to write many application protocols to be run with both transports. While it is possible to write almost identical code for the two transports, the Sockets API favors different functions for the two. This unnecessarily complicates the application code.
Hi Miika, On May 4, 2013, at 8:54 AM, Miika Komu <mkomu@cs.hut.fi> wrote:
with connected sockets, (almost) the same API calls could be used and the details could be possibly abstracted away (some kind of a flag still needed to choose between TCP and UDP). A number of applications appear to support both TCP and UDP-based connectivity:
Supporting both TCP (stream-based) and UDP (datagram-based) communications requires explicit decisions on the part of the application, both in the design of the protocol and in the logic of its parsing. The biggest difference is that TCP applications must implement logic to deal with framing, where as UDP applications do not need to, but there are other nuances as well. See this frequently asked question for more information: <http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.dataReceivedcalledwithonlypartofthedataIcalledtransport.writewith>
While it is possible to write almost identical code for the two transports, the Sockets API favors different functions for the two. This unnecessarily complicates the application code.
It does not "unnecessarily complicate" the application code; it introduces a different code path for a semantically distinct operation (transmit a datagram vs. transmit a stream segment). Based on my experience helping people understand UDP in the past, I would hazard a guess that many of these application protocols that can run in hybrid TCP/UDP mode are simply full of bugs because their developers did UDP testing on a fast local network, where, under low load, UDP can superficially behave like TCP and did not appreciate the semantics of the low-level networking operations they were invoking. Twisted at least makes the distinction explicit, so that in order to make this mistake you need to consciously implement the same code against two transports. Thanks for looking into Twisted, though; the fact that we have already addressed most of your other criticisms in our development process indicates that most of the other issues you pointed out were spot on :). I just hope that next time you'll have a look at a more recent version! -glyph
participants (3)
-
Glyph -
Itamar Turner-Trauring -
Miika Komu