[Twisted-Python] Flush socket
For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. Currently, I am using reactor.doSelect(0) after a transport.write() to force out outstanding octets to wire. I know this should not be done like this, but is there a clean and supported way to do that? def syncSocket(self): ## FIXME: find suitable replacement for this code, which appears to break ## sometimes .. ## ## From the web: "You should never call reactor.doSelect. This isn't portable across ## reactors, and it could easily break the reactor by re-entering it where it isn't ## expecting to be re-entered." ## try: reactor.doSelect(0) return True except: return False # socket has already gone away ..
On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
well, it does not necessarily correspond, but from what I see on tcpdump with loopback device and on LAN (with TCP NoDelay set), it practically works. the problem with your suggestion: I want to test the other end .. and that is not Twisted (its i.e. Firefox .. the protocol to test is WebSockets) On 11.08.11 23:13, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
On Thu, Aug 11, 2011 at 5:18 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Maybe I'm missing something - are you actually writing functional tests for Firefox itself? -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
yes. for FF and Chrome, and other WebSockets clients On 11.08.11 23:40, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:18 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: well, it does not necessarily correspond, but from what I see on tcpdump with loopback device and on LAN (with TCP NoDelay set), it practically works. the problem with your suggestion: I want to test the other end .. and that is not Twisted (its i.e. Firefox .. the protocol to test is WebSockets) Maybe I'm missing something - are you actually writing functional tests for Firefox itself?
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()? ps: to give an example of how that was at least practically useful in what i try to do .. and not only blabbing away: http://www.tavendo.de/autobahn/testsuite/report/ for these 2 cases, the only difference are the calls to reactor.select() http://www.tavendo.de/autobahn/testsuite/report/firefox_7_0a_20110809_case_5... http://www.tavendo.de/autobahn/testsuite/report/firefox_7_0a_20110809_case_5... by which I found https://bugzilla.mozilla.org/show_bug.cgi?id=675961 On 11.08.11 23:13, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()?
You can disable nagle in Twisted, if that's the question (transport.setTcpNoDelay(True)). But you can't determine TCP packet size, that's up to the operating system.
sorry, my question wasn't clear: I do already disable Nagle by setting TCP NoDelay. And I do a reactor.select(0), which sometimes breaks I guess because of reactor reentry not expected, and it will break when the reactor is not select() based I guess. So the question is: is there an alternative to reactor.select(0) after a transport.write() to make the reactor call write() on the underlying socket for all stuff buffered within Twisted? On 11.08.11 23:47, "Itamar Turner-Trauring" <itamar@itamarst.org> wrote:
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()?
You can disable nagle in Twisted, if that's the question (transport.setTcpNoDelay(True)). But you can't determine TCP packet size, that's up to the operating system. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
You should not be calling private reactor methods, it will break things. You can make the max write size (default 128 or 64kb, I forget) bigger by setting some attribute on the transport, but again, the OS will split things up across TCP packets however you want. This is not a very useful thing to attempt - better to instrument firefox to only do certain size reads.
? I know: therefor I was asking ..
You can make the max write size (default 128 or 64kb, I forget) bigger by
I don't want to make it bigger, but smaller ..
What I am aiming at is a test suite for the WebSockets protocol which can test (by fuzzing) _any_ WebSockets client. For example, I've got a HTML file with embedded JavaScript which will connect to the test server and run all test cases to get fuzzed. That HTML works with any browser (that claims to support WebSockets). I don't want to instrument 5 browsers;) At least for IE and Opera that would not work anyway .. no source.
On 09:54 pm, tobias.oberstein@tavendo.de wrote:
Sure. Implement your own IFileDescriptor. When you add the descriptor to the reactor using IReactorFDSet.addWriter, it will call back onto your object's doWrite method to tell it there is space in the write buffer. Then you can write exactly as many bytes as you want, as well as implementing other policies - for example you can remove the descriptor from the reactor's write set using IReactorFDSet.removeWriter, thus delaying future writes until it is re- added. This will always be a somewhat unreliable way to test a remote process's handling of packetization, since there are still two TCP/IP stacks which can mess around with the data in a variety of ways, but it's as good as you can do if you want to use normal sockets for this testing. A more reliable way might be to synthesize the IP datagrams yourself, and inject them into the recipient's TCP/IP stack. Or skip the TCP/IP stack and inject them into the recipient process directly, by replacing the BSD socket APIs with an alternate implementation you control (perhaps using an LD_PRELOAD hook, for example). Twisted doesn't offer much in the way of assistance for those latter approaches, though. Jean-Paul
On Aug 11, 2011, at 7:43 PM, exarkun@twistedmatrix.com wrote:
This bears repeating. It's really unreliable. Really. The sizes of buffers passed to send() and recv() bear only a coincidental resemblance to each other; one test setup may reproduce them reliably when the next will suddenly behave completely differently. If you want even a reasonably reliable heuristic here, you need to send() and then introduce a delay. You can do this without your own IWriteDescriptor implementation though; just implement an ITransport that does its write() by breaking things up and then calling the underlying write() with callLater()s in-between. -glyph
Thanks for the tip, I've gone that route as it was the simplest to realize. Since I had centralized all calls to transport.write() in a wrapper in the code, I didn't had to implement an ITransport .. It seems to "work" (modulo the cautions you iterated) .. and I no longer use reactor internals. I had to introduce a layer of buffering though (which kicks in as soon as I do a "synch"/"chopped" send .. test prg below. Thanks again, cheers === CLIENT === import binascii, time from collections import deque from twisted.internet import reactor, protocol from twisted.internet.defer import inlineCallbacks, Deferred class TricklingClientProtocol(protocol.Protocol): def __init__(self): self.send_queue = deque() self.triggered = False def _trigger(self): if not self.triggered: self.triggered = True self._send() def _send(self): if len(self.send_queue) > 0: e = self.send_queue.popleft() self.transport.write(e) reactor.callLater(0.000001, self._send) else: self.triggered = False def send(self, data, sync = False, chopsize = None): if chopsize > 0: i = 0 n = len(data) done = False while not done: j = i + chopsize if j >= n: done = True j = n self.send_queue.append(data[i:j]) i += chopsize self._trigger() #print "chopped send" else: if sync or len(self.send_queue) > 0: self.send_queue.append(data) self._trigger() #print "synced send" else: self.transport.write(data) #print "normal send" def connectionMade(self): self.transport.setTcpNoDelay(True) self.part1() def part1(self): LEN = 50 self.send("123" * LEN) for i in xrange(0, LEN): self.send("456", sync = True) self.send("789" * LEN, chopsize = 1) self.send("123" * LEN) reactor.callLater(0.3, self.part2) def part2(self): self.send("xyz" * 5) self.send("abc" * 5, chopsize = 1) reactor.callLater(5, self.transport.loseConnection) class TricklingClientFactory(protocol.ClientFactory): protocol = TricklingClientProtocol def clientConnectionFailed(self, connector, reason): reactor.stop() def clientConnectionLost(self, connector, reason): reactor.stop() if __name__ == '__main__': factory = TricklingClientFactory() reactor.connectTCP("localhost", 9000, factory) reactor.run() === SERVER ==== import binascii from twisted.internet import reactor, protocol class TricklingServerProtocol(protocol.Protocol): def __init__(self): pass def connectionMade(self): print "client accepted" self.transport.setTcpNoDelay(True) self.stats = {} def connectionLost(self, reason): print "client lost" for s in sorted(self.stats): print "%dx chop of length %d" % (self.stats[s], s) def dataReceived(self, data): l = len(data) self.stats[l] = self.stats.get(l, 0) + 1 #print data class TricklingServerFactory(protocol.ServerFactory): protocol = TricklingServerProtocol def __init__(self): pass def startFactory(self): pass def stopFactory(self): pass if __name__ == '__main__': factory = TricklingServerFactory() reactor.listenTCP(9000, factory) reactor.run() Von: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] Im Auftrag von Glyph Lefkowitz Gesendet: Freitag, 12. August 2011 06:06 An: Twisted general discussion Betreff: Re: [Twisted-Python] Flush socket On Aug 11, 2011, at 7:43 PM, exarkun@twistedmatrix.com<mailto:exarkun@twistedmatrix.com> wrote: This will always be a somewhat unreliable way to test a remote process's handling of packetization, since there are still two TCP/IP stacks which can mess around with the data in a variety of ways, but it's as good as you can do if you want to use normal sockets for this testing. This bears repeating. It's really unreliable. Really. The sizes of buffers passed to send() and recv() bear only a coincidental resemblance to each other; one test setup may reproduce them reliably when the next will suddenly behave completely differently. If you want even a reasonably reliable heuristic here, you need to send() and then introduce a delay. You can do this without your own IWriteDescriptor implementation though; just implement an ITransport that does its write() by breaking things up and then calling the underlying write() with callLater()s in-between. -glyph
Ok, I see. Thats wicked. I don't feel ready to go that far though .. would be probably a multi-week project. Perhaps there is a network testing tool which receives TCP on one leg and trickles out octets (by varying/random amounts) on a forwarding leg to the receiver? Kind of "trickling" TCP forwarder to test stream cleanness of endpoints implementing some TCP based protocol .. is there something?
Sounds less work than above ..
Twisted doesn't offer much in the way of assistance for those latter approaches, though.
.. I see. Both would require more or less tricky stuff to be created outside Python/Twisted and take significant (for me) efforts.
On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
well, it does not necessarily correspond, but from what I see on tcpdump with loopback device and on LAN (with TCP NoDelay set), it practically works. the problem with your suggestion: I want to test the other end .. and that is not Twisted (its i.e. Firefox .. the protocol to test is WebSockets) On 11.08.11 23:13, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
On Thu, Aug 11, 2011 at 5:18 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Maybe I'm missing something - are you actually writing functional tests for Firefox itself? -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
yes. for FF and Chrome, and other WebSockets clients On 11.08.11 23:40, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:18 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: well, it does not necessarily correspond, but from what I see on tcpdump with loopback device and on LAN (with TCP NoDelay set), it practically works. the problem with your suggestion: I want to test the other end .. and that is not Twisted (its i.e. Firefox .. the protocol to test is WebSockets) Maybe I'm missing something - are you actually writing functional tests for Firefox itself?
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()? ps: to give an example of how that was at least practically useful in what i try to do .. and not only blabbing away: http://www.tavendo.de/autobahn/testsuite/report/ for these 2 cases, the only difference are the calls to reactor.select() http://www.tavendo.de/autobahn/testsuite/report/firefox_7_0a_20110809_case_5... http://www.tavendo.de/autobahn/testsuite/report/firefox_7_0a_20110809_case_5... by which I found https://bugzilla.mozilla.org/show_bug.cgi?id=675961 On 11.08.11 23:13, "Christopher Armstrong" <radix@twistedmatrix.com> wrote: On Thu, Aug 11, 2011 at 5:06 PM, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote: For testing implementations of a TCP-based protocol, I am testing if an implementation is agnostic wrt to in what chops in receives octets from the wire ("stream clean"). To do that, I force out octets from the protocol test driver (written in Twisted) in differently sized chops, down to single octets. While this is a noble goal, your method isn't really right: in TCP, the chunks you write on one end do not correspond to the chunks you receive on the other end after they've gone through a TCP stack and network. The best way to verify that your client is stream clean is to write unit tests that explicitly call its dataReceived method with different-sized chunks. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()?
You can disable nagle in Twisted, if that's the question (transport.setTcpNoDelay(True)). But you can't determine TCP packet size, that's up to the operating system.
sorry, my question wasn't clear: I do already disable Nagle by setting TCP NoDelay. And I do a reactor.select(0), which sometimes breaks I guess because of reactor reentry not expected, and it will break when the reactor is not select() based I guess. So the question is: is there an alternative to reactor.select(0) after a transport.write() to make the reactor call write() on the underlying socket for all stuff buffered within Twisted? On 11.08.11 23:47, "Itamar Turner-Trauring" <itamar@itamarst.org> wrote:
so there is no Twisted sanctioned (reactor independent and reentry safe) alternative to disable nagle (tcp nodelay) and doing a select()?
You can disable nagle in Twisted, if that's the question (transport.setTcpNoDelay(True)). But you can't determine TCP packet size, that's up to the operating system. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
You should not be calling private reactor methods, it will break things. You can make the max write size (default 128 or 64kb, I forget) bigger by setting some attribute on the transport, but again, the OS will split things up across TCP packets however you want. This is not a very useful thing to attempt - better to instrument firefox to only do certain size reads.
? I know: therefor I was asking ..
You can make the max write size (default 128 or 64kb, I forget) bigger by
I don't want to make it bigger, but smaller ..
What I am aiming at is a test suite for the WebSockets protocol which can test (by fuzzing) _any_ WebSockets client. For example, I've got a HTML file with embedded JavaScript which will connect to the test server and run all test cases to get fuzzed. That HTML works with any browser (that claims to support WebSockets). I don't want to instrument 5 browsers;) At least for IE and Opera that would not work anyway .. no source.
On 09:54 pm, tobias.oberstein@tavendo.de wrote:
Sure. Implement your own IFileDescriptor. When you add the descriptor to the reactor using IReactorFDSet.addWriter, it will call back onto your object's doWrite method to tell it there is space in the write buffer. Then you can write exactly as many bytes as you want, as well as implementing other policies - for example you can remove the descriptor from the reactor's write set using IReactorFDSet.removeWriter, thus delaying future writes until it is re- added. This will always be a somewhat unreliable way to test a remote process's handling of packetization, since there are still two TCP/IP stacks which can mess around with the data in a variety of ways, but it's as good as you can do if you want to use normal sockets for this testing. A more reliable way might be to synthesize the IP datagrams yourself, and inject them into the recipient's TCP/IP stack. Or skip the TCP/IP stack and inject them into the recipient process directly, by replacing the BSD socket APIs with an alternate implementation you control (perhaps using an LD_PRELOAD hook, for example). Twisted doesn't offer much in the way of assistance for those latter approaches, though. Jean-Paul
On Aug 11, 2011, at 7:43 PM, exarkun@twistedmatrix.com wrote:
This bears repeating. It's really unreliable. Really. The sizes of buffers passed to send() and recv() bear only a coincidental resemblance to each other; one test setup may reproduce them reliably when the next will suddenly behave completely differently. If you want even a reasonably reliable heuristic here, you need to send() and then introduce a delay. You can do this without your own IWriteDescriptor implementation though; just implement an ITransport that does its write() by breaking things up and then calling the underlying write() with callLater()s in-between. -glyph
Thanks for the tip, I've gone that route as it was the simplest to realize. Since I had centralized all calls to transport.write() in a wrapper in the code, I didn't had to implement an ITransport .. It seems to "work" (modulo the cautions you iterated) .. and I no longer use reactor internals. I had to introduce a layer of buffering though (which kicks in as soon as I do a "synch"/"chopped" send .. test prg below. Thanks again, cheers === CLIENT === import binascii, time from collections import deque from twisted.internet import reactor, protocol from twisted.internet.defer import inlineCallbacks, Deferred class TricklingClientProtocol(protocol.Protocol): def __init__(self): self.send_queue = deque() self.triggered = False def _trigger(self): if not self.triggered: self.triggered = True self._send() def _send(self): if len(self.send_queue) > 0: e = self.send_queue.popleft() self.transport.write(e) reactor.callLater(0.000001, self._send) else: self.triggered = False def send(self, data, sync = False, chopsize = None): if chopsize > 0: i = 0 n = len(data) done = False while not done: j = i + chopsize if j >= n: done = True j = n self.send_queue.append(data[i:j]) i += chopsize self._trigger() #print "chopped send" else: if sync or len(self.send_queue) > 0: self.send_queue.append(data) self._trigger() #print "synced send" else: self.transport.write(data) #print "normal send" def connectionMade(self): self.transport.setTcpNoDelay(True) self.part1() def part1(self): LEN = 50 self.send("123" * LEN) for i in xrange(0, LEN): self.send("456", sync = True) self.send("789" * LEN, chopsize = 1) self.send("123" * LEN) reactor.callLater(0.3, self.part2) def part2(self): self.send("xyz" * 5) self.send("abc" * 5, chopsize = 1) reactor.callLater(5, self.transport.loseConnection) class TricklingClientFactory(protocol.ClientFactory): protocol = TricklingClientProtocol def clientConnectionFailed(self, connector, reason): reactor.stop() def clientConnectionLost(self, connector, reason): reactor.stop() if __name__ == '__main__': factory = TricklingClientFactory() reactor.connectTCP("localhost", 9000, factory) reactor.run() === SERVER ==== import binascii from twisted.internet import reactor, protocol class TricklingServerProtocol(protocol.Protocol): def __init__(self): pass def connectionMade(self): print "client accepted" self.transport.setTcpNoDelay(True) self.stats = {} def connectionLost(self, reason): print "client lost" for s in sorted(self.stats): print "%dx chop of length %d" % (self.stats[s], s) def dataReceived(self, data): l = len(data) self.stats[l] = self.stats.get(l, 0) + 1 #print data class TricklingServerFactory(protocol.ServerFactory): protocol = TricklingServerProtocol def __init__(self): pass def startFactory(self): pass def stopFactory(self): pass if __name__ == '__main__': factory = TricklingServerFactory() reactor.listenTCP(9000, factory) reactor.run() Von: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] Im Auftrag von Glyph Lefkowitz Gesendet: Freitag, 12. August 2011 06:06 An: Twisted general discussion Betreff: Re: [Twisted-Python] Flush socket On Aug 11, 2011, at 7:43 PM, exarkun@twistedmatrix.com<mailto:exarkun@twistedmatrix.com> wrote: This will always be a somewhat unreliable way to test a remote process's handling of packetization, since there are still two TCP/IP stacks which can mess around with the data in a variety of ways, but it's as good as you can do if you want to use normal sockets for this testing. This bears repeating. It's really unreliable. Really. The sizes of buffers passed to send() and recv() bear only a coincidental resemblance to each other; one test setup may reproduce them reliably when the next will suddenly behave completely differently. If you want even a reasonably reliable heuristic here, you need to send() and then introduce a delay. You can do this without your own IWriteDescriptor implementation though; just implement an ITransport that does its write() by breaking things up and then calling the underlying write() with callLater()s in-between. -glyph
Ok, I see. Thats wicked. I don't feel ready to go that far though .. would be probably a multi-week project. Perhaps there is a network testing tool which receives TCP on one leg and trickles out octets (by varying/random amounts) on a forwarding leg to the receiver? Kind of "trickling" TCP forwarder to test stream cleanness of endpoints implementing some TCP based protocol .. is there something?
Sounds less work than above ..
Twisted doesn't offer much in the way of assistance for those latter approaches, though.
.. I see. Both would require more or less tricky stuff to be created outside Python/Twisted and take significant (for me) efforts.
participants (5)
-
Christopher Armstrong -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Itamar Turner-Trauring -
Tobias Oberstein