[Twisted-Python] How to detect dead client connections in twisted
I've been using xmlstream for quite sometime now and I noticed one such problem where my server doesn't detect clients that went to hardware shutdown without closing the socket. I also reproduced it using a simple twisted TCP server: #!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor class EchoProtocol(protocol.Protocol): def __init__(self): pass def connectionMade(self): print "Client Connected Detected!" def connectionLost(self, reason): print "Client Connection Lost!" def dataReceived(self, data): self.transport.write(data) factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run() Using a different machine, telnet to the echo server and then shut down the computer (without exiting telnet). It seems that the connectionLost function doesn't get called. I even had an XMPP account appeared online on our XMPP server for almost three days even though I shut down the PC hosting my client. Should we implement a Hearbeat functionality? Thank you in advance! Alvin Delagon -- http://www.alvinatorsplayground.blogspot.com/
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :) Alvin Delagon On Mon, Jun 2, 2008 at 10:41 AM, Alvin Delagon <adelagon@gmail.com> wrote:
Alvin Delagon wrote:
Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open? Gabriel
Yes it supposed to be handled by the tcp_keepalive itself (as I assumed). The weird thing was that we had this configuration: Ubuntu Linux Server 2.6.15-26 tcp_keepalive_time 7200 tcp_keepalive_probes 9 tcp_keepalive_intvl 75 I have an XMPP client last sent a packet to the server at May 29 15:57:25 telling that it will enter an idle state. On May 30 approx 08:00:00 I shut down my PC which hosts my XMPP client thus causing a half-opened state. The next day (17:41:11) while checking the logs, its still in half-opened state and my server still thinks that I'm online. It seems that tcp_keepalive kernel feature that's supposed to run every 2 hours didn't kick in. --- Alvin Delagon On Tue, Jun 3, 2008 at 2:42 PM, Gabriel Rossetti <mailing_lists@evotex.ch> wrote:
Problem solved! I digged through the twisted.internet source codes and found this facility: def setTcpKeepAlive(self, enabled): self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, enabled) I modifying the simple twisted server: #!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor class EchoProtocol(protocol.Protocol): def __init__(self): pass def connectionMade(self): print "Client Connected Detected!" ### enable keepalive self.transport.setTcpKeepAlive(1) def connectionLost(self, reason): print "Client Connection Lost!" def dataReceived(self, data): self.transport.write(data) factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run() Now the server is aware of keepalive timeouts now. Might as well use it in my xmpp server. :) --- Alvin Delagon On Tue, Jun 3, 2008 at 3:21 PM, Alvin Delagon <adelagon@gmail.com> wrote:
Yes, I found that in the meantime, but I tried it and it didn't work. Sometimes when the connection is closed abruptly, the server doesn't detect the lose. I looked at the oscar protocol implementation and they have an application level keepalive, thus I though maybe it had to be done that way and the server verifies every x seconds or minutes if it hasn't received anything from a client, it removes it from it's connected list. So Does anyone know how to deal with this problem? Gabriel Alvin Delagon wrote:
On Tue, 2008-06-03 at 14:10 +0200, Gabriel Rossetti wrote:
Application level pings are the only way to go if you want quick detection of timeouts, yes. You could do that with xmlstream by sending a no-op command that expects a response every few minutes.
My initial tests when enabling the TCP keepalive were successful. I've reduced my tcp_keepalive_time for 30secs instead of 2hours (7200secs) for testing. I'll update this thread if my problem goes back. :) --- Alvin Delagon On Tue, Jun 3, 2008 at 9:00 PM, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :) Alvin Delagon On Mon, Jun 2, 2008 at 10:41 AM, Alvin Delagon <adelagon@gmail.com> wrote:
Alvin Delagon wrote:
Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open? Gabriel
Yes it supposed to be handled by the tcp_keepalive itself (as I assumed). The weird thing was that we had this configuration: Ubuntu Linux Server 2.6.15-26 tcp_keepalive_time 7200 tcp_keepalive_probes 9 tcp_keepalive_intvl 75 I have an XMPP client last sent a packet to the server at May 29 15:57:25 telling that it will enter an idle state. On May 30 approx 08:00:00 I shut down my PC which hosts my XMPP client thus causing a half-opened state. The next day (17:41:11) while checking the logs, its still in half-opened state and my server still thinks that I'm online. It seems that tcp_keepalive kernel feature that's supposed to run every 2 hours didn't kick in. --- Alvin Delagon On Tue, Jun 3, 2008 at 2:42 PM, Gabriel Rossetti <mailing_lists@evotex.ch> wrote:
Problem solved! I digged through the twisted.internet source codes and found this facility: def setTcpKeepAlive(self, enabled): self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, enabled) I modifying the simple twisted server: #!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor class EchoProtocol(protocol.Protocol): def __init__(self): pass def connectionMade(self): print "Client Connected Detected!" ### enable keepalive self.transport.setTcpKeepAlive(1) def connectionLost(self, reason): print "Client Connection Lost!" def dataReceived(self, data): self.transport.write(data) factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run() Now the server is aware of keepalive timeouts now. Might as well use it in my xmpp server. :) --- Alvin Delagon On Tue, Jun 3, 2008 at 3:21 PM, Alvin Delagon <adelagon@gmail.com> wrote:
Yes, I found that in the meantime, but I tried it and it didn't work. Sometimes when the connection is closed abruptly, the server doesn't detect the lose. I looked at the oscar protocol implementation and they have an application level keepalive, thus I though maybe it had to be done that way and the server verifies every x seconds or minutes if it hasn't received anything from a client, it removes it from it's connected list. So Does anyone know how to deal with this problem? Gabriel Alvin Delagon wrote:
On Tue, 2008-06-03 at 14:10 +0200, Gabriel Rossetti wrote:
Application level pings are the only way to go if you want quick detection of timeouts, yes. You could do that with xmlstream by sending a no-op command that expects a response every few minutes.
My initial tests when enabling the TCP keepalive were successful. I've reduced my tcp_keepalive_time for 30secs instead of 2hours (7200secs) for testing. I'll update this thread if my problem goes back. :) --- Alvin Delagon On Tue, Jun 3, 2008 at 9:00 PM, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
participants (3)
-
Alvin Delagon -
Gabriel Rossetti -
Itamar Shtull-Trauring