[Twisted-Python] How to detect dead client connections in twisted
I've been using xmlstream for quite sometime now and I noticed one such problem where my server doesn't detect clients that went to hardware shutdown without closing the socket. I also reproduced it using a simple twisted TCP server:
#!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor
class EchoProtocol(protocol.Protocol): def __init__(self): pass
def connectionMade(self): print "Client Connected Detected!"
def connectionLost(self, reason): print "Client Connection Lost!"
def dataReceived(self, data): self.transport.write(data)
factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run()
Using a different machine, telnet to the echo server and then shut down the computer (without exiting telnet). It seems that the connectionLost function doesn't get called. I even had an XMPP account appeared online on our XMPP server for almost three days even though I shut down the PC hosting my client. Should we implement a Hearbeat functionality? Thank you in advance!
Alvin Delagon
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :)
Alvin Delagon
On Mon, Jun 2, 2008 at 10:41 AM, Alvin Delagon adelagon@gmail.com wrote:
I've been using xmlstream for quite sometime now and I noticed one such problem where my server doesn't detect clients that went to hardware shutdown without closing the socket. I also reproduced it using a simple twisted TCP server:
#!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor
class EchoProtocol(protocol.Protocol): def __init__(self): pass
def connectionMade(self): print "Client Connected Detected!" def connectionLost(self, reason): print "Client Connection Lost!" def dataReceived(self, data): self.transport.write(data)factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run()
Using a different machine, telnet to the echo server and then shut down the computer (without exiting telnet). It seems that the connectionLost function doesn't get called. I even had an XMPP account appeared online on our XMPP server for almost three days even though I shut down the PC hosting my client. Should we implement a Hearbeat functionality? Thank you in advance!
Alvin Delagon
Alvin Delagon wrote:
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :)
Alvin Delagon
Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open?
Gabriel
Yes it supposed to be handled by the tcp_keepalive itself (as I assumed). The weird thing was that we had this configuration:
Ubuntu Linux Server 2.6.15-26 tcp_keepalive_time 7200 tcp_keepalive_probes 9 tcp_keepalive_intvl 75
I have an XMPP client last sent a packet to the server at May 29 15:57:25 telling that it will enter an idle state. On May 30 approx 08:00:00 I shut down my PC which hosts my XMPP client thus causing a half-opened state. The next day (17:41:11) while checking the logs, its still in half-opened state and my server still thinks that I'm online. It seems that tcp_keepalive kernel feature that's supposed to run every 2 hours didn't kick in.
--- Alvin Delagon
On Tue, Jun 3, 2008 at 2:42 PM, Gabriel Rossetti mailing_lists@evotex.ch wrote:
Alvin Delagon wrote:
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :)
Alvin Delagon
Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open?
Gabriel
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Problem solved! I digged through the twisted.internet source codes and found this facility:
def setTcpKeepAlive(self, enabled): self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, enabled)
I modifying the simple twisted server:
#!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor
class EchoProtocol(protocol.Protocol): def __init__(self): pass
def connectionMade(self): print "Client Connected Detected!" ### enable keepalive self.transport.setTcpKeepAlive(1)
def connectionLost(self, reason): print "Client Connection Lost!"
def dataReceived(self, data): self.transport.write(data)
factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run()
Now the server is aware of keepalive timeouts now. Might as well use it in my xmpp server. :)
--- Alvin Delagon
On Tue, Jun 3, 2008 at 3:21 PM, Alvin Delagon adelagon@gmail.com wrote:
Yes it supposed to be handled by the tcp_keepalive itself (as I assumed). The weird thing was that we had this configuration:
Ubuntu Linux Server 2.6.15-26 tcp_keepalive_time 7200 tcp_keepalive_probes 9 tcp_keepalive_intvl 75
I have an XMPP client last sent a packet to the server at May 29 15:57:25 telling that it will enter an idle state. On May 30 approx 08:00:00 I shut down my PC which hosts my XMPP client thus causing a half-opened state. The next day (17:41:11) while checking the logs, its still in half-opened state and my server still thinks that I'm online. It seems that tcp_keepalive kernel feature that's supposed to run every 2 hours didn't kick in.
Alvin Delagon
On Tue, Jun 3, 2008 at 2:42 PM, Gabriel Rossetti mailing_lists@evotex.ch wrote:
Alvin Delagon wrote:
I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :)
Alvin Delagon
Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open?
Gabriel
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Yes, I found that in the meantime, but I tried it and it didn't work. Sometimes when the connection is closed abruptly, the server doesn't detect the lose. I looked at the oscar protocol implementation and they have an application level keepalive, thus I though maybe it had to be done that way and the server verifies every x seconds or minutes if it hasn't received anything from a client, it removes it from it's connected list. So Does anyone know how to deal with this problem?
Gabriel
Alvin Delagon wrote:
Problem solved! I digged through the twisted.internet source codes and found this facility:
def setTcpKeepAlive(self, enabled): self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, enabled)
I modifying the simple twisted server:
#!/usr/bin/python from twisted.internet import protocol from twisted.internet import reactor
class EchoProtocol(protocol.Protocol): def __init__(self): pass
def connectionMade(self): print "Client Connected Detected!" ### enable keepalive self.transport.setTcpKeepAlive(1) def connectionLost(self, reason): print "Client Connection Lost!" def dataReceived(self, data): self.transport.write(data)factory = protocol.Factory() factory.protocol = EchoProtocol reactor.listenTCP(8000, factory) reactor.run()
Now the server is aware of keepalive timeouts now. Might as well use it in my xmpp server. :)
Alvin Delagon
On Tue, Jun 3, 2008 at 3:21 PM, Alvin Delagon <adelagon@gmail.com mailto:adelagon@gmail.com> wrote:
Yes it supposed to be handled by the tcp_keepalive itself (as I assumed). The weird thing was that we had this configuration: Ubuntu Linux Server 2.6.15-26 tcp_keepalive_time 7200 tcp_keepalive_probes 9 tcp_keepalive_intvl 75 I have an XMPP client last sent a packet to the server at May 29 15:57:25 telling that it will enter an idle state. On May 30 approx 08:00:00 I shut down my PC which hosts my XMPP client thus causing a half-opened state. The next day (17:41:11) while checking the logs, its still in half-opened state and my server still thinks that I'm online. It seems that tcp_keepalive kernel feature that's supposed to run every 2 hours didn't kick in. --- Alvin Delagon On Tue, Jun 3, 2008 at 2:42 PM, Gabriel Rossetti <mailing_lists@evotex.ch> wrote: Alvin Delagon wrote: I just recently discovered that it's a TCP half-open socket scenario that can be remedied by monitoring client heartbeats. This is not twisted related issue. Thanks anyway. :) Alvin Delagon Just a question, could setting the tcp keepalive feature on the server side (not sure if that is possible), would that make the server detect that the connection is dead/half-open? Gabriel _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- http://www.alvinatorsplayground.blogspot.com/-- http://www.alvinatorsplayground.blogspot.com/
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Tue, 2008-06-03 at 14:10 +0200, Gabriel Rossetti wrote:
Yes, I found that in the meantime, but I tried it and it didn't work. Sometimes when the connection is closed abruptly, the server doesn't detect the lose. I looked at the oscar protocol implementation and they have an application level keepalive, thus I though maybe it had to be done that way and the server verifies every x seconds or minutes if it hasn't received anything from a client, it removes it from it's connected list. So Does anyone know how to deal with this problem?
Application level pings are the only way to go if you want quick detection of timeouts, yes. You could do that with xmlstream by sending a no-op command that expects a response every few minutes.
My initial tests when enabling the TCP keepalive were successful. I've reduced my tcp_keepalive_time for 30secs instead of 2hours (7200secs) for testing. I'll update this thread if my problem goes back. :)
--- Alvin Delagon
On Tue, Jun 3, 2008 at 9:00 PM, Itamar Shtull-Trauring itamar@itamarst.org wrote:
On Tue, 2008-06-03 at 14:10 +0200, Gabriel Rossetti wrote:
Yes, I found that in the meantime, but I tried it and it didn't work. Sometimes when the connection is closed abruptly, the server doesn't detect the lose. I looked at the oscar protocol implementation and they have an application level keepalive, thus I though maybe it had to be done that way and the server verifies every x seconds or minutes if it hasn't received anything from a client, it removes it from it's connected list. So Does anyone know how to deal with this problem?
Application level pings are the only way to go if you want quick detection of timeouts, yes. You could do that with xmlstream by sending a no-op command that expects a response every few minutes.
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Alvin Delagon -
Gabriel Rossetti -
Itamar Shtull-Trauring