stdlib socket usage and "keepalive"
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Debugging a strange problem today, I got the following result:
Sockets open by stdlib libraries are open without the "keepalive" option, so the system default is used. The system default under linux is "no keepalive".
So, if you are using a URLlib connection, POP3 connection, IMAP connection, etc., any stdlib that internally creates a socket, and your server goes away suddendly (you lose network connectivity, by instance), the library will wait FOREVER for the server. The client can't detect that the server is not longer available.
The "keepalive" option will send a probe packed every X minutes of inactivity, to check if the other side is still alive, even if the connection is idle.
The issue is bad, but the solution is simple enough. Options:
1. All "client" libraries should create sockets with the "KEEPALIVE" option.
2. Modify the socket C module to create all sockets as "Keepalive" by default.
3. To have a global variable in the socket module to change the default for future sockets. Something like current "socket.setdefaulttimeout()". The default should be "keepalive".
4. Modify client libraries to accept a new optional socket-like object as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This is an issue in Linux because by default the sockets are not "keepalive". In other Unix systems, the default is "keepalive". I don't know about MS Windows.
What do you think?. The solution seems trivial, after deciding the right way to go.
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Are you sure about this? ISTM that in most cases when a server goes away unexpectedly the local host will discover this when it next tries to use the socket. Also I recall reading that keepalives are a very controversial concept (since they may actually break connections unnecessarily if the internet merely has a hiccup).
--Guido
On Mon, Apr 12, 2010 at 2:51 PM, Jesus Cea jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Debugging a strange problem today, I got the following result:
Sockets open by stdlib libraries are open without the "keepalive" option, so the system default is used. The system default under linux is "no keepalive".
So, if you are using a URLlib connection, POP3 connection, IMAP connection, etc., any stdlib that internally creates a socket, and your server goes away suddendly (you lose network connectivity, by instance), the library will wait FOREVER for the server. The client can't detect that the server is not longer available.
The "keepalive" option will send a probe packed every X minutes of inactivity, to check if the other side is still alive, even if the connection is idle.
The issue is bad, but the solution is simple enough. Options:
All "client" libraries should create sockets with the "KEEPALIVE" option.
Modify the socket C module to create all sockets as "Keepalive" by
default.
- To have a global variable in the socket module to change the default
for future sockets. Something like current "socket.setdefaulttimeout()". The default should be "keepalive".
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This is an issue in Linux because by default the sockets are not "keepalive". In other Unix systems, the default is "keepalive". I don't know about MS Windows.
What do you think?. The solution seems trivial, after deciding the right way to go.
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBS8OV2Zlgi5GaxT1NAQIJhwP8CH+gij4KfrJ1oHW5ys4PboH5Ru0pgdly Wbsza0+uj3p68P1vDnC9jIr7j+fI3ql3DOc8zUoIKGpJoaWVspbv3c4vI4ATLHo+ J6I18dpkviRT8/sT/69vMvghaGndiO0Sks/S4tDjhNstYH7oGjWxi63cKqtGPY/p WSTLpwrd4SY= =vkT4 -----END PGP SIGNATURE----- _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:09 AM, Guido van Rossum wrote:
Are you sure about this? ISTM that in most cases when a server goes away unexpectedly the local host will discover this when it next tries to use the socket. Also I recall reading that keepalives are a very controversial concept (since they may actually break connections unnecessarily if the internet merely has a hiccup).
The case is this:
1. The client does a request. Wait "read"ing the answer.
2. The request is "slow", so the server "acks" the TCP datagram and start to process the request.
3. Now the connection is idle. The server is working and the client is waiting, blocked in the socket "read()".
4. Now you switch off the server (or unplug the ethernet wire).
5. If the client kernel is not doing "keepalive", the client will be blocked FOREVER (days), waiting for the reply. If the client uses "keepalive", its kernel will send a probe after a while (30 minutes, for instance), the probe will fail, and the kernel will shutdown the connection. The problem is: linux doesn't uses KEEPALIVE by default.
I have validated this behaviour with Ubuntu 9.10 and a network sniffer.
About controversial... keepalive are usually sent only when the connection is 100% idle for a while, when "while" can be >15 minutes, so the load should be "none" for regular connections.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:09 AM, Guido van Rossum wrote:
Also I recall reading that keepalives are a very controversial concept (since they may actually break connections unnecessarily if the internet merely has a hiccup).
That is true, but parameters are usually very conservative.
In standard Ubuntu box I host out there, the (default) parameters used (if I activate keepalive):
+ Send the keepalive only after 1800 seconds of complete connection inactivity.
+ After starting sending keepalives, send NINE probes.
+ Between probes, wait at least 75 seconds.
So you have to have 30 minutes of idle and then at least 11 minutes of no conectivity to "lose" the connection.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
All "client" libraries should create sockets with the "KEEPALIVE" option.
Modify the socket C module to create all sockets as "Keepalive" by
default.
I don't know whether there are any negative implications of such solutions. Granted, the first one is less radical than the second (because servers wouldn't be impacted).
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
Regards
Antoine.
On Mon, Apr 12, 2010 at 5:34 PM, Jesus Cea jcea@jcea.es wrote:
The problem is: linux doesn't uses KEEPALIVE by default.
If you believe the problem is with the Linux kernel, perhaps you should take up your case on a more appropriate mailing list?
Python's socket module is a fairly low-level module, as it's just a thin wrapper around the corresponding operating system calls. Anyone using it has to be prepared to deal with a certain amount of exposed operating system details.
If you want to use TCP KEEPALIVE on Linux, then just call: my_socket_object.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
Most non-trivial applications use select() or poll() to avoid blocking calls and do their own timeout-checking at the application layer, so they don't need KEEPALIVE. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:59 AM, Daniel Stutzbach wrote:
Most non-trivial applications use select() or poll() to avoid blocking calls and do their own timeout-checking at the application layer, so they don't need KEEPALIVE.
I am thinking about python stdlibs like imaplib, poplib, smtplib, ftplib, urllib, etc., etc., ...
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
What bugs me is that the socket creation is deep inside the stdlibs. You can not control it easily (I have overloaded socket.socket() in the past for controlling the number of concurrent connections to servers, for instance, or providing encryption), and it is difficult to test.
If these stdlib methods could accept an optional parameter instead of creating the socket internally, test is trivial, and you can reuse the lib to access the service via an arbitrary object (this weekend I just tunneled TCP/IP via DNS requests/answers, shame on you, airport wifi hotspots!).
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On Mon, Apr 12, 2010 at 3:59 PM, Daniel Stutzbach daniel@stutzbachenterprises.com wrote:
On Mon, Apr 12, 2010 at 5:34 PM, Jesus Cea jcea@jcea.es wrote:
The problem is: linux doesn't uses KEEPALIVE by default.
If you believe the problem is with the Linux kernel, perhaps you should take up your case on a more appropriate mailing list?
Python's socket module is a fairly low-level module, as it's just a thin wrapper around the corresponding operating system calls. Anyone using it has to be prepared to deal with a certain amount of exposed operating system details.
Bingo. I expect that changing this will have too many unanticipated ramifications to be safe.
If you want to use TCP KEEPALIVE on Linux, then just call: my_socket_object.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
Most non-trivial applications use select() or poll() to avoid blocking calls and do their own timeout-checking at the application layer, so they don't need KEEPALIVE.
-- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC
On Mon, Apr 12, 2010 at 4:19 PM, Jesus Cea jcea@jcea.es wrote:
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
What bugs me is that the socket creation is deep inside the stdlibs. You can not control it easily (I have overloaded socket.socket() in the past for controlling the number of concurrent connections to servers, for instance, or providing encryption), and it is difficult to test.
If these stdlib methods could accept an optional parameter instead of creating the socket internally, test is trivial, and you can reuse the lib to access the service via an arbitrary object (this weekend I just tunneled TCP/IP via DNS requests/answers, shame on you, airport wifi hotspots!).
Yeah, this sounds like a useful change. Would be very useful for testing too.
On 12 Apr, 11:19 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like
object as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
Every once in a while I make a little bit more progress on the PEP I'm working on for this. If you want to talk more about this, you can find me in #python-dev or #twisted on freenode.
Jean-Paul
Jesus Cea wrote:
About controversial... keepalive are usually sent only when the connection is 100% idle for a while, when "while" can be >15 minutes, so the load should be "none" for regular connections.
I guess the concern would be that the keepalive probe itself is subject to uncertain delays, so how long do you wait for a reply before deciding that the connection is dead? If you don't wait long enough, you could end up killing a connection that would have come back if you had waited longer.
Jesus Cea wrote:
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
This strikes me as the best way forward (albeit far from easy to do):
1. Existing code which doesn't use the new arguments should be unaffected.
2. No changes to low level socket code, so no chance of unintended side effects on library behaviour
3. Solves a broad class of problems (i.e. allows all sorts of tunnelling and various socket options) rather than just this one problem.
4. Allows easier library testing with "mock" socket objects.
The problem actually reminds me of Spolsky's Law of Leaky Abstractions essay. If the abstraction layer is going to leak anyway (and they always do), it's best to give the developer access to all of the tools they need to plug the hole.
Cheers, Nick.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 13/04/10 04:03, exarkun@twistedmatrix.com wrote:
On 12 Apr, 11:19 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
Every once in a while I make a little bit more progress on the PEP I'm working on for this. If you want to talk more about this, you can find me in #python-dev or #twisted on freenode.
Jean-Paul
Jean-Paul, I would like to have this for 3.2. How is the PEP going?.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On 03:11 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 13/04/10 04:03, exarkun@twistedmatrix.com wrote:
On 12 Apr, 11:19 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like
object as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
Every once in a while I make a little bit more progress on the PEP I'm working on for this. If you want to talk more about this, you can find me in #python-dev or #twisted on freenode.
Jean-Paul
Jean-Paul, I would like to have this for 3.2. How is the PEP going?.
It's still little more than an outline. You can see it here:
http://twistedmatrix.com/trac/wiki/ProtocolPEP
And if you're interested in helping, we can figure out a way to do that (you can have edit permission on the wiki or we can move the document elsewhere, whatever).
Jean-Paul
On Sun, Jul 11, 2010 at 9:06 PM, exarkun@twistedmatrix.com wrote:
On 03:11 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 13/04/10 04:03, exarkun@twistedmatrix.com wrote:
On 12 Apr, 11:19 pm, jcea@jcea.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04/13/2010 12:47 AM, Antoine Pitrou wrote:
Jesus Cea <jcea <at> jcea.es> writes:
PS: "socket.setdefaulttimeout()" is not enough, because it could shutdown a perfectly functional connection, just because it was idle for too long.
The socket timeout doesn't shutdown anything. It just puts a limit on how much time recv() and send() can block. Then it's up to you to detect whether the server is still alive (for example by pinging it through whatever means the application protocol gives you).
A regular standard library (let say, poplib) would abort, after getting the timeout exception.
- Modify client libraries to accept a new optional socket-like object
as an optional parameter. This would allow things like transparent compression or encryption, or to replace the socket connection by anything else (read/write to shared memory or database, for example).
This could be useful too.
I have been thinking about this for years. Do you actually think this could be formally proposed?.
Every once in a while I make a little bit more progress on the PEP I'm working on for this. If you want to talk more about this, you can find me in #python-dev or #twisted on freenode.
Jean-Paul
Jean-Paul, I would like to have this for 3.2. How is the PEP going?.
It's still little more than an outline. You can see it here:
http://twistedmatrix.com/trac/wiki/ProtocolPEP
And if you're interested in helping, we can figure out a way to do that (you can have edit permission on the wiki or we can move the document elsewhere, whatever).
Jean-Paul
This seems like an interesting idea to me. I would like to figure out some way I could help with the PEP. If you move the document, could you please keep me updated on the new location?
:.:: mattias
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/thebrasse%40brasse.org
On 12:30 pm, thebrasse@brasse.org wrote:
On Sun, Jul 11, 2010 at 9:06 PM, exarkun@twistedmatrix.com wrote:
It's still little more than an outline. You can see it here:
http://twistedmatrix.com/trac/wiki/ProtocolPEP
And if you're interested in helping, we can figure out a way to do that (you can have edit permission on the wiki or we can move the document elsewhere, whatever).
Jean-Paul
This seems like an interesting idea to me. I would like to figure out some way I could help with the PEP. If you move the document, could you please keep me updated on the new location?
Sure. If it moves, there'll be a pointer from its existing location to the new one.
Jean-Paul
participants (8)
-
Antoine Pitrou -
Daniel Stutzbach -
exarkun＠twistedmatrix.com -
Greg Ewing -
Guido van Rossum -
Jesus Cea -
Mattias Brändström -
Nick Coghlan