[Twisted-Python] ludicrous timeouts in protocols.http.HTTPFactory and web.server.Site
Hi, I'm accessing a twisted-based XMLRPC server through an HTTP proxy which seems to leave outgoing connections open for a very long time (I think it waits for the server to close them on its own). This means that after a handful of remote method calls through this proxy, my server runs into file descriptor limits: Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 65, in callWithLogger File "/usr/lib/python2.3/site-packages/twisted/python/log.py", line 52, in callWithContext File "/usr/lib/python2.3/site-packages/twisted/python/context.py", line 43, in callWithContext File "/usr/lib/python2.3/site-packages/twisted/internet/default.py", line 535, in _doReadOrWrite --- <exception caught here> --- File "/usr/lib/python2.3/site-packages/twisted/internet/tcp.py", line 625, in doRead File "/usr/src/build/475206-i386/install/usr/lib/python2.3/socket.py", line 167, in accept socket.error: (24, 'Too many open files') Which is easily checked by a netstat dump : $ netstat --inet --tcp -n -p | grep :8550 tcp 0 0 192.33.178.29:8550 193.49.124.107:30839 ESTABLISHED 3076/python tcp 0 0 192.33.178.29:8550 193.49.124.107:25463 ESTABLISHED 3076/python tcp 0 0 192.33.178.29:8550 193.49.124.107:37494 ESTABLISHED 3076/python tcp 0 0 192.33.178.29:8550 193.49.124.107:33910 ESTABLISHED 3076/python tcp 0 0 192.33.178.29:8550 193.49.124.107:20854 ESTABLISHED 3076/python ... ... and so on for exactly 998 lines Looking at the Twisted code, I see that very long timeouts have been defined for both protocols.http.HTTPFactory and web.server.Site (60*60*12, that is 12 hours!). If I override the "timeout" parameter when constructing the Site receiving XMLRPC connections, then the problem disappears: reactor.listenTCP(xmlrpc_port, server.Site(self, timeout=30), interface=xmlrpc_host) I think the default values in Twisted are quite bogus and should be changed to more sensible ones. 30 or 60 seconds is ok in the context of an HTTP connection. Very long timeouts on the other hand make the server very vulnerable. Regards Antoine.
On Thu, Dec 30, 2004 at 03:44:32PM +0100, Antoine Pitrou wrote: [...]
Looking at the Twisted code, I see that very long timeouts have been defined for both protocols.http.HTTPFactory and web.server.Site (60*60*12, that is 12 hours!). If I override the "timeout" parameter when constructing the Site receiving XMLRPC connections, then the problem disappears: reactor.listenTCP(xmlrpc_port, server.Site(self, timeout=30), interface=xmlrpc_host)
I think the default values in Twisted are quite bogus and should be changed to more sensible ones. 30 or 60 seconds is ok in the context of an HTTP connection. Very long timeouts on the other hand make the server very vulnerable.
One person's "reasonable" timeout is another person's "ludicrous"... this sort of thing will need tuning per-application. 12 hours is conservative, but highly unlikely to interfere with what anyone with pre-timeout Twisted expects, while solving the problem twistedmatrix.com's web site was having with the occasional connection not being closed. And having long-running connections mysteriously die due to a timeout setting you didn't even know existed can be very surprising and frustrating to debug. That said, the HTTP code is now quite careful with timeouts: it disables them while sending the response (or more accurately, after receiving a complete request, and then reenables them after it finishes sending the response), so the only time a timeout can occur is while waiting for a request to be sent -- i.e. it won't time out someone that's downloading an ISO over a dial-up link. So there probably is room to reduce it. However, for very large POST requests over a slow link (ever tried to attach a large file to your webmail over dial-up?), 30 or 60 seconds is still too slow, but I could see an argument for, say, 30 minutes being the default. Part of the issue here is backwards compatibility -- there may be deployments of old versions of Twisted out there that depends on the conservative (or no!) timeouts, and we don't want to gratuitously break them. If we do decide to be more aggressive by default, we need to make this very clear in the documentation for the next version. Regardless, if a particular application has specific needs, they can always change the setting, just like you did. It's not a major issue. -Andrew.
However, for very large POST requests over a slow link (ever tried to attach a large file to your webmail over dial-up?), 30 or 60 seconds is still too slow, but I could see an argument for, say, 30 minutes being the default.
Well, yes, actually it depends on which timeout we are really testing. The timeout I had problems with is when a request has already been answered, but the client leaves its socket open for whatever reason, without sending or waiting for anything: so it's a case of "keepalive" HTTP connection that is kept alive much too long, I think. I agree that if data is being transfered in whatever direction (whether request payload or reply payload), the connection should not be closed ;) Since this happened with a well-known proxy server (according to the Via header, it is "NetCache NetApp/5.6.1"), some other people might encounter the same problem and wonder why their low traffic HTTP server sometimes runs out of file descriptors after 30 minutes of activity. I must insist that this happened with only _one_ client sending XMLRPC requests every 2 or 3 seconds. By the way, I've written a little routine to detect HTTP proxy settings. It works under Windows and Linux, but could surely be improved. Here it is in case it interests someone: import urlparse def discover_proxy(): """ Returns a (host, port) tuple if a proxy is found in the current machine configuration, (None, None) otherwise. """ host_port = None # Un*x et al. if 'http_proxy' in os.environ: parts = urlparse.urlparse(os.environ['http_proxy']) if not parts[0] or parts[0] == 'http': host_port = parts[1] # Windows try: import _winreg as winreg except ImportError: pass else: try: # Try to grab current proxy settings from the registry regkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings') regval = winreg.QueryValueEx(regkey, 'ProxyServer') regkey.Close() regval = str(regval[0]) # Regval can be of two types: # - 'myproxy:3128' if one proxy for all protocols # - 'ftp=myftpproxy:3128;http=myhttpproxy:3128;...' if several different proxies values = regval.split(';') if len(values) > 1: for s in values: scheme, p = s.split('=') if scheme == 'http': host_port = p break else: host_port = values[0] except Exception, e: print str(e) pass # Split host and port if host_port is not None: t = host_port.split(':') host = t[0].strip() if host: try: port = int(t[1]) except: port = 80 return host, port return None, None Regards Antoine.
participants (2)
-
Andrew Bennetts -
Antoine Pitrou