Stuck twisted-web2 worker threads after client closes socket after sending data < content length of HTTP Post
On my app server which is built upon twisted+web2 I have seen that at times some of worker threads will get blocked and won't get released ever. Eventually my app server runs out of worker threads and starts throwing message "503 Service not available" On more debugging I have found that I can easily recreate this situation by doing a incomplete HTTP Post with data less then content length specified in POST header. After this incomplete HTTP Post with lesser bytes of data if I close the client socket or kill client process or reboot client machine, in all these three cases it results in a hanged thread on my app server. This hanged threads remains on app server for ever till kill app server process, no time out of any kind. Further debugging into web2 code I found that stream.BufferedStream.readExactly doesn't come out of stream.BufferedStrem._readUntil even if I close client side socket. After putting some debug statements I found that self.stream.read() in stream.BufferedStream._readUntil doesn't return None as a signal for End Of File even after client has closed the socket connection. Since it doesn't get None it will never return with whatever data it has received so far from client. To reproduce this issue, I have written a minimal twisted-web2 server and a client that does a HTTP Post to it with incomplete data. After Post it closes the client socket and sleeps for sometime. Finally it prints the stack of all process threads, which clearly shows one blocked twisted worker thread. In the same program if you change it to do complete HTTP Post, in the thread dump there is no blocked twisted worker thread. Here is the stack traceback of blocked twisted worker thread. File "C:\Python24\lib\threading.py", line 442, in __bootstrap self.run() File "C:\Python24\lib\threading.py", line 422, in run self.__target(*self.__args, **self.__kwargs) File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/threadpool.py", line 161, in _worker File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context.py", line 59, in callWithContext File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context.py", line 37, in callWithContext File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py", line 190, in run File "C:\work\twisted-head\twisted_server3.py", line 18, in dummy_app out_content = environ['wsgi.input'].read(int(environ['CONTENT_LENGTH'])) File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py", line 74, in read File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/internet/threads.py", line 81, in blockingCallFromThread File "C:\Python24\lib\Queue.py", line 119, in get self.not_empty.wait() File "C:\Python24\lib\threading.py", line 203, in wait waiter.acquire() To run this program twisted, twisted.web2, zope and threadframe modules are required.
On my app server which is built upon twisted+web2 I have seen that at times some of worker threads will get blocked and won't get released ever. Eventually my app server runs out of worker threads and starts throwing message "503 Service not available" On more debugging I have found that I can easily recreate this situation by doing a incomplete HTTP Post with data less then content length specified in POST header. After this incomplete HTTP Post with lesser bytes of data if I close the client socket or kill client process or reboot client machine, in all these three cases it results in a hanged thread on my app server. This hanged threads remains on app server for ever till kill app server process, no time out of any kind. Further debugging into web2 code I found that stream.BufferedStream.readExactly doesn't come out of stream.BufferedStrem._readUntil even if I close client side socket. After putting some debug statements I found that self.stream.read() in stream.BufferedStream._readUntil doesn't return None as a signal for End Of File even after client has closed the socket connection. Since it doesn't get None it will never return with whatever data it has received so far from client. To reproduce this issue, I have written a minimal twisted-web2 server and a client that does a HTTP Post to it with incomplete data. After Post it closes the client socket and sleeps for sometime. Finally it prints the stack of all process threads, which clearly shows one blocked twisted worker thread. In the same program if you change it to do complete HTTP Post, in the thread dump there is no blocked twisted worker thread. Here is the stack traceback of blocked twisted worker thread. File "C:\Python24\lib\threading.py", line 442, in __bootstrap self.run() File "C:\Python24\lib\threading.py", line 422, in run self.__target(*self.__args, **self.__kwargs) File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/threadpool.py", line 161, in _worker File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context.py", line 59, in callWithContext File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/python/context.py", line 37, in callWithContext File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py", line 190, in run File "C:\work\twisted-head\twisted_server3.py", line 18, in dummy_app out_content = environ['wsgi.input'].read(int(environ['CONTENT_LENGTH'])) File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/web2/wsgi.py", line 74, in read File "/home/lotus/Desktop/pywork3.3/lm/foreign-common/src/twisted/internet/threads.py", line 81, in blockingCallFromThread File "C:\Python24\lib\Queue.py", line 119, in get self.not_empty.wait() File "C:\Python24\lib\threading.py", line 203, in wait waiter.acquire() To run this program twisted, twisted.web2, zope and threadframe modules are required. import sys, time, socket from threading import Thread import threadframe, traceback from twisted.web2 import server, channel, static, wsgi, resource from twisted.internet import reactor class TwistedWebServerThread(Thread): def __init__(self, app): Thread.__init__(self, name="Twisted") factory = channel.HTTPFactory(server.Site(wsgi.WSGIResource(app))) reactor.listenTCP(8080, factory, interface="0.0.0.0") def run(self): reactor.run(installSignalHandlers=False) def dummy_app(environ, start_response): out_content = environ['wsgi.input'].read(int(environ['CONTENT_LENGTH'])) start_response('200 OK', [('Content-type', 'text/plain')]) return [out_content] #data intentionally less then content length D = "POST /cgi-bin/minapi.py HTTP/1.0\r\n"+ \ "Host: 127.0.0.1:8080\r\n"+\ "Content-Type: text/xml\r\n"+\ "Content-Length: 152\r\n"+\ "\r\n"+\ "<?xml version='1.0'?>\n"+\ "<methodCall>\n"+\ "<methodName>getStateName</methodName>\n" +\ "<params>\n" +\ "<param>\n" +\ "<value><int>41</int></value>\n" #+\ # "</param>\n" +\ # "</params>\n" +\ # "</methodCall>\n" #D = open("data.txt3", "rb").read() def half_post(): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(("127.0.0.1", 8080)) s.send(D) s.close() def write_server_threads(): frames = threadframe.threadframe() for frame in frames: print '-' * 72 for linestr in traceback.format_stack(frame): print linestr if __name__ == "__main__": s = TwistedWebServerThread(dummy_app) s.start() time.sleep(0.2) half_post() time.sleep(5) write_server_threads()
participants (1)
-
Kamal Choudhary