On my app server which is built upon twisted+web2 I have seen that at times
some of worker threads will get blocked and won't get released ever.
Eventually my app server runs out of worker threads and starts throwing
message "503 Service not available"
On more debugging I have found that I can easily recreate this situation by
doing a incomplete HTTP Post with data less then content length specified in
POST header. After this incomplete HTTP Post with lesser bytes of data if I
close the client socket or kill client process or reboot client machine, in
all these three cases it results in a hanged thread on my app server. This
hanged threads remains on app server for ever till kill app server process,
no time out of any kind.
Further debugging into web2 code I found that
stream.BufferedStream.readExactly doesn't come out of
stream.BufferedStrem._readUntil even if I close client side socket. After
putting some debug statements I found that self.stream.read() in
stream.BufferedStream._readUntil doesn't return None as a signal for End Of
File even after client has closed the socket connection. Since it doesn't
get None it will never return with whatever data it has received so far from
To reproduce this issue, I have written a minimal twisted-web2 server and a
client that does a HTTP Post to it with incomplete data. After Post it
closes the client socket and sleeps for sometime. Finally it prints the
stack of all process threads, which clearly shows one blocked twisted worker
thread. In the same program if you change it to do complete HTTP Post, in
the thread dump there is no blocked twisted worker thread.
Here is the stack traceback of blocked twisted worker thread.
File "C:\Python24\lib\threading.py", line 442, in __bootstrap
File "C:\Python24\lib\threading.py", line 422, in run
line 161, in _worker
line 59, in callWithContext
line 37, in callWithContext
line 190, in run
File "C:\work\twisted-head\twisted_server3.py", line 18, in dummy_app
out_content = environ['wsgi.input'].read(int(environ['CONTENT_LENGTH']))
line 74, in read
line 81, in blockingCallFromThread
File "C:\Python24\lib\Queue.py", line 119, in get
File "C:\Python24\lib\threading.py", line 203, in wait
To run this program twisted, twisted.web2, zope and threadframe modules are