[Twisted-Python] Strange behaviour for epoll reactor
We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu. It has also some threads for single isolated functions (using deferToThred). First we suspected that threads don't end and goes to an infinite loop. Because application still accepts connections and executes the requested functions. Then I ran the following ps command ps h -Leo pid,pcpu,args,lwp Looking at the output, the main thread seemed to be wasting the whole cpu time. Later on I attached the process to the strace and saw the output below. I don't how does it happen or when does it happen. It is not regular nor repeatable, but this is the strace output when it gets into loop. I send to the list, because it may be useful for someone. Now we switched back to the select reactor. P.S.: We use perspective broker on the server side and qt4reactor for the client. futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 796897}, NULL) = 0 gettimeofday({1197377318, 796940}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797127}, NULL) = 0 gettimeofday({1197377318, 797169}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 15) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797388}, NULL) = 0 gettimeofday({1197377318, 797430}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797617}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797692}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 797879}, NULL) = 0 gettimeofday({1197377318, 797921}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798139}, NULL) = 0 gettimeofday({1197377318, 798181}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 14) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798368}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798442}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 13) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 gettimeofday({1197377318, 798627}, NULL) = 0 gettimeofday({1197377318, 798669}, NULL) = 0 futex(0x8ab6898, FUTEX_WAKE, 1) = 0 epoll_wait(8, {{EPOLLERR|EPOLLHUP, {u32=19, u64=593541665440923667}}}, 8, 13) = 1 futex(0x8ab6898, FUTEX_WAKE, 1) = 0
Le Tuesday 11 December 2007 14:06:45 Mustafa Sakalsiz, vous avez écrit :
We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu.
Yes this a known problem in Twisted 2.5, that has been corrected in trunk. Hopefully we'll have a release soon, until then you can backport the changes in epollreactor into your installation. -- Thomas
On Tue, 11 Dec 2007 15:06:45 +0200, Mustafa Sakalsiz
We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu. It has also some threads for single isolated functions (using deferToThred). First we suspected that threads don't end and goes to an infinite loop. Because application still accepts connections and executes the requested functions. Then I ran the following ps command
ps h -Leo pid,pcpu,args,lwp
Looking at the output, the main thread seemed to be wasting the whole cpu time. Later on I attached the process to the strace and saw the output below. I don't how does it happen or when does it happen. It is not regular nor repeatable, but this is the strace output when it gets into loop. I send to the list, because it may be useful for someone.
Now we switched back to the select reactor.
P.S.: We use perspective broker on the server side and qt4reactor for the client.
This sounds like this issue: http://twistedmatrix.com/trac/ticket/2809 This is fixed in current trunk@HEAD and the fix will be included in the next release of Twisted. Jean-Paul
To summarize Jean-Paul's link, you can just a two-liner patch in the
epollreactor.py to fix this (after line 201 if you're using the
twisted 2.50release)
if event & _epoll.HUP or event & _epoll.ERR:
why = main.CONNECTION_LOST
On Dec 11, 2007 9:57 PM, Jean-Paul Calderone
On Tue, 11 Dec 2007 15:06:45 +0200, Mustafa Sakalsiz
wrote: We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu. It has also some threads for single isolated functions (using deferToThred). First we suspected that threads don't end and goes to an infinite loop. Because application still accepts connections and executes the requested functions. Then I ran the following ps command
ps h -Leo pid,pcpu,args,lwp
Looking at the output, the main thread seemed to be wasting the whole cpu time. Later on I attached the process to the strace and saw the output below. I don't how does it happen or when does it happen. It is not regular nor repeatable, but this is the strace output when it gets into loop. I send to the list, because it may be useful for someone.
Now we switched back to the select reactor.
P.S.: We use perspective broker on the server side and qt4reactor for the client.
This sounds like this issue:
http://twistedmatrix.com/trac/ticket/2809
This is fixed in current trunk@HEAD and the fix will be included in the next release of Twisted.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Yes, I saw the ticket and the patch. Thanks for your help. It works well now.
On Dec 12, 2007 3:56 AM, Alvin Delagon
To summarize Jean-Paul's link, you can just a two-liner patch in the epollreactor.py to fix this (after line 201 if you're using the twisted 2.50 release)
if event & _epoll.HUP or event & _epoll.ERR: why = main.CONNECTION_LOST
On Dec 11, 2007 9:57 PM, Jean-Paul Calderone
wrote: On Tue, 11 Dec 2007 15:06:45 +0200, Mustafa Sakalsiz
wrote:
We have a twisted application uses the epoll reactor. Sometimes it wastes %100 cpu. It has also some threads for single isolated functions (using deferToThred). First we suspected that threads don't end and goes to an infinite loop. Because application still accepts connections and executes the requested functions. Then I ran the following ps command
ps h -Leo pid,pcpu,args,lwp
Looking at the output, the main thread seemed to be wasting the whole cpu time. Later on I attached the process to the strace and saw the output below. I don't how does it happen or when does it happen. It is not regular nor repeatable, but this is the strace output when it gets into loop. I send to the list, because it may be useful for someone.
Now we switched back to the select reactor.
P.S.: We use perspective broker on the server side and qt4reactor for the client.
This sounds like this issue:
http://twistedmatrix.com/trac/ticket/2809
This is fixed in current trunk@HEAD and the fix will be included in the next release of Twisted.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- ---------------------------------------------------------------- Likya Yazılım ve Bilişim Hizmetleri Ltd.Şti. ODTÜ Teknokent Galyum Blok B-08 06531 ANKARA Tel :: (312) 210 00 90 Fax :: (312) 210 00 91 Web :: www.ly.com.tr ---------------------------------------------------------------- mustafa@ly.com.tr
participants (4)
-
Alvin Delagon
-
Jean-Paul Calderone
-
Mustafa Sakalsiz
-
Thomas Hervé