Hi Paul, Is is sporadic? (the CPU burst) If so, you may be seeing the log rotation code kicking...and it's probably blocking your app too until it completes. I've got a couple patches to fix the blocking issue: http://twistedmatrix.com/trac/ticket/4372 The patch which eliminates old log rotation will probably get rid of your CPU problem too. -J On Fri, Apr 9, 2010 at 2:35 AM, Paul Goins <general@vultaire.net> wrote:
There may not be a clear answer to this problem, and the answer may or may not be Twisted-specific. I'm just trying to fish for ideas here.
The server I'm working on suffers from extremely high CPU time spent in the logging functions. I'm spending between 30 and 40% of all CPU time just in twisted.python.log.msg.
Describing the environment:
We use DailyLogFiles for file rotation along with the PythonLoggingObserver to allow for logLevel support. (Of course, this gives us significant overhead from Python's complex logging architecture; maybe we should drop this...)
On a 1000 request test of this server, we generate 58,274 calls to t.p.log.msg. There's room for cleanup, but there's lots of pressure to keep many of the log calls we have for forensic purposes, so we can only cut this back so far.
Anyway, does anyone have any ideas for speedups? Any stories from experience especially? Shots in the dark are welcome. ;-)
- Paul Goins
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python