[Twisted-Python] Removing cruft from logs
Hi, I am currently developing an application which will first and foremost be an xmlrpc server with an adbapi or sasync backend. The number of xmlrpc-requests is expected to be up to around a hundred per second. Unfortunately, the logging system by default logs the following for each and every request, leading to ridiculous logfiles: 2007/02/11 12:54 CET [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [11/Feb/2007:11:54:57 +0000] "POST /RPC2 HTTP/1.0" 200 122 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)" It seems that even if I define my own logservice, as per example 11-5 in Abe Fettig's book, I am unable to catch and filter this logentry in the emit function before it is written to file. So I guess there's another LogObserver used by twistd, to which I have no direct access? What would be the proper way to get rid of this cruft? Cheers, Einar S. Idsø
On Sun, 11 Feb 2007 06:10:04 -0600, Einar S. Idsø <einar@norsk-esport.no> wrote:
Hi,
I am currently developing an application which will first and foremost be an xmlrpc server with an adbapi or sasync backend. The number of xmlrpc-requests is expected to be up to around a hundred per second. Unfortunately, the logging system by default logs the following for each and every request, leading to ridiculous logfiles:
2007/02/11 12:54 CET [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [11/Feb/2007:11:54:57 +0000] "POST /RPC2 HTTP/1.0" 200 122 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"
It seems that even if I define my own logservice, as per example 11-5 in Abe Fettig's book, I am unable to catch and filter this logentry in the emit function before it is written to file. So I guess there's another LogObserver used by twistd, to which I have no direct access?
What would be the proper way to get rid of this cruft?
Cheers, Einar S. Idsø
I take it you're using twistd? In that case, yes, twistd adds a log observer when it starts. There also isn't any supported way of getting access to it. This and other issues are an unfortunate component of the present Twisted reality :) Here's a hack: from twisted.python import log log.removeObserver(log.theLogPublisher.observers[0]) Then your custom observer should be the only one left. Hope that helps, -- Eric Mangold Twisted/Win32 Co-Maintainer
Eric Mangold wrote:
I take it you're using twistd? In that case, yes, twistd adds a log observer when it starts. There also isn't any supported way of getting access to it. This and other issues are an unfortunate component of the present Twisted reality :)
Here's a hack:
from twisted.python import log log.removeObserver(log.theLogPublisher.observers[0])
Then your custom observer should be the only one left.
Hope that helps, --Eric Mangold Twisted/Win32 Co-Maintainer
Thank you for your answer! It's too bad that it isn't possible to manipulate the default logger when using twistd. I tried your hack, but even though it stops the twistd.log from spewing cruft, I am unable to remove it from my own log. If I simply return from emit(), nothing is printed, as expected. But if I don't return but instead call log.fileObserver.emit(self, logEntryDict), both the line I wish to log and the line I wish to get rid of are printed. It seems as if both lines are printed as the result of one call to emit(). How can I prevent the second from being printed? Below is a simple working example. Cheers, Einar ******* Server ******** from twisted.application import internet, service, strports from twisted.web import server, xmlrpc from twisted.internet import protocol as tiprotocol, reactor, task from twisted.python import log, logfile import sys class ErrorLog(log.FileLogObserver): def emit(self, logEntryDict): # We can take a look at the message we are asked to log # sys.stdout.write("MSG: ---->%s<----" %logEntryDict['message']) # Alternatively just return to have no lines logged # return log.FileLogObserver.emit(self, logEntryDict) class ErrorLogService(service.Service): def __init__(self, logName, logDir): self.logName = logName self.logDir = logDir self.maxLogSize = 1000000 def startService(self): self.logFile = logfile.LogFile( self.logName, self.logDir, rotateLength = self.maxLogSize) #self.logFile.rotate() self.log = ErrorLog(self.logFile) self.log.start() def stopService(self): self.log.stop() self.logFile.close() del(self.logFile) class XmlRpcServer(xmlrpc.XMLRPC): def xmlrpc_test(self, data): log.msg("Received: %s" %data) return 2*data; xmlServer = XmlRpcServer() application = service.Application('MyApp') serviceCollection = service.IServiceCollection(application) xmlSite = server.Site(xmlServer) xmlServer = strports.service("35550", xmlSite) errorLogServer = ErrorLogService('myapp.err.log', './') xmlServer.setServiceParent(application) errorLogServer.setServiceParent(application) ****** Simple test ******
import xmlrpclib s=xmlrpclib.Server('http://localhost:35550') s.test(1) 2
***** Results ******* The following two lines should be logged to myapp.err.log. I need to get rid of the second: 2007/02/11 23:12 CET [HTTPChannel,1,127.0.0.1] Received: 1 2007/02/11 23:12 CET [HTTPChannel,1,127.0.0.1] 127.0.0.1 - - [11/Feb/2007:22:12:19 +0000] "POST /RPC2 HTTP/1.0" 200 121 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)
participants (3)
-
"Einar S. Idsø" -
"Einar S. Idsø" -
Eric Mangold