[Twisted-Python] logging in twisted based projects
I'm implementing some project based on twisted. I would like to use some logging mechanism in my modules and see both twisted generated log records and log records from my modules and be able to filter log records by level and source (generated by twisted, or subset of my modules). What should I do in this case? Log with python logging module in my code and send log records generated by twisted to PythonLoggingObserver? Or should I avoid using of python logging module and log only with twisted logging module? I didn't find features like filtering logs in twisted logging. Am I right that twisted based log observer uses blocking i/o? Example: observer = log.FileLogObserver(sys.stdout) # sys.stdio.write used in implementation
On Mar 14, 2013, at 3:05 AM, Sergey Gerasimov wrote:
I’m implementing some project based on twisted.
I would like to use some logging mechanism in my modules and see both twisted generated log records and log records from my modules and be able to filter log records by level and source (generated by twisted, or subset of my modules).
What should I do in this case? Log with python logging module in my code and send log records generated by twisted to PythonLoggingObserver? Or should I avoid using of python logging module and log only with twisted logging module? I didn’t find features like filtering logs in twisted logging.
Am I right that twisted based log observer uses blocking i/o? Example: observer = log.FileLogObserver(sys.stdout) # sys.stdio.write used in implementation
FileLogObserver will honor a 'system' kwarg that can be used to log the source of a message, so that: log.msg('Important message', system='MY_MODULE') yields: 2012-11-12 18:53:55-0700 [MY_MODULE] Important message Arbitrary message formatting can be accomplished by suppling a `format` kwarg in place of a message: log.msg(format='[%(level)s] %(msg)s', level='CATASTROPHIC', msg='Important Message', system='MY_MODULE') yields: 2012-11-12 18:53:55-0700 [MY_MODULE] [CATASTROPHIC] Important message That's a bit verbose, so creating a partial wrapper is useful: from functools import partial alert = partial(log.msg, format='[%(level)s] %(msg)s', level='CATASTROPHIC', system='MY_MODULE') alert(msg='Something terrible') alert(msg='Not so bad', level='INFO') # override the default level More generally, calls to log.msg() accept keyword arguments that can be inspected by any registered observer. You may want to write your own log observer and take advantage of this if you have specific filtering requirements.
It's also perfectly fine to use python's logging module from within twisted, you don't have to use twisted.log, but you should be aware this will make your log files hard to correlate (between twistd.log and your own logfiles). On Thu, Mar 14, 2013 at 10:05 AM, Sergey Gerasimov <sergun@gmail.com> wrote:
I’m implementing some project based on twisted.****
** **
I would like to use some logging mechanism in my modules and see both twisted generated log records and log records from my modules****
and be able to filter log records by level and source (generated by twisted, or subset of my modules).****
** **
What should I do in this case?****
Log with python logging module in my code and send log records generated by twisted to PythonLoggingObserver? ****
Or should I avoid using of python logging module and log only with twisted logging module? I didn’t find features like filtering logs in twisted logging.****
** **
Am I right that twisted based log observer uses blocking i/o? Example:****
observer = log.FileLogObserver(sys.stdout) # sys.stdio.write used in implementation****
** **
** **
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 01:52 pm, stephen@thorne.id.au wrote:
It's also perfectly fine to use python's logging module from within twisted, you don't have to use twisted.log, but you should be aware this will make your log files hard to correlate (between twistd.log and your own logfiles).
For posterity: within programs that *use* Twisted. Please don't use it within code being contributed to Twisted itself. :) Also, there is no such thing as "twisted.log". There is "twisted.python.log". Also, using stdlib logging doesn't necessarily force you to have two log files. It sounds like Sergey already knows how to avoid this, since he mentioned PythonLoggingObserver which bridges twisted.python.log to stdlib logging. Also, twisted.python.log doesn't prevent any filtering you might want to do, it just doesn't support it the same way the stdlib logging module does it. If you want to filter log messages, a good way to do it is probably to write a log observer that implements your filtering logic and then passes any unfiltered messages to another log observer. In this way you can apply filtering regardless of what ultimate observer you want to use. For example: def makeLevelFilter(level, observer): def filter(event): if event.get("level", 0) > level: return observer(event) return filter obs = FileLogObserver(...) addObserver(makeLevelFilter(7, obs.emit)) Jean-Paul
On Thu, Mar 14, 2013 at 10:05 AM, Sergey Gerasimov <sergun@gmail.com> wrote:
I’m implementing some project based on twisted.****
** **
I would like to use some logging mechanism in my modules and see both twisted generated log records and log records from my modules****
and be able to filter log records by level and source (generated by twisted, or subset of my modules).****
** **
What should I do in this case?****
Log with python logging module in my code and send log records generated by twisted to PythonLoggingObserver? ****
Or should I avoid using of python logging module and log only with twisted logging module? I didn’t find features like filtering logs in twisted logging.****
** **
Am I right that twisted based log observer uses blocking i/o? Example:****
observer = log.FileLogObserver(sys.stdout) # sys.stdio.write used in implementation****
** **
** **
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (4)
-
exarkun＠twistedmatrix.com -
Lucas Taylor -
Sergey Gerasimov -
Stephen Thorne