[Chicago] Taking Some Pain Out Of Python Logging

[Kumar McMillan]

On Thu, Mar 7, 2013 at 9:58 AM, Philip Doctor <diomedestydeus at gmail.com>wrote:

> Maybe a dumb question on my part, but can someone sort of clarify what the
> exact problem with logging is?  The author states people hate it (then
> cites a tweet) and claims it "reeks of NIH" without much justification.  I
> guess I don't find the standard logging to be that difficult?  Is there
> something that's specifically hard to do with it?
>

The model is flawed: You write logs to a stream. The stream is meaningless
unless it has a registered handler. Where do you register handlers to
guarantee they are registered only once and not overwritten? That's the
flaw. Also, the defaults suck and the API is too complicated.

I have more rage for logging than any other stdlib module combined then
multiplied by all modules. It usually boils down to stream registration
problems. You could be staring at the registration code you *think* is
running but some code somewhere else is overriding it. There is no easy way
to debug such problems. Also, logging wants to swallow all exceptions and
pretend they never happened. Exceptions like Unicode errors that only
happen when you have Unicode bugs thus defeating the purpose of logging
info that you can use to debug with later. Naturally, these problems
typically only appear in production and you only discover them when you are
looking for a log message that never got logged :(



>
> As justification for a switch to syslog he states I'll gain all kinds of
> good things:
>
> * Consistent time stamps all over the server
> correct me if I'm wrong please, but asctime is pulling from time.time()
> which is getting seconds since epoch from the OS right?  Shouldn't I
> already have consistent time stamps all over the sever?  I haven't
> personally noticed any inconsistencies, but I'm admittedly usually only
> looking at my application logs.
>
> * Filtering into files by application
> This confuses me also, when I configure a logger in an application, I can
> also tell it to us a specific file for that application.  I'm not seeing a
> win here.
>
> *Log rotation
> He might have a point there regarding compression of old logs etc
>
> * Remote logging
> I guess I use splunk for this so I do see a big win for me, but I could
> see it as a win for some scenarios.
>
> Any replies and horror stories from the world of python logging are
> welcome.  I'd love to know what problems might be waiting for me up the
> road as I'm currently using the default logging.
>
>
>
>
>
>
>
> On Thu, Mar 7, 2013 at 9:13 AM, Peter Fein <pete at wearpants.org> wrote:
>
>> On Thu, Mar 7, 2013 at 9:01 AM, Skip Montanaro <skip at pobox.com> wrote:
>> > At work we've used the stdlib logger in the platform I work on for
>> > quite awhile.  We log heavily, and were always disappointed in logging
>> > performance.  It was mostly caused by all the features baked into the
>> > stdlib logger which we never used (I'm sure inherited from log4j or
>> > whatever its logical parent was).  Over time we tried various things,
>> > including inlining a stripped down __init__, threading the loggers,
>> > rewriting things in C++, yadda, yadda, yadda.  A few months ago I
>> > broke down and wrote a very simple class which implements the basic
>> > logging API, can only log in one format, and only uses those features
>> > we actually need.  The platform is very mature, so I'm not worried
>>
>> Code please? :-)
>>
>> > (*) When you think about it, running a logger in its own thread is
>> > probably not that useful.  The OS probably does a much better job of
>> > scheduling writes to disk, and to switch threads you have to go
>> > through the kernel anyway. The threaded logger has more trips through
>> > the kernel to write to the file, once to switch to the logger thread,
>> > once to write the bits. Just write the damn message and be done with
>> > it...
>>
>> In my logger Twiggy, I have support for what I call "asynchronous
>> logging" - use a child process to do the actual disk write.
>> http://twiggy.wearpants.org/
>>
>> --
>> Peter Fein | wearpants.org | @wearpants
>>
>> I read email at the start and end of each day. IM if urgent.
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
>
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20130307/f12114e2/attachment.html>