[Python-Dev] PEP282 and the warnings framework

[Vinay Sajip]

> After having implement several real-time debuggers for military
> systems (and lurking for months on this list) let me weigh in.
>
> First, the "Severity" of an event is one of many attributes.
> The attributes include an event name (including a subject),
> time, the host, the environment, a descriptive string, an
> instance of an object, etc.

Yes. You can boil all these attributes down to "what happened?", "when did
it happen?", "where did it happen?" (i.e. which area of the application) and
"how important is it?", all of which are more or less orthogonal.

> Second, the Severity is a tricky business to get right.  As has
> been observed, one person's "Fatal" is another person's
> "Warning".  Depends on context.  I can remember putting in lists
> of overrides that would treat ordinarily fatal OS exceptions as
> warnings so our application could continue to run in spite of
> lossed I/O interrupts.

This is why a logging system should not pre-empt designer decisions about
severity, nor take unilateral actions such as shutdown. It's there for only
one task - notifying interested audiences of interesting events which occur
during an application's execution. The events themselves are usually defined
by the logging calls defined by the developer; the definitions of
"interesting events" and "interested audience" are mutable by changing the
logging configuration.

> Third, all programming boils down to comparison of integers.  If
> we start from comparison of integers, we've lost any possibility
> of putting more meaning into the program.  If we start with
> class definitions, then the final implementation may be some
> kind of RTTI that is implemented as an integer comparison.

I don't think you lose much flexibility if you use integers to define
levels, as ultimately levels are only one component of log filtering anyway.
Their main advantage is that they are cheap to construct, quick to compare
with each other, easy to understand and readily available :-)

> Fourth, all logging systems need filters.  Often those filters
> are subject-area based, and have little to do with severity or
> with the ultimate exception/error class to which the event
> belongs.  For "normal production" we only want to see severe
> errors or fatal errors that are the final outcome of the
> application.  For "QA testing" we want to see all errors from
> all component modules.  For "Integration testing" we want to see
> all warnings from all classes.  etc.

Yes. PEP282/logging.py offers these mechanisms. Logger channels offer
subject-area filtering, levels offer severity filtering, and filters offer
pretty much anything you want in terms of flexibility.

> Simple numeric codes won't provide much useful filtering.
> Programmers will work around it by dynamically assigning a
> severity to permit logging under some circumstances.  People
> will build applications that use an environment variable to set
> message severity so that it will be very severe during testing
> (and get logged) but not severe in production (to reduce the log
> blather).

Earlier, you said  'For "normal production" we only want to see severe
errors or fatal errors that are the final outcome of the application'. To
me, that maps most naturally to a numeric level.

> A subject area string (or tuple) is a cool way to do business.

You get this by using logging channels with subject-based names.

> It allows one to label the event with information like module,
> class, method, event.  If someone wants to include severity as
> part of their subject string, that's good.  Then you can filter
> on subject areas.

Not only, but also. Why constrain needlessly to just one way of filtering?

> The idea is to use a simple fnmatch-style filter.
> log.watch( 'error.*' ) or log.watch( 'warn.*', 'error.*' )
> or log.watch( 'error.pil.gif.*' ).

The logging.py distribution includes a simple example of a MatchFilter (in a
test script) which works
something like this. It's relatively easy to adapt to more specialized
needs.

> I'm not the only one who's found this effective.  If you dig
> into the TIBCO commercial product, they devote some real energy
> to making useful subject names for the message traffic in their
> middleware product.

Yes, and developers who use logging should, for large applications, devote
the same energy to ensuring that their logging design is in keeping with the
complexity of their application. After all, logging is a kind of
instrumentation.

> My vote is:
> A) existing exception/error hierarchy
> B) LogEvent class with message string, subject string, delayed
> introspection, automatic timestamping.  LogEventObject and
> LogEventEnviron subclasses would drag around additional details.
> C) The logger would filter LogEvents based on subject string.
> Since each event is an object, the logger could simply pickle
> the events, or write their __str__ to stderr.

Not sure what you mean by (A). As far as I can see, logging.py provides all
of (B)'s first sentence. You can use derived classes where needed, too.
(C)'s first sentence I find a little restrictive, why does it *need* to work
like that?

Regards

Vinay Sajip