A new way to configure Python logging

[Vinay Sajip]

Wolodja Wentland <wentland <at> cl.uni-heidelberg.de> writes:

> Could a HTMLHandler be added to the standard set? Preferably one that
> leaves the choice of the template engine to the user.

I haven't done this precisely because users' requirements will be very
different for such a handler. For the same reason, there's no XMLHandler
in the stdlib either. Users can easily define their own handlers to do
whatever they want, which typically will be very application-specific.

> 
> logging.config.fromFiles(['/etc/foo/logging.conf,
>                            os.path.expanduser(
>                            '~/.foo/logging.conf')])
> 

I think this sort of requirement varies sufficiently across developers
and applications that it's best not to bake it into the stdlib. With PEP391
developers are free to put together a configuration from whatever sources
and conventions make sense to them, and then expect logging to just follow
instructions which are very specific to logging configuration, and make no
assumptions about company ans application environments and conventions.

> The user adaptations will overrule the defaults in the shipped
> configuration. I know that I could implement that myself using
> {}.update() and the like, but the use case might be common enough to
> justify inclusion in the logging module.
> 

Unfortunately, not in a standard enough way. If in the future it becomes clear
that a standard approach has emerged/evolved, then this can always be added
later.

> I will give an example.. The basic problem I have with *all* config file
> based configuration right now is that I have to *register* every single
> handler/filter with a logger *within* the configuration or their
> configuration will be lost.
[snip]
> In this configuration the handlers will be lost. There is no way to
> retrieve he configured handlers later on. (or is there?).

You are right, unless handlers (and filters, formatters etc.) are given
names which can be used to refer to them across multiple configuration calls.
This is something I am thinking about and will probably update PEP 391
with my thoughts.

> What I would like to do is:
> 
> --- snip ---
> ...
> if options.user_wants_h1:
>   try:
>       someLogger.addHandler(logging.getConfiguredHandler('h1'))
>   except HandlerNotFound as handler_err:
>     # handle exception
> 
> if options.user_wants_h2:
>   try:
>       someLogger.addHandler(logging.getConfiguredHandler('h2'))
>   except HandlerNotFound as handler_err:
>     # handle exception
> --- snip ---
> 
> ... same for loggers, filters, etc.
> 
> That would enable me to:
> 
> * Create a comprehensive logging building block configuration in its
>   entirety in a nice configuration format. (ie. config file)
> 
> * Easily combine these blocks programmatically

I think your way of working is entirely reasonable, but IMO is not likely to
be so widespread as to make it worthwile baking into the stdlib. You can
easily build your own configuration from which you build the dict to pass
to dictConfig().

> In a way I see three members to the party in the development/usage of
> logging:
> 
> * Logging Expert
> 
>  Will design the logging system for an application/library. Knows the
>  requirements and will be able to design different parts of the system.
>  She will then tell another developer (see below) which blocks are
>  available.
> 
> * Developer
> 
>   A person that knows about the blocks and combines them
>   programmatically, designs the user interface and complains about
>   bugs/new requirements in/for the logging system to the "Logging
>   Expert".
> 
> * User
> 
>   A user gets exposed to different ways in which to change the logging
>   system:
> 
>   - command line options (switches to turn whole blocks off/on)
>   - configuration files
> 
>     These *may* a subset of the configuration options that the developer
>     wants to expose to the user (format, dateformat, ...)
>     (see above)
> 

Those three roles appear reasonable, but I would say that the expert-designed
blocks would be specialised handlers, filters and formatters. That's not a 
full-time job, though ;-)

In addition there are system admin users, who can tweak logging configurations
in response to user community feedback about problems, and to help developers
diagnose faults. In some companies and environments, there are strict walls
between developers and production support teams. In order not to assume too
much about such environmental, non-technical constraints, logging configuration
should not try to be too clever.


Thanks for your thoughts,

Vinay Sajip