A new way to configure logging
At present, configuration of Python's logging package can be done in one of two ways: 1. Create a ConfigParser-readable configuration file and use logging.config.fileConfig() to read and implement the configuration therein. 2. Use the logging API to programmatically configure logging using getLogger(), addHandler() etc. The first of these works for simple cases but does not cover all of the logging API (e.g. Filters). The second of these provides maximal control, but besides requiring users to write the configuration code, it fixes the configuration in Python code and does not facilitate changing it easily at runtime. In addition, the ConfigParser format appears to engender dislike (sometimes strong dislike) in some quarters. Though it was chosen because it was the only configuration format supported in the stdlib at that time, many people regard it (or perhaps just the particular schema chosen for logging's configuration) as 'crufty' or 'ugly', in some cases apparently on purely aesthetic grounds. Recent versions of Python of course support an additional battery-included format which can be used for configuration - namely, JSON. Other options, such as YAML, are also possible ways of configuring systems, Google App Engine-style, and PyYAML has matured nicely. There has also been talk on the django-dev mailing list about providing better support for using Python logging in Django. When it happens (as of course I hope it does) this has the consequence that many new users who use Django but are relatively inexperienced in Python (e.g. in PHP shops which are moving to Django) will become exposed to Python logging. As Django is configured using a Python module and use of ConfigParser-style files is not a common approach in that ecosystem, users will find either of the two approaches outlined above a particular pain point when configuring logging for their Django applications and websites, unless something is done to avoid it. All three of the contenders for the title of "commonly found configuration mechanism" - JSON, YAML and Python code - will be expressible, in Python, as Python dicts. So it seems to make sense to add, to logging.config, a new callable bound to "dictConfig" which will take a single dictionary argument and configure logging from that dictionary. An important facet of implementing such a scheme will be the format or schema which the dictionary has to adhere to. I have started working on what such a schema would look like, and if people here think it's a good idea to go ahead with this, I'll provide the details of the schema on a separate post which I'll also cross-post on comp.lang.python so that I can get feedback from there, too. In outline, the scheme I have in mind will look like this, in terms of the new public API: class DictConfigurator: def __init__(self, config): #config is a dict-like object (duck-typed) import copy self.config = copy.deepcopy(config) def configure(self): # actually do the configuration here using self.config dictConfigClass = DictConfigurator def dictConfig(config): dictConfigClass(config).configure() This allows easy replacement of DictConfigurator with a suitable subclass where needed. What's the general feeling here about this proposal? All comments and suggestions will be gratefully received. Regards, Vinay Sajip
2009/10/7 Vinay Sajip <vinay_sajip@yahoo.co.uk>:
What's the general feeling here about this proposal? All comments and suggestions will be gratefully received.
+1 One option I would have found useful in some code I wrote would be to extend the configuration - class DictConfigurator: ... def extend(self, moreconfig): import copy more = copy.deepcopy(moreconfig) # Not sure if this is needed? self.config.update(more) Paul.
Paul Moore <p.f.moore <at> gmail.com> writes:
One option I would have found useful in some code I wrote would be to extend the configuration -
class DictConfigurator: ... def extend(self, moreconfig): import copy more = copy.deepcopy(moreconfig) # Not sure if this is needed? self.config.update(more)
See my response to Paul Rudin's post about incremental configuration. Regards, Vinay Sajip
On Wed, Oct 7, 2009 at 9:49 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
At present, configuration of Python's logging package can be done in one of two ways:
1. Create a ConfigParser-readable configuration file and use logging.config.fileConfig() to read and implement the configuration therein. 2. Use the logging API to programmatically configure logging using getLogger(), addHandler() etc.
[...] Wow ! Great explanation ! ;o)
All three of the contenders for the title of "commonly found configuration mechanism" - JSON, YAML and Python code - will be expressible, in Python, as Python dicts.
.INI files too { 'section1' : {'opt11' : 'val1', 'opt12' : 32323}, 'section1' : {'opt21' : 'val2', 'opt22' : 32323}, 'section1' : {'opt31' : 'val3', 'opt32' : 32323}, ... } In fact it seems that this is a typical characteristic of config formats (CMIIW)
So it seems to make sense to add, to logging.config, a new callable bound to "dictConfig" which will take a single dictionary argument and configure logging from that dictionary.
This kind of problems is similar to the one mentioned in another thread about modifying config options after executing commands. In that case I mentioned that the same dict-like interface also holds for WinReg and so on ... So thinking big (yes ! I have a big head ! ) I think that the best approach in this case is to build an adaptation (simple) layer on top of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific extensions for these formats . Perhaps the proper interfaces are already there (e.g. `dict`, `shelve` ) and I'm just blind and looking somewhere else ;o) If `dict` (possibly nested ;o) is enough to represent config files then the new method should be ok ... IMHO [...]
In outline, the scheme I have in mind will look like this, in terms of the new public API:
class DictConfigurator: def __init__(self, config): #config is a dict-like object (duck-typed)
========
import copy self.config = copy.deepcopy(config)
Why ?
def configure(self): # actually do the configuration here using self.config
dictConfigClass = DictConfigurator
def dictConfig(config): dictConfigClass(config).configure()
This allows easy replacement of DictConfigurator with a suitable subclass where needed.
extension is cool ... what's the point about adding the new method instead of using `DictConfigurator` directly ?
What's the general feeling here about this proposal?
I'm feeling things ... :) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Nabble - Trac Users - Coupling trac and symfony framework - http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-an...
Olemis Lang <olemis <at> gmail.com> writes:
This kind of problems is similar to the one mentioned in another thread about modifying config options after executing commands. In that case I mentioned that the same dict-like interface also holds for WinReg and so on ...
So thinking big (yes ! I have a big head ! ) I think that the best approach in this case is to build an adaptation (simple) layer on top of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific extensions for these formats . Perhaps the proper interfaces are already there (e.g. `dict`, `shelve` ) and I'm just blind and looking somewhere else ;o)
Sorry, you've lost me :-)
import copy self.config = copy.deepcopy(config)
Why ?
So I'm free to mutate self.config as I see fit.
extension is cool ... what's the point about adding the new method instead of using `DictConfigurator` directly ?
When you say "the new method", if you mean "configure" - the pattern is so that a subclass can override __init__ and do additional setup before configure() is called. Regards, Vinay Sajip
On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Olemis Lang <olemis <at> gmail.com> writes:
This kind of problems is similar to the one mentioned in another thread about modifying config options after executing commands. In that case I mentioned that the same dict-like interface also holds for WinReg and so on ...
So thinking big (yes ! I have a big head ! ) I think that the best approach in this case is to build an adaptation (simple) layer on top of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific extensions for these formats . Perhaps the proper interfaces are already there (e.g. `dict`, `shelve` ) and I'm just blind and looking somewhere else ;o)
Sorry, you've lost me :-)
Never mind . I was just trying to say that using `dict`, an adapter could be implemented (if not already there ;o) for multiple formats like the ones I mentioned above and the solution would cover many config formats ... and also I was saying that I was not sure about whether this is correct , I mean for *ANY* config formats, but definitely will work for many ;o)
import copy self.config = copy.deepcopy(config)
Why ?
So I'm free to mutate self.config as I see fit.
why not to let the user do it if he | she thinks so. I mean if somebody supplies in a temporary mapping that can be written then why to spend that time cloning the dict ? If the function has side-effects, well just document it ;o)
extension is cool ... what's the point about adding the new method instead of using `DictConfigurator` directly ?
When you say "the new method", if you mean "configure" - the pattern is so that a subclass can override __init__ and do additional setup before configure() is called.
Sorry I was confused. I was talking about `dictConfig` *FUNCTION* (to be added to logging.config ... isn't it ? ;o). I mean if dictConfig is sematically equivalent to a simple `dc.configure` why to add such function ? PS: Not a big problem anyway ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Looking for a technique to create flexible, graphical dashboards ... - http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789
On Wed, Oct 7, 2009 at 11:26 AM, Olemis Lang <olemis@gmail.com> wrote:
On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Olemis Lang <olemis <at> gmail.com> writes:
This kind of problems is similar to the one mentioned in another thread about modifying config options after executing commands. In that case I mentioned that the same dict-like interface also holds for WinReg and so on ...
So thinking big (yes ! I have a big head ! ) I think that the best approach in this case is to build an adaptation (simple) layer on top of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific extensions for these formats . Perhaps the proper interfaces are already there (e.g. `dict`, `shelve` ) and I'm just blind and looking somewhere else ;o)
Sorry, you've lost me :-)
Never mind . I was just trying to say that using `dict`, an adapter could be implemented (if not already there ;o) for multiple formats like the ones I mentioned above and the solution would cover many config formats ... and also I was saying that I was not sure about whether this is correct , I mean for *ANY* config formats,
Intention = answer myself + sharing with you => for a better Pyland Maybe we could learn something from Augeas [1]_ and use their experience to include some of those features to `stdlib` . It seems to be an ambitious attempt to put all the pieces in config-land together; and AFAICS they handle (a lof of) config formats and have an API for that, and ... CMIIW anyway .. [1] Augeas (http://augeas.net/) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article:
Vinay Sajip <vinay_sajip@yahoo.co.uk> writes:
What's the general feeling here about this proposal? All comments and suggestions will be gratefully received.
How about the global logging configuration being available as an object supporting the usual dictionary interface? So you could, for example, do something like: "logging.dictconfig.update(partialconfig)"
On Wed, Oct 7, 2009 at 11:06 AM, Paul Rudin <paul@rudin.co.uk> wrote:
Vinay Sajip <vinay_sajip@yahoo.co.uk> writes:
What's the general feeling here about this proposal? All comments and suggestions will be gratefully received.
How about the global logging configuration being available as an object supporting the usual dictionary interface? So you could, for example, do something like: "logging.dictconfig.update(partialconfig)"
Seems interesting ... yes ! ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Search for milestone change in change history - Trac Users ... - http://feedproxy.google.com/~r/TracGViz-full/~3/Gsj4xLjuCtk/578d40b734e5e753
Paul Rudin <paul@...> writes:
How about the global logging configuration being available as an object supporting the usual dictionary interface? So you could, for example, do something like: "logging.dictconfig.update(partialconfig)"
A "partial configuration" only makes sense under certain limited conditions. For example, a dict used for configuration will map to a graph of objects - filters, formatters, handlers and loggers - which by necessity need to be referred to via a string key in the dict - because you can't easily encode a graph in a dict which comes from, say, a YAML or JSON file. These keys are temporary and just used to indicate e.g. which handlers to attach to which loggers for that particular configuration call: { "handlers": { "h1": { configuration for a handler }, "h2": { configuration for another handler }, "h3": { configuration for yet another handler }, }, "loggers": { "foo": { "level": "DEBUG", "handlers": ["h1", "h3"], }, "bar.baz": { "level": "INFO", "handlers": ["h2", "h3"], }, "spam.eggs": { "level": "ERROR", "handlers": ["h1", "h3"], }, } } Here, the keys "h1", "h2" etc. are just used to encode connections in the graph, and are useless outside the context of the specific configuration call. If you pass a partial configuration, to be implemented incrementally, there's no way of knowing if (in general) it's going to be consistent with the existing configuration. Nor can you refer to e.g. handlers, formatters or filters you passed in earlier configuration calls. Since logger names such as "foo", "bar.baz" in the above example *are* meaningful across multiple configuration calls, it *would be* possible to make certain changes in the configuration - such as changing levels of loggers - by passing in a partial configuration. However, if this "partial" functionality is provided without restriction, it's hard to ensure that a user doesn't inadvertently misconfigure logging. And while that is equally true with a one-shot configuration, it's at least easier to debug. Things can get pretty murky from a diagnostic point of view if code can make arbitrary changes to the logging configuration during application execution, so that's not a good practice to encourage :-) Also, providing a readable dict-like interface to the internal configuration will almost certainly lead to backward-compatibility headaches in the future. Regards, Vinay Sajip
On approximately 10/7/2009 7:49 AM, came the following characters from the keyboard of Vinay Sajip:
In outline, the scheme I have in mind will look like this, in terms of the new public API:
class DictConfigurator: def __init__(self, config): #config is a dict-like object (duck-typed) import copy self.config = copy.deepcopy(config)
def configure(self): # actually do the configuration here using self.config
dictConfigClass = DictConfigurator
def dictConfig(config): dictConfigClass(config).configure()
This allows easy replacement of DictConfigurator with a suitable subclass where needed.
Concept sounds good, and the idea of separating the syntax of the configuration file from the action of configuring is a clever way of avoiding the "syntax of the (day|platform|environment)" since everyone seems to invent new formats. So people that want to expose a text file to their users have the choice of syntaxes to expose, and then they do logCfg = readMyFavoriteSyntax( logCfgFile ) # or extract a subset of a larger config file for their project DictConfigurator( logCfg ) But DictConfigurator the name seems misleading... like you are configuring how dicts work, rather than how logs work. Maybe with more context this is not a problem, but if it is a standalone class, it is confusing what it does, by name alone. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
Glenn Linderman <v+python <at> g.nevcal.com> writes:
But DictConfigurator the name seems misleading... like you are configuring how dicts work, rather than how logs work. Maybe with more context this is not a problem, but if it is a standalone class, it is confusing what it does, by name alone.
Of course the fully qualified name would be logging.config.DictConfigurator which does provide the requisite context, but if I think of/someone suggests a better name I'll certainly use it.
On approximately 10/7/2009 10:45 PM, came the following characters from the keyboard of Vinay Sajip:
Glenn Linderman <v+python <at> g.nevcal.com> writes:
But DictConfigurator the name seems misleading... like you are configuring how dicts work, rather than how logs work. Maybe with more context this is not a problem, but if it is a standalone class, it is confusing what it does, by name alone.
Of course the fully qualified name would be logging.config.DictConfigurator which does provide the requisite context, but if I think of/someone suggests a better name I'll certainly use it.
Yes, I suspected it might be something like that, and that does provide the context, so I'm content. Thanks for verifying it, and the rest is bikeshedding... In thinking more about this, I guess I would have the same comment about the existing fileConfig() API also... it isn't file that is being configured, but the logs. readConfigFromFile would have been more descriptive (and lots longer :( ). Had fileConfig been called "bulkConfig" or "configAll" (these names attempt to contrast the function of this API versus the individual APIs that configure one thing at a time) taking a file parameter, then it could just simply/suddenly also allow a dict parameter, and no new API would have to be created. Too late for this, however, and with the new API, it would be possible to deprecate the fileConfig API, and tell the user to use the ConfigParser (or JSON or YAML or whatever) to create the dict, and then pass that to DictConfigurator (or bulkConfig or whatever). -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
On Thu, Oct 8, 2009 at 2:44 AM, Glenn Linderman <v+python@g.nevcal.com> wrote:
On approximately 10/7/2009 10:45 PM, came the following characters from the keyboard of Vinay Sajip:
Glenn Linderman <v+python <at> g.nevcal.com> writes:
But DictConfigurator the name seems misleading... like you are configuring how dicts work, rather than how logs work. Maybe with more context this is not a problem, but if it is a standalone class, it is confusing what it does, by name alone.
Of course the fully qualified name would be logging.config.DictConfigurator which does provide the requisite context, but if I think of/someone suggests a better name I'll certainly use it.
[...]
In thinking more about this, I guess I would have the same comment about the existing fileConfig() API also... it isn't file that is being configured, but the logs. readConfigFromFile would have been more descriptive (and lots longer :( ).
cfgFromIni | cfgFromFile ? I prefer the former because YAML, shelves, JSON & Co can also be stored in a file , so at least I get a better understanding of the format in use. Otherwise I'd had to read the docs or figure out which one
Had fileConfig been called "bulkConfig" or "configAll" (these names attempt to contrast the function of this API versus the individual APIs that configure one thing at a time) taking a file parameter, then it could just simply/suddenly also allow a dict parameter, and no new API would have to be created. Too late for this, however, and with the new API, it would be possible to deprecate the fileConfig API, and tell the user to use the ConfigParser (or JSON or YAML or whatever) to create the dict, and then pass that to DictConfigurator (or bulkConfig or whatever).
similar and consistent with what I mentioned before but IMO fileConfig should remain for backwards compatibility Objections ? -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: Looking for a technique to create flexible, graphical dashboards ... - http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789
On Oct 7, 2009, at 10:49 AM, Vinay Sajip wrote:
All three of the contenders for the title of "commonly found configuration mechanism" - JSON, YAML and Python code - will be expressible, in Python, as Python dicts. So it seems to make sense to add, to logging.config, a new callable bound to "dictConfig" which will take a single dictionary argument and configure logging from that dictionary.
I've had bad experiences in the past with dictionary-based APIs. They seem "simpler" in the short run, because the user "only needs to create some dictionaries". Once the complexity of that nested dictionary grows to a certain point, though, one has to refer back to documentation constantly to make sure the structure conforms to the "schema". Building a simple config tree using light-weight classes with documented APIs tends to be more sustainable in the long run. Doug
I've had bad experiences in the past with dictionary-based APIs. They seem
"simpler" in the short run, because the user "only needs to create some dictionaries". Once the complexity of that nested dictionary grows to a certain point, though, one has to refer back to documentation constantly to make sure
Fair point, and agreed that the schema needs some care, here's roughly what I'm thinking of as an example configuration (YAML): formatters: brief: format: '%(levelname)-8s: %(name)-15s: %(message)s' precise: format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s' filters: allow_foo: name: foo handlers: console: class : logging.StreamHandler formatter: brief level : INFO stream : sys.stdout filters: [allow_foo] file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3 debugfile: class : logging.FileHandler formatter: precise filename: logconfig-detail.log mode: a loggers: foo: level : ERROR handlers: [debugfile] spam: level : CRITICAL handlers: [debugfile] propagate: no bar.baz: level: WARNING root: level : DEBUG handlers : [console, file] It's not too deeply nested, and I can't see any need for it being more deeply nested. It more or less mirrors the way you'd have to write the corresponding ConfigParser-compatible configuration, but if you use a dict as a common format, you have the benefits which I mentioned of supporting JSON, YAML or Python code in a Django settings.py.
the structure conforms to the "schema". Building a simple config tree using light-weight classes with documented APIs tends to be more sustainable in the long run.
When you say 'config tree using light-weight classes', I presume you mean a parallel set of classes to the actual logging classes which will be instantiated? ISTM logging classes are already reasonably usable for programmatic configuration, but this doesn't address e.g. updating configurations easily without program code changes, or the ability to configure from say a YAML node, where YAML may be being used for application configuration as a whole. (Unless of course I've misunderstood what you're getting at.) Regards, Vinay Sajip
On Oct 8, 2009, at 10:47 AM, Vinay Sajip wrote:
I've had bad experiences in the past with dictionary-based APIs. They seem
"simpler" in the short run, because the user "only needs to create some dictionaries". Once the complexity of that nested dictionary grows to a certain point, though, one has to refer back to documentation constantly to make sure
Fair point, and agreed that the schema needs some care, here's roughly what I'm thinking of as an example configuration (YAML):
formatters: brief: format: '%(levelname)-8s: %(name)-15s: %(message)s' precise: format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s' filters: allow_foo: name: foo handlers: console: class : logging.StreamHandler formatter: brief level : INFO stream : sys.stdout filters: [allow_foo] file: class : logging.handlers.RotatingFileHandler formatter: precise filename: logconfig.log maxBytes: 1024 backupCount: 3 debugfile: class : logging.FileHandler formatter: precise filename: logconfig-detail.log mode: a loggers: foo: level : ERROR handlers: [debugfile] spam: level : CRITICAL handlers: [debugfile] propagate: no bar.baz: level: WARNING
root: level : DEBUG handlers : [console, file]
It's not too deeply nested, and I can't see any need for it being more deeply nested. It more or less mirrors the way you'd have to write the corresponding ConfigParser-compatible configuration, but if you use a dict as a common format, you have the benefits which I mentioned of supporting JSON, YAML or Python code in a Django settings.py.
Yes, that makes sense.
the structure conforms to the "schema". Building a simple config tree using light-weight classes with documented APIs tends to be more sustainable in the long run.
When you say 'config tree using light-weight classes', I presume you mean a parallel set of classes to the actual logging classes which will be instantiated? ISTM logging classes are already reasonably usable for programmatic configuration, but this doesn't address e.g. updating configurations easily without program code changes, or the ability to configure from say a YAML node, where YAML may be being used for application configuration as a whole. (Unless of course I've misunderstood what you're getting at.)
The probably are light-weight enough, but I didn't want to assume those classes would be used directly. I suspect they could be, though. I guess there aren't *so* many levels or keys that the data structure will become unwieldy. Doug
participants (6)
-
Doug Hellmann -
Glenn Linderman -
Olemis Lang -
Paul Moore -
Paul Rudin -
Vinay Sajip