[Python-ideas] Fwd: [Python-Dev] An yocto change proposal in logging module to simplify structured logs support

[Ludovic Gasc]

2015-05-25 4:19 GMT+02:00 Steven D'Aprano <steve at pearwood.info>:

> On Mon, May 25, 2015 at 12:26:33AM +0200, Ludovic Gasc wrote:
> > Hi Python-Ideas ML,
> >
> > To resume quickly the idea: I wish to add "extra" attribute to
> LogMessage,
> > to facilitate structured logs generation.
>
> The documentation for the logging module already includes a recipe for
> simple structured logging:
>
>
> https://docs.python.org/2/howto/logging-cookbook.html#implementing-structured-logging


If I understand correctly this recipe, it's "only" to standardize log
message content => not really sysadmin friendly to be read, but the most
important, you must continue to parse and construct a database of
structured logs to query inside.
When you have more than 400+ log messages each second on only one server,
rebuild the data structure isn't a negligible cost, contrary to push
directly a structured data directly on the wire, directly understandable by
your structured log daemon.


>
>
> At the other extreme, there is the structlog module:
>
> https://structlog.readthedocs.org/en/stable/


Thank you for the link, it's an interesting project, it's like "logging"
module but on steroids, some good logging ideas inside.
However, in fact, if I understand correctly, it's the same approach that
the previous recipe: Generate a log file with JSON content,
use logstash-forwarder to reparse the JSON content, to finally send the
structure to logstash, for the query part:
https://structlog.readthedocs.org/en/stable/standard-library.html#suggested-configuration


> How does your change compare to those?
>

In the use case of structlog, drop the logstash-forwarder step to
interconnect directly Python daemon with structured log daemon.
Even if logstash-forwarder should be efficient, why to have an additional
step to rebuild a structure you have at the beginning ?

It's certainly possible to monkey patch or override the logging module to
have this behaviour, nevertheless, it should be cleaner to be directly
integrated in Python.
Moreover, in fact, with the "extra" parameter addition, 99% of the work is
already done in Python, my addition is only to keep explicit the list of
metadata in the LogMessage.

The nice to have, at least to me, is that extra dict should be also usable
to format string message, to avoid to pass two times the same information.

If I don't raise blocking remarks in this discussion, I'll send a patch on
bugs.python.org.

Regards.


>
>
> --
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150525/4de19ea8/attachment.html>