2015-05-25 4:19 GMT+02:00 Steven D'Aprano <steve@pearwood.info>:
On Mon, May 25, 2015 at 12:26:33AM +0200, Ludovic Gasc wrote:
Hi Python-Ideas ML,
To resume quickly the idea: I wish to add "extra" attribute to LogMessage, to facilitate structured logs generation.
The documentation for the logging module already includes a recipe for simple structured logging:
https://docs.python.org/2/howto/logging-cookbook.html#implementing-structure...
If I understand correctly this recipe, it's "only" to standardize log message content => not really sysadmin friendly to be read, but the most important, you must continue to parse and construct a database of structured logs to query inside. When you have more than 400+ log messages each second on only one server, rebuild the data structure isn't a negligible cost, contrary to push directly a structured data directly on the wire, directly understandable by your structured log daemon.
At the other extreme, there is the structlog module:
Thank you for the link, it's an interesting project, it's like "logging" module but on steroids, some good logging ideas inside. However, in fact, if I understand correctly, it's the same approach that the previous recipe: Generate a log file with JSON content, use logstash-forwarder to reparse the JSON content, to finally send the structure to logstash, for the query part: https://structlog.readthedocs.org/en/stable/standard-library.html#suggested-...
How does your change compare to those?
In the use case of structlog, drop the logstash-forwarder step to interconnect directly Python daemon with structured log daemon. Even if logstash-forwarder should be efficient, why to have an additional step to rebuild a structure you have at the beginning ? It's certainly possible to monkey patch or override the logging module to have this behaviour, nevertheless, it should be cleaner to be directly integrated in Python. Moreover, in fact, with the "extra" parameter addition, 99% of the work is already done in Python, my addition is only to keep explicit the list of metadata in the LogMessage. The nice to have, at least to me, is that extra dict should be also usable to format string message, to avoid to pass two times the same information. If I don't raise blocking remarks in this discussion, I'll send a patch on bugs.python.org. Regards.
-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/