[Twisted-Python] ANN: Eliot 0.9, the logging system with causality - now with journald support
Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support [1]. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.
Most logging systems can tell you what happened; Eliot tells you _why_ it happened:
$ python linkcheck.py | eliot-tree 4c42a789-76f5-4f0b-b154-3dd0e3041445 +-- check_links@1/started `-- urls: [u'http://google.com', u'http://nosuchurl'] +-- download@2,1/started `-- url: http://google.com +-- download@2,2/succeeded +-- download@3,1/started `-- url: http://nosuchurl +-- download@3,2/failed |-- exception: requests.exceptions.ConnectionError |-- reason: ('Conn aborted', gaierror(-2, 'Name unknown')) +-- check_links@4/failed |-- exception: exceptions.ValueError |-- reason: ('Conn aborted.', gaierror(-2, 'Name unknown'))
And here's the code that generated these logs (eliot-tree [2] was used to render the output):
import sys from eliot import start_action, to_file import requests to_file(sys.stdout)
def check_links(urls): with start_action(action_type="check_links", urls=urls): for url in urls: try: with start_action(action_type="download", url=url): response = requests.get(url) response.raise_for_status() except Exception as e: raise ValueError(str(e))
check_links(["http://google.com"], ["http://nosuchurl"])
Interested? Read more at https://eliot.readthedocs.org/.
Eliot is released under the Apache License 2 by ClusterHQ [3], the Container Data People. We're hiring! [4]
Links: ------ [1] http://eliot.readthedocs.org/en/0.9.0/journald.html [2] https://warehouse.python.org/project/eliot-tree/ [3] https://clusterhq.com [4] https://clusterhq.com/careers/
On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring itamar@itamarst.org wrote:
Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support http://eliot.readthedocs.org/en/0.9.0/journald.html. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.
Most logging systems can tell you what happened; Eliot tells you why it happened:
Cool, this looks awesome Itamar, thanks for letting us know!
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...
-glyph
On 8 Oct, 11:38 pm, glyph@twistedmatrix.com wrote:
On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring itamar@itamarst.org wrote:
Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support http://eliot.readthedocs.org/en/0.9.0/journald.html. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.
Most logging systems can tell you what happened; Eliot tells you why it happened:
Cool, this looks awesome Itamar, thanks for letting us know!
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...
As a user of Eliot and Twisted, an unpleasantness I commonly encounter is that the logs written by trial when running tests for code that uses Eliot for logging are a mish-mash of text messages and mangled Eliot logs. The Eliot bits often get in the way of me reading the text part and the text parts and mangling ensure I can't use any Eliot tools to interpret the Eliot bits.
My understanding is that there aren't any existing hooks in Twisted to customize trial's logging which would allow this to be improved.
There is some related discussion (maybe more oriented towards a particular solution than a full description of the problem, though) on a ticket soon to be a couple years old:
https://twistedmatrix.com/trac/ticket/6939
Jean-Paul
On Oct 8, 2015, at 5:47 PM, exarkun@twistedmatrix.com wrote:
On 8 Oct, 11:38 pm, glyph@twistedmatrix.com wrote:
On Oct 8, 2015, at 6:04 AM, Itamar Turner-Trauring itamar@itamarst.org wrote:
Eliot 0.9 is out, with a new utility for pretty-printing log messages and native journald support http://eliot.readthedocs.org/en/0.9.0/journald.html. You can now route Eliot logs to journald and when an error occurs easily find all logged actions that led up to that particular error, as opposed to most logging systems where this would involve reading all the logs and figuring out which ones apply and which to ignore.
Most logging systems can tell you what happened; Eliot tells you why it happened:
Cool, this looks awesome Itamar, thanks for letting us know!
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...
As a user of Eliot and Twisted, an unpleasantness I commonly encounter is that the logs written by trial when running tests for code that uses Eliot for logging are a mish-mash of text messages and mangled Eliot logs. The Eliot bits often get in the way of me reading the text part and the text parts and mangling ensure I can't use any Eliot tools to interpret the Eliot bits.
My understanding is that there aren't any existing hooks in Twisted to customize trial's logging which would allow this to be improved.
There is some related discussion (maybe more oriented towards a particular solution than a full description of the problem, though) on a ticket soon to be a couple years old:
https://twistedmatrix.com/trac/ticket/6939 https://twistedmatrix.com/trac/ticket/6939
I actually ran into a variant of this exact same issue recently; thank you for directing my attention to this ticket.
I agree that the log output should be customizable, but I'm curious: if the logs were just serialized as JSON by default everywhere, would that make it easier to make the Eliot tooling work well?
-glyph
On 2015-10-08 19:38, Glyph Lefkowitz wrote:
Does Eliot make use of the new logging system at all? If not, is there anything that might make Eliot's Twisted backend easier to maintain if we added it? Just kind of curious about adoption of those new APIs in the world of logging...
For Flocker we have a Twisted->Eliot gateway based on the old logging API. Eventually that'll get moved into Eliot and I guess at that point it'd make sense to switch to new API but haven't really looked at it. Might be able to get structured info at that point which'd be nice.
I should steal some ideas from the new system, but at this point I consider anything that doesn't log actions as legacy logging so I don't personally write anything that *emits* logs with the new Twisted API.
participants (3)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Itamar Turner-Trauring