[Twisted-Python] ANN: Eliot 1.7, the causal logging library, now with inlineCallbacks support
Python and Twisted's built-in `logging` output a stream of factoids: they’re interesting, but you can’t really tell what’s going on. * Why is your application slow? * What caused this code path to be chosen? * Why did this error happen? Standard logging can’t answer these questions. But with a better model you could understand what and why things happened in your application. You could pinpoint performance bottlenecks, you could understand what happened when, who called what. That is what Eliot does. `Eliot` is a Python logging system that outputs causal chains of *actions*: actions can spawn other actions, and eventually they either *succeed or fail*. The resulting logs tell you the story of what your software did: what happened, and what caused it. Eliot has had good Twisted support from the start, but the latest release also supports inlineCallbacks, thanks to Jean-Paul Calderone. You can learn more about Eliot at https://eliot.readthedocs.io/en/stable/ -- Itamar Turner-Trauring
Hey Stranger, long time no speak :-) On 21/03/2019 15:27, Itamar Turner-Trauring wrote:
Python and Twisted's built-in |logging| output a stream of factoids: they’re interesting, but you can’t really tell what’s going on.
* Why is your application slow? * What caused this code path to be chosen? * Why did this error happen?
Standard logging can’t answer these questions.
But with a better model you could understand what and why things happened in your application. You could pinpoint performance bottlenecks, you could understand what happened when, who called what.
That is what Eliot does. |Eliot| is a Python logging system that outputs causal chains of *actions*: actions can spawn other actions, and eventually they either *succeed or fail*. The resulting logs tell you the story of what your software did: what happened, and what caused it.
Have you looked at tools like https://www.jaegertracing.io/? Feels like quite a similar approach to Eliot, and it's be fantastic if Eliot could provide contexts to Jaeger for Twisted applications! cheers, Chris
Hi! On Fri, Mar 22, 2019, at 3:01 AM, Chris Withers wrote:
Have you looked at tools like https://www.jaegertracing.io/? Feels like quite a similar approach to Eliot, and it's be fantastic if Eliot could provide contexts to Jaeger for Twisted applications!
There are some similarities, yes. Eliot was in part inspired by a tool called Frost that was created before—and was similar to—Dapper. And Dapper inspired a whole generation of tracing systems. However, these tracing systems are mostly about large scale distributed systems with lots of different services, so from what I've seen their operational requirements tend to be more heavyweight. And they often seem less interested in tracing within a program. Eliot is very Python-focused, and about smaller scale systems (and e.g. very useful for scientific computing, which Jaeger and similar systems likely have no interest in). I have vaguely considered hooking up Eliot to OpenTracing so they can interoperate (https://github.com/itamarst/eliot/issues/361). Easiest thing if you want to use Jaeger might be extending Jaeger's Python library to support Twisted, e.g. borrow DeferredContext API design from Eliot. But it's possible Eliot could go in the other direction and feed into Jaeger (I'm not sure whether OpenTracing spec provides for compatibility on wire level, or just language-level API). -Itamar (I might, by the way, have availability for related consulting at some point.)
On Mar 21, 2019, at 8:27 AM, Itamar Turner-Trauring
wrote: Python and Twisted's built-in logging output a stream of factoids: they’re interesting, but you can’t really tell what’s going on.
Why is your application slow? What caused this code path to be chosen? Why did this error happen? Standard logging can’t answer these questions.
But with a better model you could understand what and why things happened in your application. You could pinpoint performance bottlenecks, you could understand what happened when, who called what.
I should note that if anyone out there is interested in getting involved with contributing to Twisted, but the bugs look big and daunting, finding a module that uses twisted.python.log and updating it to: 1. use twisted.logger 2. format all of its fields with structured information rather than string formatting would really help with the consumption of Twisted's native log messages by more advanced tools like Eliot, and generally make Twisted better and push it into the future. -glyph
On Thursday, 21 March 2019 15:27:35 GMT Itamar Turner-Trauring wrote:
Python and Twisted's built-in `logging` output a stream of factoids: they’re interesting, but you can’t really tell what’s going on.
* Why is your application slow? * What caused this code path to be chosen? * Why did this error happen? Standard logging can’t answer these questions.
But with a better model you could understand what and why things happened in your application. You could pinpoint performance bottlenecks, you could understand what happened when, who called what.
That is what Eliot does. `Eliot` is a Python logging system that outputs causal chains of *actions*: actions can spawn other actions, and eventually they either *succeed or fail*. The resulting logs tell you the story of what your software did: what happened, and what caused it.
Eliot has had good Twisted support from the start, but the latest release also supports inlineCallbacks, thanks to Jean-Paul Calderone.
You can learn more about Eliot at https://eliot.readthedocs.io/en/stable/
Interesting library. It seems that I cannot have a action finish with an error unless I raise an exception. That seems limiting, there are lots of cases where I have code that hits an error condition and does not raise an exception. For example an HTTP request that gets a non 200 status. Have I missed something? Barry
Interesting library.
It seems that I cannot have a action finish with an error unless I raise an exception. That seems limiting, there are lots of cases where I have code that hits an error condition and does not raise an exception. For example an HTTP request that gets a non 200 status.
Have I missed something?
You don't have to raise an exception, though the current API requires you to create an exception instance: --- from eliot import start_action action = start_action(action_type="http_request"): with action.context(): response = http_request() if response.code == 200: action.finish() return response.json else: action.finish(HTTPError(code=response.code)) return None --- The Twisted APIs for Eliot likewise support Deferreds using a DeferredContext object, again with no need to raise an exception: https://eliot.readthedocs.io/en/stable/generating/twisted.html#actions-and-d... -Itamar
participants (4)
-
Chris Withers
-
Glyph
-
Itamar Turner-Trauring
-
Scott, Barry