today, I’ve release the 0.2.0 of my structlog project whose ambition is no less than changing the way we log in Python. :)
And since it sports dedicated support for Twisted <http://www.structlog.org/en/latest/twisted.html> (but is not limited to it, it wraps *any* logger), I’d like to introduce you to it.
Basically, its premise is that events happen in a context and you want to log out both effortlessly. Nowadays logs are usually consumed by parsers anyway, so regular, easily parsable logs are a good thing™.
This is a nice example of a Twisted application showing off a bit what structlog is capable of:
from twisted.internet import protocol, reactor
logger = structlog.getLogger()
i = 0
self.i += 1
self._counter = Counter()
self._log = logger.new(
def dataReceived(self, data):
log = self._log.bind(data=data)
if __name__ == "__main__":
It will give you an output like:
2013-09-17 17:40:23+0200 [-] Log opened.
2013-09-17 17:40:23+0200 [-] Factory starting on 1234
2013-09-17 17:40:23+0200 [-] Starting factory <twisted.internet.protocol.Factory instance at 0x108301488>
2013-09-17 17:40:28+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=1 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='foo\n' event='echoed data!'
2013-09-17 17:40:33+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=2 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='bar\n' event='echoed data!'
2013-09-17 17:40:44+0200 [Echo,1,127.0.0.1] peer='127.0.0.1' count=1 connection_id='3118f673-482b-471f-a206-e8f97f9a9c2c' data='qux\n' event='echoed data!'
2013-09-17 17:40:53+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=3 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='twisted prevails\n' event='echoed data!'
(you can find this and more examples at <http://www.structlog.org/en/latest/examples.html>)
- It allows you to build context by binding values to loggers. This context is just a dictionary. Once you log an event out (again, arbitrary number of key/value pairs), it gets merged with the context, processed by configurable processor chain and passed to your original logger.
- Your loggers are immutable (by default infidels can use thread local storage) and you’ll get a new one on each binding. Immutable local data is awesome.
- You can define processors that can mangle, filter, and format your log entries.
- Configure once, then just call getLogger (which is a Twisted-friendly alias for get_logger) in regular code.
- Of course there is a JSON renderer built right in – just tell log stash about it and be merry.
*Please* have a look at <http://www.structlog.org/en/0.2.0-0/getting-started.html>, I don’t want to reproduce it here. :)
Credit where credit is due: I picked up the idea of bound loggers from David Reid’s work on otter <https://github.com/rackerlabs/otter/tree/master/otter/log> which in turn was based on ideas by JP (I’ve been told).
You can find everything you need to know at http://www.structlog.org/ – I’d be happy about any constructive feedback and even more so about contributions!
My name is Harry Bock. I'm interested in helping out porting Twisted to
Python 3, and I've popped in IRC a few times to introduce myself and ask a
few questions. A few developers agreed that working on trial dependencies
would be a big help.
In doing some porting work on trial, I stumbled upon a previous porting
effort (possibly by Itamar?) for twisted.python.filepath and related
modules. It seemed like the porting effort included forcing all pathname
inputs to be byte strings instead of native strings. After some
investigation, I believe this is the wrong approach, but I wanted to start
a discussion here first. Some thoughts:
(a) As of Python 3.3, use of the ANSI API in Windows is deprecated, so
many functions in os and os.path raise DeprecationWarning when given byte
strings as input. Although win32 is not an initial target of the porting
effort, we'll have to support it eventually and the API should be supported
(b) Misunderstandings at the application level about the underlying
filesystem's path encoding is not the problem of the Twisted API. Correct
me if I'm wrong, but that's the responsibility of the system administrator
or individual user (at least on UNIX) to set the LANG environment variable,
or for the application to call setlocale(3) to explicitly override it.
(c) If we do not allow unicode strings, we will be forcing the application
developer to decide how to encode paths when using the FilePath API. Per
(b) above, the user will have to call sys.getfilesystemencoding() to
divine what encoding to use before using the API at all, which to me is
terribly annoying and would just add str.encode calls everywhere.
Thus, my vote is that on Python 2.x, Twisted should accept either the
native str or unicode types for path names, and on Python 3.x, only accept
the str type to prevent deprecation issues with system calls. I have a
patch set that will make this happen including unittest modifications; if
there's a consensus I'm happy to open a ticket and submit the patches.
I've found such a code example on "Stack Overflow" regarding errbacks:
self.agent = Agent(reactor, pool=pool)
self.deferred = self.agent.request(
"""you might have cleanup code here, or mark the url as bad in the
database, or something similar"""
Normally only Failure object is passed to gotBadResponse() . I would
like to pass full response body to it - is it possible?
Do I have to encapsulate the response body inside Failure object?
in the method "elements" of twisted.words.xish.domish.Element, the
function "generateElementsQNamed" is called event if the uri if None, so
the uri is checked against None instead of not checked at all.
I think the function "generateElementsNamed" should be called instead,
it looks like a bug for me.
Can you confirm ?
I am writing a simple ftp client which I am trying to dispose of. Right now
I send a quit command:
(the ftp_client reference is a FTPClient built with a ClientCreator)
deferred = ftp_client.quit()
And I register a callback:
print "quit acknowledged...connected?", ftp_client.transport.connected
If I invoke the above the connection is printed as connected.
However if I schedule a callback to check the status later:
print "quit acknowledged...connected?", ftp_client.transport.connected
If I schedule a callback for a second later the transport is not connected.
I've searched quite a bit but I cannot determine a way to register a
callback for when the TCP transport is actually disconnected. Can anyone
offer some insight? I'd like to know explicitly when the client is closed.
I'm still pretty new to twisted and feel I'm slowly getting the hang of it,
enough to realise that this line of code is probably going to block and not
do me any favours, the line is the first line in a resource.Resource
json_request = json.loads(request.content.read())
The resource.Resource is a child of another resource which is passed to
server.Site which is passed to internent.TCPServer.
The problem is I can't work out how I can read the post data from the
request in an async way.
I have a feeling I need to implement a protocol, I'm guessing a
LineReceiver but I can't figure out how I'd tie that in with my current
solution or specifically how the LineReceiver would even read
asynchronously to be honest..
Maybe the read is fine? I need the whole post data to do anything useful I
guess as I can't string to a json decoder that I'm aware of. Just it will
block everything up while I read, which shouldn't be long but I guess I'm
bound to the speed of the person posting.