[Twisted-Python] Modifying the logstring

Hi, I have an application which logs activity on a number of servers by two methods: 1. TCP-based queries (app is client) 2. UDP-based logging (app is server) Each server that is being logged is represented by an instance of a server class. TCP-queries are performed by each instance at set intervals, while the UDP-logs are gathered by a global logging instance which listens to a specific port. Events for a given server can be triggered by both protocols, but are handled by the server instance. In other words: An instance of a server class gathers events by polling the server via TCP /and/ by receiving strings from a UDP logging instance which passes received datagrams as strings to the correct instance. Whenever a loggable event results from the TCP queries, the name of the protocol class, which is a subclass of twisted.internet.protocol.Protocol), is used as the logstring. That is fine, as it allows me to distinguish between different server classes, although I cannot distinguish between individual servers. However, when a loggable event occurs on the UDP logging instance, it writes out a longish protocol name which is the same for all the different server classes that are subscribing to events on that UDP port. I have managed to modify the string by doing the following: self.udpLogger = reactor.listenUDP(myPort, UDPListener()) self.udpLogger.logstr = "UDPLogger" Yet when the logged string is passed, as a string, to a server instance for processing, the logstring for any result remains "UDPLogger" no matter how I pass it or what I do to it. I suppose it is expected behaviour that the protocol should "follow" a result, no matter how it is processed down the line, but is there any way in which I can alter the logstring so that instead of "UDPLogger" or the name of the protocol class, it instead uses the server id or any other value that I specify? What I am getting in my logs is: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [UDPLogger] Something happened on UDP What I would like is: 2010-11-30 10:23:09+0100 [CSS (TCP),12345] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS (UDP),12345] Something happened on UDP where 12345 is a unique identifier of the specific server. Or at least: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS] Something happened on UDP so that I know the result from UDP concerns a CSS-server. I hope this was possible to understand. Please let me know if anything needs to be clarified. :) Cheers, Einar

I was able to modify the logstring for the TCP query by following a similar approach to what I did to the UDP logger: def logPrefix(self): return self.connection.transport.protocol.__class__.__name__ + ", " + str(self.id) def startClient(self): self.f = self.factory(self, password=self.queryPassword) self.connection = reactor.connectTCP(self.ip, self.queryPort, self.f) self.connection.transport.logPrefix = self.logPrefix Perhaps not very elegant, but it does the job and allows me to get a log prefix which not only tells me the protocol but also the exact serverinstance. But is it even possibly to do something similar for the UDP scenario where I have one UDP-logger which calls bound functions from the individual server-representing instances to do the actual parsing? The UDP-listener class is defined like this: class UDPListener(DatagramProtocol): _serverList = {} def addParser(self, ip, port, f): self._serverList[(str(ip), int(port))] = f def removeParser(self, ip, port): if (str(host), int(port)) in self._serverList: del self._serverList[(str(ip), int(port))] def datagramReceived(self, data, (host, port)): #print "received %r from %s:%d" % (data, host, port) if (host, port) in self._serverList: self._serverList[(host, port)](data) Cheers, Einar On Mon, Dec 6, 2010 at 10:21 AM, Einar S. Idsø <einar.twisted@norsk-esport.no> wrote:
Hi,
I have an application which logs activity on a number of servers by two methods: 1. TCP-based queries (app is client) 2. UDP-based logging (app is server)
Each server that is being logged is represented by an instance of a server class. TCP-queries are performed by each instance at set intervals, while the UDP-logs are gathered by a global logging instance which listens to a specific port. Events for a given server can be triggered by both protocols, but are handled by the server instance. In other words: An instance of a server class gathers events by polling the server via TCP /and/ by receiving strings from a UDP logging instance which passes received datagrams as strings to the correct instance.
Whenever a loggable event results from the TCP queries, the name of the protocol class, which is a subclass of twisted.internet.protocol.Protocol), is used as the logstring. That is fine, as it allows me to distinguish between different server classes, although I cannot distinguish between individual servers.
However, when a loggable event occurs on the UDP logging instance, it writes out a longish protocol name which is the same for all the different server classes that are subscribing to events on that UDP port. I have managed to modify the string by doing the following:
self.udpLogger = reactor.listenUDP(myPort, UDPListener()) self.udpLogger.logstr = "UDPLogger"
Yet when the logged string is passed, as a string, to a server instance for processing, the logstring for any result remains "UDPLogger" no matter how I pass it or what I do to it. I suppose it is expected behaviour that the protocol should "follow" a result, no matter how it is processed down the line, but is there any way in which I can alter the logstring so that instead of "UDPLogger" or the name of the protocol class, it instead uses the server id or any other value that I specify?
What I am getting in my logs is: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [UDPLogger] Something happened on UDP
What I would like is: 2010-11-30 10:23:09+0100 [CSS (TCP),12345] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS (UDP),12345] Something happened on UDP
where 12345 is a unique identifier of the specific server. Or at least: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS] Something happened on UDP
so that I know the result from UDP concerns a CSS-server.
I hope this was possible to understand. Please let me know if anything needs to be clarified. :)
Cheers, Einar

To continue my monologue: For future reference, it is worth noting that the code previously posted will not work for reconnecting clients since only the first connection receives the modified logPrefix method. A better approach is to bind logPrefix() in onClientConnected() to ensure it is bound for each new connection. def startClient(self): self.f = self.getFactory() reactor.connectTCP(self.ip, self.queryPort, self.f) def f_onClientConnected(self, protocolInstance): def logPrefix(): return "whateveryouwant" self.f.connection = protocolInstance self.f.connection.transport.logPrefix = logPrefix def getFactory(self): f = ReconnectingClientFactory() f.protocol = whatever f.onClientConnected = self.f_onClientConnected return f I am still in the dark when it comes to modifying the logstring in other cases, though. I am sure I could transfer the results from my UDP logger to my service using strings instead of bound methods, but then the logstring would end up being simply [-], which is of no use to me. Is there really no way in which I can tell Twisted that "All log messages generated by this object shall be on the following format"? It seems like such a simple task to accomplish, yet it completely avoids me. Also let me add that I am completely taken aback with the genius that is Twisted Python. The more I work with it, the more I realise this. Writing clients or servers for *anything* is so simple, and the efficiency of the code, especially with respect to CPU-usage, is wonderful. My application started by polling each server via an external executable written in C, but now fetches info from the same number of servers using protocols written in Python, and even gathers a whole lot more info than before via event logs.CPU-usage has dropped from 20% to 1%, and the memory footprint is only around 40 MB :) Cheers, Einar On Mon, Dec 6, 2010 at 11:51 AM, Einar S. Idsø <einar.twisted@norsk-esport.no> wrote:
I was able to modify the logstring for the TCP query by following a similar approach to what I did to the UDP logger:
def logPrefix(self): return self.connection.transport.protocol.__class__.__name__ + ", " + str(self.id)
def startClient(self): self.f = self.factory(self, password=self.queryPassword) self.connection = reactor.connectTCP(self.ip, self.queryPort, self.f) self.connection.transport.logPrefix = self.logPrefix
Perhaps not very elegant, but it does the job and allows me to get a log prefix which not only tells me the protocol but also the exact serverinstance.
But is it even possibly to do something similar for the UDP scenario where I have one UDP-logger which calls bound functions from the individual server-representing instances to do the actual parsing? The UDP-listener class is defined like this:
class UDPListener(DatagramProtocol): _serverList = {}
def addParser(self, ip, port, f): self._serverList[(str(ip), int(port))] = f
def removeParser(self, ip, port): if (str(host), int(port)) in self._serverList: del self._serverList[(str(ip), int(port))]
def datagramReceived(self, data, (host, port)): #print "received %r from %s:%d" % (data, host, port) if (host, port) in self._serverList: self._serverList[(host, port)](data)
Cheers, Einar
On Mon, Dec 6, 2010 at 10:21 AM, Einar S. Idsø <einar.twisted@norsk-esport.no> wrote:
Hi,
I have an application which logs activity on a number of servers by two methods: 1. TCP-based queries (app is client) 2. UDP-based logging (app is server)
Each server that is being logged is represented by an instance of a server class. TCP-queries are performed by each instance at set intervals, while the UDP-logs are gathered by a global logging instance which listens to a specific port. Events for a given server can be triggered by both protocols, but are handled by the server instance. In other words: An instance of a server class gathers events by polling the server via TCP /and/ by receiving strings from a UDP logging instance which passes received datagrams as strings to the correct instance.
Whenever a loggable event results from the TCP queries, the name of the protocol class, which is a subclass of twisted.internet.protocol.Protocol), is used as the logstring. That is fine, as it allows me to distinguish between different server classes, although I cannot distinguish between individual servers.
However, when a loggable event occurs on the UDP logging instance, it writes out a longish protocol name which is the same for all the different server classes that are subscribing to events on that UDP port. I have managed to modify the string by doing the following:
self.udpLogger = reactor.listenUDP(myPort, UDPListener()) self.udpLogger.logstr = "UDPLogger"
Yet when the logged string is passed, as a string, to a server instance for processing, the logstring for any result remains "UDPLogger" no matter how I pass it or what I do to it. I suppose it is expected behaviour that the protocol should "follow" a result, no matter how it is processed down the line, but is there any way in which I can alter the logstring so that instead of "UDPLogger" or the name of the protocol class, it instead uses the server id or any other value that I specify?
What I am getting in my logs is: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [UDPLogger] Something happened on UDP
What I would like is: 2010-11-30 10:23:09+0100 [CSS (TCP),12345] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS (UDP),12345] Something happened on UDP
where 12345 is a unique identifier of the specific server. Or at least: 2010-11-30 10:23:09+0100 [CSS,client] Something happened on TCP 2010-11-30 10:23:09+0100 [CSS] Something happened on UDP
so that I know the result from UDP concerns a CSS-server.
I hope this was possible to understand. Please let me know if anything needs to be clarified. :)
Cheers, Einar
participants (1)
-
Einar S. Idsø