[Twisted-Python] Instrumenting Reactors
I have a reactor which is getting busier over time and I'd like to find out where the cycles are going. Using the profiler isn't really practical on a server running for days, so I'd like to instrument the reactor to show me which I/O events are taking the longest to deal with. Should I just hack into the reactor somewhere? Or is there something sitting in a library I haven't seen that will help with this? Paul.
Hi!
No guarantee it's an optimal solution, but when I wanted to inspect a
running process (which sounds like what you're doing), I had a pretty good
time with manhole. (If you're not familiar with that: it's in Twisted Conch,
and it's basically just a way to SSH in to a running box and get a REPL).
In that REPL, you can of course do things like inspect the running reactor
:-)
cheers
lvh
On Wed, Apr 6, 2011 at 11:55 AM, Paul Thomas
I have a reactor which is getting busier over time and I'd like to find out where the cycles are going. Using the profiler isn't really practical on a server running for days, so I'd like to instrument the reactor to show me which I/O events are taking the longest to deal with.
Should I just hack into the reactor somewhere? Or is there something sitting in a library I haven't seen that will help with this?
Paul.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- cheers lvh
On Wed, Apr 6, 2011 at 1:35 PM, Laurens Van Houtven <_@lvh.cc> wrote:
Hi!
No guarantee it's an optimal solution, but when I wanted to inspect a running process (which sounds like what you're doing), I had a pretty good time with manhole. (If you're not familiar with that: it's in Twisted Conch, and it's basically just a way to SSH in to a running box and get a REPL).
In that REPL, you can of course do things like inspect the running reactor :-)
It would be nice to be able to attach signals emitters to some parts of running reactor in real-time. If manhole can allow to attach to these emitters - it will be possible to visualize them in funny way. Does anybody know how a twisted reactor model should look like in Blender? =) -- anatoly t.
Hello all. I have been using the twisted library for quite some time and I must say it is the best library for xmlrpc. I have a tcp server listening on a port and I use reactor.run() to start it. Now I want to learn how to daemonise the entire code. I am paisting the code below. from twisted.web import xmlrpc, server #reactor from the twisted library starts the server with a published object and listens on a given port. from twisted.internet import reactor from time import strftime import rpc_organisation import rpc_vendor import rpc_customer import rpc_product #rest of the modules similarly imported, each containing a class inherited from xmlrpc.XMLRPC #the main class class gnukhata(xmlrpc.XMLRPC): #note that all the functions to be accessed by the client must have the xmlrpc_ prefix. def __init__(self): xmlrpc.XMLRPC.__init__(self) #the client however will not use the prefix to call the functions. #self.client_id = dbconnect.getConnection() def xmlrpc_getOrganisationNames(self): #this function is used to return the list of organsations found in gnukhata.xml located at /etc.Returns a list of organisations already present in the file #calling the function for getting list of organisation nodes. orgs = dbconnect.getOrgList() #initialising an empty list for organisation names orgnames = [] for org in orgs: orgname=org.find("orgname") #checking for unique names. #we will use the not in clause here. if orgname.text not in orgnames: orgnames.append(orgname.text) return orgnames #similarly a few other methods for this class. #create an instance of the class to be published as the service. print "initialising application" gnukhata = gnukhata() organisation = rpc_organisation.organisation() gnukhata.putSubHandler('organisation',organisation) people = rpc_people.people() gnukhata.putSubHandler('people',people) vendor = rpc_vendor.vendor() gnukhata.putSubHandler('vendor',vendor) #... and so on. #Now the code for running the server. #publish the object and make it to listen on the given port through reactor print "starting server" reactor.listenTCP(7081, server.Site(gnukhata)) #start the service by running the reactor. reactor.run() The code is from a python script called rpc_main.py which is in a package gnukahta where all other classes are also kept. to run it right now, I get into the gnukhata directory and do a sudo ./rpc_main.py because some actions in this file need sudo permission. Now could some one tell me how to convert this reactor running process into a daemon? right now when I run this rpc_main.py file, the terminal does not return the prompt. I used ./rpc_main.py & to get the terminal back. But on a remote server this process is naturally killed when i logout from the remote shell. Besides I have to make this as a package so making a daeminised version to run is a good option. I have been reading a lot on this issue but did not find any good answer. I am not so much into the details of how twisted works except for this much. I guess my code would reflect how little I know about this amaising library. So I just want to know if I have to write a seperate file to run this as a daemon? As a side note, I used twistd -y rpc_main.py to see what happens. I got complaints on the imports like the one given at the top of my code. These modules are in the same gnukhata package as the rpc_main.py itself and run perfectly in other conditions. So please help, this is really confusing me. Thanking all, Happy hacking. Krishnakant.
hackingKK wrote:
Hello all. I have been using the twisted library for quite some time and I must say it is the best library for xmlrpc. I have a tcp server listening on a port and I use reactor.run() to start it. Now I want to learn how to daemonise the entire code. […] So I just want to know if I have to write a seperate file to run this as a daemon? As a side note, I used twistd -y rpc_main.py to see what happens.
Using twistd -y is a good way to daemonise your Twisted program, but the -y option takes a “TAC” file, not just any Python file. They are briefly described, including an example, here: http://twistedmatrix.com/documents/current/core/howto/application.html#auto5 The really short version is rather than calling reactor methods like reactor.connectTCP directly, you create a bunch of service objects that will do those things when started, and attach those to an Application instance called “application”. That's a TAC file. -Andrew.
On Wed, 2011-04-06 at 10:55 +0100, Paul Thomas wrote:
I have a reactor which is getting busier over time and I'd like to find out where the cycles are going. Using the profiler isn't really practical on a server running for days, so I'd like to instrument the reactor to show me which I/O events are taking the longest to deal with.
You may just want to wrap manual profiling code around your code's entry points (e.g. dataReceived/connectionMade/connectionLost/whatever gets callLatered). It would be nice if you could get the reactor to turn this sort of info on/off in a running system, though.
On 06/04/11 13:19, Itamar Turner-Trauring wrote:
On Wed, 2011-04-06 at 10:55 +0100, Paul Thomas wrote:
I have a reactor which is getting busier over time and I'd like to find out where the cycles are going. Using the profiler isn't really practical on a server running for days, so I'd like to instrument the reactor to show me which I/O events are taking the longest to deal with.
You may just want to wrap manual profiling code around your code's entry points (e.g. dataReceived/connectionMade/connectionLost/whatever gets callLatered). It would be nice if you could get the reactor to turn this sort of info on/off in a running system, though.
I've done this with a decorator & dictionary before, including counting time inside deferred return values separately from the original function call.
On 6 April 2011 10:55, Paul Thomas
Should I just hack into the reactor somewhere? Or is there something sitting in a library I haven't seen that will help with this?
You can time blocking calls by instrumenting twisted.python.log.callWithContext and you could try writing the timing info to something fast, like Redis. Michael
participants (8)
-
anatoly techtonik
-
Andrew Bennetts
-
hackingKK
-
Itamar Turner-Trauring
-
Laurens Van Houtven
-
Michael Thompson
-
Paul Thomas
-
Phil Mayers