Re: [Twisted-Python] Advice on porting Python application to Twisted
![](https://secure.gravatar.com/avatar/7edc0cf41bada7a000ca80712e36a14a.jpg?s=120&d=mm&r=g)
Hi Matthew - I have a couple of idioms I use for #2 and #3 in your message. Here they are. #2) For timer events, I create a function that when called, continuously schedules itself again in the reactor, does some work for the current tick, and then exits. I've used this down to 1-second intervals. If you're looking for sub-millisecond level timing, this may not be appropriate for your application. def timerFunction(reactor): reactor.callLater(1.0, timerFunction, reactor) # do the work for this time tick # etc etc return # Somewhere in main do this to kick it off from twisted.internet import reactor timerFunction(reactor) #3) For subprocesses, I like to create a custom protocol for each type of sub-command I am calling. I also like to create an object to manage the process, its arguments, its results and its temp files. The idiom below is suitable for calling a subprocess that accepts a small amount of buffered data on stdin, produces some output on stdout, and logs its stderr. Be careful examining the value of reason.value.exitCode in processExited. The twisted docs show printing the exitCode as a "%d", but sometimes the value is None --- if the process was terminated by a signal. The mere printing of the value with "%d" will then trigger an exception! Here's my idiom: class FooprocProtocol(protocol.ProcessProtocol): def __init__(self, foomgr): # the object managing my subprocess self.foomgr = foomgr # my stdout data self.data = "" def connectionMade(self): # Pump input data in using this, and then close stdin log.msg("connectionMade!") # self.transport.write("...") # if there is any data to shove into stdin self.transport.closeStdin() def outReceived(self, data): # collect up our stdout log.msg("outReceived! with %d bytes!" % len(data)) self.data = self.data + data def errReceived(self, data): # echo stderr messages to log with a marker log.msg(">%s" % data) def inConnectionLost(self): print "inConnectionLost! stdin is closed! (we probably did it)" def outConnectionLost(self): log.msg("outConnectionLost! The child closed their stdout!") def errConnectionLost(self): log.msg("errConnectionLost! The child closed their stderr.") def processExited(self, reason): log.msg("processExited:%s:" % reason) exitcode = reason.value.exitCode # an integer or None # do some work upon processExit potentially make a decision on exitcode ... log.msg("processExited:%s" % exitcode) def processEnded(self, reason): print "processEnded, status %s" % (reason.value.exitCode,) # process the data in the process manager exitcode = reason.value.exitCode # might be non-numeric result = self.foomgr.processData(exitcode) # The main job of the Process Manager is to build the command list and # process the results. It gives us a handy place to encapsulate this # logic. class FooprocManager(object): CMD = "/usr/local/foocmd" def __init__(self, arg1, arg2, arg3) # create a Deferred to fire when we succeed or fail self.d = Deferred() # build our command argument list as appropriate for our command self.cmdargs = self.build_cmd_args(arg1, arg2, arg3) # define places to store the transport, pid and other things self.ptransport = None self.pid = None def build_cmd_args(self, arg1, arg2, arg3): # in my projects, this method has become fairly involved as it creates # tmp files and builds potentially complicated argument lists. arglist = [self.CMD, arg1, arg2, arg3] return arglist def run(self): # instantiate a protocol connected to this manager pp = FooprocProtocol(self) # spawn the process, save the PID self.ptransport = reactor.spawnProcess(pp, self.CMD, self.cmdargs, { }) self.pid = self.ptransport.pid def processData(exitcode): # in my projects, this method opens up result files, parses results, # moves things around, deletes tmp files, etc. # return the result that we ran this subprocess for return result # Instantiate a new process manager and run it this way. mgr = FooprocManager(args ...) d = mgr.run(args ...) ================= T
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Phil - Thanks so much. This is pretty much exactly what I was looking for. I tried the approach you suggested for timer events, and it seems very clean and works quite well. I was able to get it working very consistently with intervals < 10ms (way more than I need). I haven't tried the process management stuff yet, but it looks very straightforward. Thanks again. On Wed, Aug 28, 2013 at 6:54 AM, Phil Mayers <p.mayers@imperial.ac.uk>wrote:
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Phil - I am aware of LoopingCall. I saw the mention of cooperate/coiterate on some page somewhere, and actually went off trying to find an explanation / example but did not find anything. If you have a link with some details, I would like to take a look. Thanks On Wed, Aug 28, 2013 at 6:54 AM, Phil Mayers <p.mayers@imperial.ac.uk>wrote:
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Tom - Just wanted to thank you for your help. I got everything ported over to Twisted on Friday, and the responsiveness of the web site is 10 times better. Also the number of lines of (my) code declined, which is great. Thanks much. On Wed, Aug 28, 2013 at 6:40 AM, Tom Sheffler <tom.sheffler@gmail.com>wrote:
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Phil - Thanks so much. This is pretty much exactly what I was looking for. I tried the approach you suggested for timer events, and it seems very clean and works quite well. I was able to get it working very consistently with intervals < 10ms (way more than I need). I haven't tried the process management stuff yet, but it looks very straightforward. Thanks again. On Wed, Aug 28, 2013 at 6:54 AM, Phil Mayers <p.mayers@imperial.ac.uk>wrote:
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Phil - I am aware of LoopingCall. I saw the mention of cooperate/coiterate on some page somewhere, and actually went off trying to find an explanation / example but did not find anything. If you have a link with some details, I would like to take a look. Thanks On Wed, Aug 28, 2013 at 6:54 AM, Phil Mayers <p.mayers@imperial.ac.uk>wrote:
![](https://secure.gravatar.com/avatar/5d85036af9760dc6f59c23a6587f36b7.jpg?s=120&d=mm&r=g)
Tom - Just wanted to thank you for your help. I got everything ported over to Twisted on Friday, and the responsiveness of the web site is 10 times better. Also the number of lines of (my) code declined, which is great. Thanks much. On Wed, Aug 28, 2013 at 6:40 AM, Tom Sheffler <tom.sheffler@gmail.com>wrote:
participants (3)
-
Matthew Humphrey
-
Phil Mayers
-
Tom Sheffler