Mailman 3 October 2007 - Twisted

[Twisted-Python] Returning a Deferred as a result from another Deferred
by Ryan Fugger 06 Oct '07

06 Oct '07

Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback. Maybe someone can suggest a better approach. My application takes a command from one connection and defers processing to a worker thread to permit potentially-blocking database access. That worker thread often will need to send data over another connection to fulfill the command, and await a reply before the reply can be sent back over the original command connection. The worker thread sends data over the second connection using callFromThread, and this send operation itself returns a Deferred which fires when the reply is received. It is this Deferred that I want to return as a result from the original Deferred created by deferring to the worker thread in the command handler, so I can wait until the reply comes from the second connection, and then reply on the original command connection. Can someone suggest a better approach? Thanks. Ryan

4 7

[Twisted-Python] Collecting all results from a DeferredList
by Stefan Arentz 04 Oct '07

04 Oct '07

I want to request many web pages through a DeferredList and then combine the results in an array. I don't understand how to keep an array as shared state between these requests and return it to the original caller when the DeferredList has completed. S.

1 1

[Twisted-Python] LoopingCall vs internet.TimerService
by Saptarshi Guha 03 Oct '07

03 Oct '07

Hello, Thanks for twisted - very useful. Now, my questions I have written a server which accepts connections from clients, takes requests and adds them to a Queue (a python object of Queue.Queue). Now i have two approaches a) At startup, my server(a subclass of pb.Root) runs a "dispatcher" function using task.LoopingCall(dispatcher).start(0.1) b)Create a separate subclass of internet.TimerService whose function is dispatcher [INFO: dispatcher has methods like obj.callRemote, addCallback and addErrback] So which is better, neater, safer? loopingcall of timerservice? also is it safe to do the calls mentioned in INFO within dispatcher? Thank you for your time and help. Saptarshi Guha Saptarshi Guha sapsi(a)pobox.com

3 3

[Twisted-Python] Discovering Twisted : reactor.stop()
by Steve MULLER 03 Oct '07

03 Oct '07

Hello, I discovered Python recently and even more recently the Twisted module. I have a problem with the Twisted reactor and I would like to understand for which reason my program freezes. Here the code which should loop forever : from twisted.internet import reactor from twisted.internet import task class reactorProb(object): def __init__(self): self.count = 5 self._callback = task.LoopingCall(self.Heartbeat) self._callback.start(1, now = False) def Heartbeat(self): print "Top" self.count -= 1 if (self.count == 0): print "Exit" self._callback.stop() del self._callback reactor.stop() while (True): rp = reactorProb() print "Enter into reactor.run()" reactor.run() print "Leave from reactor.run()" I got this: Enter into reactor.run() Top Top Top Top Top Exit Leave from reactor.run() Enter into reactor.run() Top Top Top Top Top Exit Thus I start first once the reactor, I stop it then I start again and it freezes instead of stopping it the reactor. Could somebody explain me it why? Thank you in advance. smu

4 6

[Twisted-Python] Project design questions
by Paul_S_Johnson＠mnb.uscourts.gov 02 Oct '07

02 Oct '07

Here's a description of my general game plan and my questions are below: THE MISSION Create a GUI-based, dashboard-like application that monitors the overall health of our resources. It would be a tool that helps pinpoint trouble spots when performance is less than desirable. Ideally, we would get such a good handle on it that it would warn us of impending trouble before it causes a failure. SAMPLE FUNCTIONS FOR THE APPLICATION Monitor web traffic by distilling down Apache logs Monitor server hardware through built in firmware http interface Monitor server health via command line commands: disk space processor load etc. Monitor database health via command line commands Monitor backup success through log output Monitor misc. server messages Monitor switch traffic Monitor processes ...And more. POSSIBLE CONSTRAINTS Our main servers are on a WAN whose security is the job of the greater org. We have access through SSL connections now, but we may not be able to get permissions to open any additional ports for our monitoring purposes. THE QUESTIONS Very basic question: am I better off leaving most of the heavy lifting (as in the programming logic) on the client side? This would mean more data would have to pass from server to client app (e.g. a whole web log or part thereof) which is the downside. The upside is that in a failure emergency situation if my server is still up enough to have a SSH connection my dashboard app may still be useful to me. Thoughts? If we are not constrained to only get our info via SSH what's the best kind of connection to have? My thought is to use the Prospective Broker since that seems to be the most Pythonistic option. Is it possible to encrypt the PB server connection? Are there any Twisted methods of tunneling through an existing SSH connection (just a wild thought)? Thanks, Paul

3 2

[Twisted-Python] reactor.spawnProcess() - differences between Python & Bash deaths
by Brendan Boerner 01 Oct '07

01 Oct '07

Hi, I'm seeing (or not seeing :-) ) behavior which I cannot explain. Attached are two .py scripts and a bash script - echo.bash - echo a message echo.py - echo a message py_spawn.py - launch either echo.bash or echo.py using reactor.spawnProcess(). If I have py_spawn.py launch echo.bash and then kill the '/bin/bash ./echo.bash' process using a SIGTERM py_spawn reports: error: errorMessage: A process has ended with a probable error condition: process ended by signal 15. error: exitCode: None error: signal: 15 error: status: 15 processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessTerminated'>> Which is what I expect. If instead I launch echo.py and then kill the '/opt/local/bin/python2.5 ./echo.py -m message' process py_spawn reports: error: exitCode: 0 error: signal: None error: status: 0 processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessDone'>> which is not what I expect. In particular I am 99% certain that this is relatively recent behaviour which showed up only last week - prior to that I was seeing that if I killed a spawned .py process that I'd see that it was terminated with a signal 15 ala the bash case. I've tested this on Mac and Debian boxes with identical results. Any suggestions or insights are appreciated. Regards, Brendan --- py_spawn.py --- #! /opt/local/bin/python2.5 from twisted.internet import error, reactor, defer, protocol import getopt import pickle import pprint import subprocess import signal import sys Dumper = pprint.pformat class MyProcessProtocol(protocol.ProcessProtocol): def __init__(self): self.outdata = "" self.errdata = "" pass def connectionMade(self): print "connectionMade!" def outReceived(self, data): print "outReceived! with %d bytes!" % len(data) self.outdata = self.outdata + data def errReceived(self, data): print "errReceived! with %d bytes!" % len(data) self.errdata = self.errdata + data def inConnectionLost(self): print "inConnectionLost! stdin is closed! (we probably did it)" def outConnectionLost(self): print "outConnectionLost! The child closed their stdout!" print "I saw them write:\n", self.outdata def errConnectionLost(self): print "errConnectionLost! The child closed their stderr." print "I saw them write:\n", self.errdata def processEnded(self, status): print "debug: type(status): %s" % type(status.value) print "debug: isinstance(status.value, error.ProcessDone): % s" % \ isinstance(status.value, error.ProcessDone) print "debug: isinstance(status.value, error.ProcessTerminated): %s" % \ isinstance(status.value, error.ProcessTerminated) if isinstance(status.value, error.ProcessDone): print "error: exitCode: %s" % status.value.exitCode print "error: signal: %s" % status.value.signal print "error: status: %s" % status.value.status elif isinstance(status.value, error.ProcessTerminated): print "error: errorMessage: %s" % status.getErrorMessage() print "error: exitCode: %s" % status.value.exitCode print "error: signal: %s" % status.value.signal print "error: status: %s" % status.value.status print "processEnded, status %s" % repr(status) print "quitting" reactor.stop() def usage(): print "Usage: py_spawn.py" def main(): processProtocol = MyProcessProtocol() # killing the bash process using SIGTERM results in: # error: errorMessage: A process has ended with a probable error condition: process ended by signal 15. # error: exitCode: None # error: signal: 15 # error: status: 15 # processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessTerminated'>> reactor.spawnProcess(processProtocol, 'bash', ["bash", "-c", "echo.bash"], env=None) # killing the bash process using SIGTERM results in: # error: exitCode: 0 # error: signal: None # error: status: 0 # processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessDone'>> #reactor.spawnProcess(processProtocol, 'bash', ["bash", "-c", "./ echo.py -m message"], env=None) reactor.run() if __name__ == "__main__": main() --- echo.py --- #!/opt/local/bin/python2.5 from optparse import OptionParser, OptParseError from twisted.internet import reactor from twisted.spread import pb from zope.interface import implements import pprint import sys Dumper = pprint.pformat def run(duration, message): duration -= 1 print "duration: %d, message: %s" % (duration, message) if duration != 0: reactor.callLater(1, run, duration, message) else: reactor.stop() def main(): print "debug: args: %s" % sys.argv[1:] try: duration = 0 message = "No message" parser = OptionParser() parser.add_option("-d", "--duration=", dest="duration", metavar="INT", default=duration, help="How long to run, 0 will run forever [default: % default]") parser.add_option("-m", "--message=", dest="message", default=message, help="Message to display [default: %default]") (opts, args) = parser.parse_args() duration = int(opts.duration) message = opts.message print "debug: opts: %s, args: %s" % (opts, args) except OptParseError, e: # print help information and exit: print "exception: %s" % e sys.exit(2) reactor.callLater(0, run, duration, message) reactor.run() if __name__ == "__main__": main() --- echo.bash --- #!/bin/bash while true; do echo -n "This is a message "; date; sleep 1; done

1 0

[Twisted-Python] Aborting callback chain with an errback
by Andy Fundinger 01 Oct '07

01 Oct '07

What is the proper way to handle an uncorrectable error? I have code that looks like: def callXMLRPC(self,stringValue,intValue): return(xmlProxy.callRemote("myFunc",{...}) \ .addCallback(self.rpcSuccess).addErrback(self.rpcFailure)) and in a derived class: def specificQuery(self): return self.callXMLRPC("Ask Thing", 3).addCallback(self.handleResult) what I'd like to have happen is if an error is raised in callRemote, rpcFailure does some cleanup and no further callbacks are executed. I can't fix the failure, but I don't want a message logged into my error log everytime, it's not unhandled, just that there's nothing to be done and callbacks won't have the data to run. If I want to deal with it further I'd want to do it in the base class. How can I do that? Am I doing something nonsensical here? Andy Fundinger -- Blog: http://channel3b.wordpress.com Second Life Name: Ciemaar Flintoff

2 1