Mailman 3 December 2008 - Twisted

[Twisted-Python] lost tracebacks?
by Alec Flett 10 Dec '08

10 Dec '08

So I am constantly "losing" my tracebacks when my defer's callbacks throw exceptions that aren't caught. When I get an uncaught exception (caught via twisted.log.addObserver()) I am basically storing the Failure instance and then later, when it's safe for me to do so, calling fail.raiseException() - but the Failure's 'tb' property is None. I traced this down to the code in Deferred._runCallbacks() which does a try: / except: with:: self.result = failure.Failure() Seems like this code should say:: exc_type, exc_value, exc_traceback = sys.exc_info() self.result = failure.Failure(exc_value, exc_type, exc_traceback) I'm dealing with two issues: 1) trying to generically log stack traces that happen 2) provide sane stack traces to my team who is fairly new to twisted I can see how this might possibly leak if the tracebacks are holding onto references to local variables or something, but maybe cleanFailure() fixes that? I don't know. Alec

3 4

[Twisted-Python] spawnProcess help needed
by Phil Christensen 10 Dec '08

10 Dec '08

Hello All- I'm running into an issue with a project I'm working on that involves coordinating a large number of external processes to encode audio into different formats, and do other bookkeeping. I seem to be having a problem when I use spawnProcess to call the md5sum binary. The processes spawned seem to stick around forever, until I kill the server. This doesn't appear to happen with any other binary. I start with the following abstract class: class DefaultProcessProtocol(protocol.ProcessProtocol): def __init__(self, preserveOut=True, preserveErr=True): self.deferred = defer.Deferred() self.debug = False self.preserveOut = preserveOut self.out = '' self.preserveErr = preserveErr self.err = '' def connectionMade(self): if(self.debug): print '%s connection made' % self.__class__.__name__ def outReceived(self, data): if(self.debug): print '%s stdout: %s' % (self.__class__.__name__, data) if(self.preserveOut): self.out += data def errReceived(self, data): if(self.debug): print '%s stderr: %s' % (self.__class__.__name__, data) if(self.preserveErr): self.err += data def processEnded(self, status): if(isinstance(status.value, error.ProcessDone)): self.deferred.callback(self) else: if(self.out): print self.out if(self.err): print self.err self.deferred.errback(status.value) Then I made a subclass as follows: class MD5Checksum(DefaultProcessProtocol): @staticmethod def run(target, md5_path='/usr/bin/md5sum'): hasher = MD5Checksum(preserveOut=True) hasher.debug = True reactor.spawnProcess(hasher, md5_path, [md5_path, target]) return hasher.deferred def get_checksum(self): filehash = self.out if(filehash.find('=') == -1): filehash = [output.strip() for output in filehash.split(' ')][0] else: filehash = [output.strip() for output in filehash.split('=')][1] return filehash With the plan of executing it inside an inlineCallbacks-decorated function, like this: proto = yield process.MD5Checksum.run(filename) checksum = proto.get_checksum() Everything is basically working, except for the fact that the md5sum processes never go away. It does seem that processEnded is being called, since that's what issues the callback on the deferred returned by run(), but the process sticks around, as a non-zombie, non-defunct process that still appears to be using small amounts of CPU time. Also, once the checksum has been retrieved, killing the server doesn't cause any errors, unlike when a process is legitimately terminated by killing the server, which displays an error (since right now i'm not actually catching ProcessTerminated scenarios): 2008-12-09 15:17:48-0500 [-] Unhandled error in Deferred: 2008-12-09 15:17:48-0500 [-] Unhandled Error Traceback (most recent call last): Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended by signal 2. Any help in this matter would be appreciated... -phil

3 3

[Twisted-Python] wxpython and xmlrpc via twisted
by Ian Jones 10 Dec '08

10 Dec '08

I'm trying to create a taskbar app (in wxpython) to monitor job progress on a render farm. I'd like the taskbar app to have an xmlrpc server built in so that other applications can inform the taskbar app additional jobs to monitor. I found this recipe online which is a great example of the functionality I want: http://code.activestate.com/recipes/298985/ However when I run the following from xmlrpclib import * s = ServerProxy('http://localhost:7080') # Get access to xmlrpc server s.title('Hallo xmlrpc app') I don't get a response (though the title in the window changes - so I know it got it) I do get this though: Traceback (most recent call last): File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\protocols\basic.py", line 239, in dataReceived return self.rawDataReceived(data) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\http.py", line 1117, in rawDataReceived self.allContentReceived() File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\http.py", line 1108, in allContentReceived req.requestReceived(command, path, version) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\http.py", line 626, in requestReceived self.process() --- <exception caught here> --- File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\server.py", line 150, in process self.render(resrc) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\server.py", line 157, in render body = resrc.render(self) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\resource.py", line 189, in render return m(request) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\xmlrpc.py", line 122, in render_POST function = self._getFunction(functionPath) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\xmlrpc.py", line 170, in _getFunction handler = self.getSubHandler(prefix) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\xmlrpc.py", line 107, in getSubHandler return self.subHandlers.get(prefix, None) exceptions.AttributeError: 'MyXMLRPCApp' object has no attribute 'subHandlers' Unhandled error in Deferred: Traceback (most recent call last): File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\resource.py", line 189, in render return m(request) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\xmlrpc.py", line 126, in render_POST defer.maybeDeferred(function, *args).addErrback( File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\internet\defer.py", line 195, in addCallback callbackKeywords=kw) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\internet\defer.py", line 186, in addCallbacks self._runCallbacks() --- <exception caught here> --- File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\internet\defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "d:\toolkit\sww\win32\python\2.4\lib\site-packages\twisted-8.1.0-py2.4-win32.egg\twisted\web\xmlrpc.py", line 144, in _cbRender allow_none=self.allowNone) exceptions.AttributeError: 'MyXMLRPCApp' object has no attribute 'allowNone' I'm assuming this is just a good example that is out of date. However I'm not sure how to update it so that it works. Can someone help me out?

2 4

[Twisted-Python] Deferred question
by Gabriel Rossetti 10 Dec '08

10 Dec '08

Hello everyone, I have a question concerning defers, I have a service that needs to query IO until it answers. The IO can be disconnected, connected but off, connected and on; depending on the state I get different exceptions. I wrote something like : def __requestId(self, _): d = self.callPlugin(PLUGIN_NAME) d.addCallback(self.__receivedAns) d.addErrback(self.__requestId) The plugin makes the request in a thread and the deferred is returned and called when either an answer is received or am exception is raised (e.g. timeout, missing hardware, etc) This seams to work well, except that when I let the system run for a long time without the hardware being present or off (several hours/days) the memory consumption goes way up. I don' t really know what is causing this, and I'm looking for possible reasons, so I was wondering if maybe this could be what is causing the problem. It's probably not that, because from what I understand it works like this : 1. the first request is made 2. there is a timeout or something that raises the exception 3. the errback gets called 4. it makes a new request 5. the deferred gets garbage collected 6. the new request follows the same path Can anyone confirm that this is correct, or are my defers somehow staying alive and not being garbage collected? Thank you, Gabriel Rossetti

2 3

[Twisted-Python] How to see the number of received msgs?
by Gabriel Rossetti 10 Dec '08

10 Dec '08

Hello eveyone, is there a way to see how many msgs have been received and are waiting to be processed by a client? Thank you, Gabriel

3 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 07 Dec '08

07 Dec '08

1 0

Re: [Twisted-Python] Re: A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones 06 Dec '08

06 Dec '08

Hi Nicola! >>>>> "Nicola" == Nicola Larosa <nico(a)teknico.net> writes: Nicola> Terry Jones wrote: >> BTW, I spent an interesting morning getting my head around *exactly* how >> inlineCallbacks does its thing... No summary of my thoughts could do it >> justice. I wonder how many people on the planet have been down that >> rabbit hole. Nicola> Not me, man, not me. I just use the bloody thing, and curse it when Nicola> it shortens tracebacks. I know exactly what you're talking about, and I hated that too. But I dug into that and found out where the problem was: I didn't know Python well enough :-) I bet you're doing something like this: @inlineCallbacks def f(): # Whatever try: # Your code here except Exception, e: # Do some cleaning up, maybe write to the log, maybe call another # function that uses inlineCallbacks raise e or just @inlineCallbacks def f(): # Whatever try: # Your code here except Exception: # Do some cleaning up, maybe write to the log, maybe call another # function that uses inlineCallbacks raise The problem here is that you're not using the 3-argument form of raise. So you're either dropping information (with the raise e) or pulling out the last exception (with the no-arg raise). Consider the latter case. Suppose your cleanup code calls another function that uses inlineCallbacks. That function will eventually raise StopIteration or raise via calling defer.returnValue. So by the time control returns to your code, your raise (which implicitly uses sys.exc_info()) raises the wrong exception! So you get e.g., a StopIteration exception coming at you out of your inlineCallbacks function and you curse the thing :-) This applies if your cleanup code calls any iterator, or if you call something that try/except catches a KeyError, etc. So just use @inlineCallbacks def f(): # Whatever try: # Your code here except Exception: info = sys.exc_info() # Do some cleaning up, maybe write to the log, maybe call another # function that uses inlineCallbacks raise info[0], info[1], info[2] and it just works. Try it :-) You cant do anything nice like raise *info because Python's parser doesn't like that (AFAIR). I nearly posted about this to the list, but I figured it might help if I had more of a clue about Python itself, seeing as what I was doing was nothing to do with Twisted. >> I hope people don't mind my peanut gallery commentary. We're all in this >> together, after all. A little gallows humor occasionally seems in order. Nicola> Again, certainly not me. Go right ahead! ;-) Don't encourage me... Terry

1 0

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones 06 Dec '08

06 Dec '08

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On 02:33 am, terry(a)jon.es wrote: >> Am I missing something here? glyph> As a matter of fact, yes :). OK, thanks, I get it. That should have been obvious. BTW, I spent an interesting morning getting my head around *exactly* how inlineCallbacks does its thing... No summary of my thoughts could do it justice. I wonder how many people on the planet have been down that rabbit hole. Was the Twisted project always called Twisted? I was thinking it's a really good thing you picked that name - cause you definitely would have had to rename it :-) I hope people don't mind my peanut gallery commentary. We're all in this together, after all. A little gallows humor occasionally seems in order. Thanks again, Terry

2 1

[Twisted-Python] one-shot reactor?
by Alec Flett 05 Dec '08

05 Dec '08

Hey folks - So I'm trying what seems to be a fairly unusual use of Twisted, but I'm hoping that someone out there has tried the same thing as me and can offer some pointers. Bear with me as I explain how we're set up. The issues I'm having are at the end.. We're running Pylons as our appserver, but almost all of our internal requests from the appserver are actually retrieved over HTTP - i.e. database requests, LOBs, etc... we also have one or two other proprietary connections that would benefit from asynchronous TCP access.. For years I've heard people raving about twisted, and I hate threads, so I thought I'd give it a shot. So here's how this is working, at least in my early prototypes: Pylons is a WSGI environment, which means it needs a real callstack for each request. Each request is more or less its own single-threaded environment, and the thread is only making new requests to internal services, not listening on any ports. So really Twisted is a client here, not a server. For each HTTP request, I'm running a "private" twisted reactor that simply runs until it runs out of reads/writes/delayedCalls. My theory is this: in a single-threaded environment that is not listening for new connections. So what I've done is make twisted.internet.reactor into a threadlocal object with Paste's StackedObjectProxy. So far I have this mostly working, but I've hit a few stumbling blocks along the way: 1) it would be nice if the standard twisted reactors had an API for running in a one-off client mode - something like a reactor.runUntilExhausted(). I did write a kind of scheduler that does this for me (more on that below) 2) it's vaguely annoying that reactors aren't restartable - it means I have to destroy any reactor that's left around from the last request. Not a huge deal, but I'd much rather just create a single reactor that lives the life of the thread, and be able to call .run() / .stop() over and over. 3) I've attached my scheduler below. I hook it up with reactor.callLater(0, stop_when_complete). The problem I'm running into with this approach is that many APIs like getPage() set a 30 second timeout, and then cancel the timeout later when it successfully retrieves the page. My scheduler picks up the fact that there is a 30-second timeout, but because HTTPClientProtocol cancels the timeout, my scheduler isn't aware of that, and has already scheduled itself for 30 seconds into the future, so it can't call reactor.stop(). So instead, I wake up at most every 0.1 seconds - but that kind of defeats the point of the reactor blocking on select()/poll() if I have to keep waking up! Maybe there's a better approach? What I kind of want is a hook into the reactor's runUntilCurrent() so I just get notified right before the select()/poll(). I'm considering just subclassing the reactor to hook into this.. ? def stop_when_complete(reactor, running=False): all_pending = (reactor.getReaders() + reactor.getWriters() + reactor.getDelayedCalls()) # depending on the platform, the waker is probably in there, and # shouldn't count as a pending event if reactor.waker in all_pending: all_pending.remove(reactor.waker) # ok, are we done? if so, tell the reactor to stop after its next # iteration. if not all_pending: reactor.stop() return # if we got here, we need to basically wait again to see if # there's anything left. We get the timeout for the next event, so # that we don't wake any more than we would have (this is how # ReactorBase.mainLoop works) timeout = reactor.timeout() if timeout is None: timeout = 0.0 print "Sleeping for no more than %s seconds" % min(timeout, 0.1) reactor.callLater(min(timeout, 0.1), stop_when_complete, reactor, running=True)

1 0

[Twisted-Python] OSCAR buddy list
by Andrey Kravchuk 05 Dec '08

05 Dec '08

Hi. I am trying to implement an ICQ service using twisted OSCAR protocol. Everything works fine so far, but i can't find out how to handle the buddy list there. I can't find anything on this topic neither in documentation, nor in this list or the source code. Please advice, or if this is not implemented yet, please tell me where to look so i could make a patch. The main problem is if i connect to ICQ service with twisted, the user appears offline for other users. In order to see his status change I need to "request authorization" and the service must "grant authorization". There was a time not so long ago when it was not required, you could just set "authorization not required" in account profile and it worked fine, but now it doesn't. Newly added users appear in my contact list always offline.

1 0