Mailman 3 August 2010 - Twisted

[Twisted-Python] Deferred's chainDeferred is too simplistic?
by Terry Jones 14 Aug '10

14 Aug '10

I think it's worth spending some time thinking about whether chainDeferred in t.i.defer.Deferred is too simplistic. I've thought for a while that it could be more helpful in preventing people from doing unintended things and/or cause fewer surprises (e.g., see point #1 at http://bit.ly/bp6iT5 which JP agreed with in the followup). Those are smaller things that people can obviously live with. But a more important one concerns deferred cancellation. See uncalled.py below. Here are five examples of chainDeferred behavior that I think could be better. In an attempt to fix these, I made a few changes to a copy of defer.py (immodestly called tdefer.py, sorry) which you can grab, with runnable examples, from http://github.com/fluidinfo/chainDeferredExamples The tdefer.py code is meant as a suggested approach. I doubt that it's bulletproof. Here are the examples. boom1.py: # Normal deferreds: this raises defer.AlreadyCalledError because # the callback of d1 causes the callback of d2 to be called, but d2 has # already been cancelled (and hence called). # With tdefer.py: there is no error because d1.callback will not call # d2 as it has already been cancelled. def printCancel(fail): fail.trap(defer.CancelledError) print 'cancelled' def canceller(d): print 'cancelling' d1 = defer.Deferred() d2 = defer.Deferred(canceller) d2.addErrback(printCancel) d1.chainDeferred(d2) d2.cancel() d1.callback('hey') boom2.py: # Normally: raises defer.AlreadyCalledError because calling d1.callback # will call d2, which has already been called. # With tdefer.py: Raises AssertionError: "Can't callback an already # chained deferred" because calling callback on a deferred that's # already been chained is asking for trouble (as above). d1 = defer.Deferred() d2 = defer.Deferred() d1.chainDeferred(d2) d2.callback('hey') d1.callback('jude') uncalled.py: # Normally: although d2 has been chained to d1, when d1 is cancelled, # d2's cancel method is never called. Even calling d2.cancel ourselves # after the call to d1.cancel has no effect, as d2 has already been # called. # With tdefer: both cancel1 and cancel2 are called when d1.cancel is # called. The additional final call to d2.cancel correctly has no # effect as d2 has been called (via d1.cancel). def cancel1(d): print 'cancel one' def cancel2(d): print 'cancel two' def reportCancel(fail, which): fail.trap(defer.CancelledError) print 'cancelled', which d1 = defer.Deferred(cancel1) d1.addErrback(reportCancel, 'one') d2 = defer.Deferred(cancel2) d2.addErrback(reportCancel, 'two') d1.chainDeferred(d2) d1.cancel() d2.cancel() unexpected1.py: # Normally: prints "called: None", instead of the probably expected # "called: hey" # tdefer.py: prints "called: hey" def called(result): print 'called:', result d1 = defer.Deferred() d2 = defer.Deferred() d1.chainDeferred(d2) d1.addCallback(called) d1.callback('hey') unexpected2.py: # Normally: prints # called 2: hey # called 3: None # tdefer.py: prints # called 2: hey # called 3: hey def report2(result): print 'called 2:', result def report3(result): print 'called 3:', result d1 = defer.Deferred() d2 = defer.Deferred().addCallback(report2) d3 = defer.Deferred().addCallback(report3) d1.chainDeferred(d2) d1.chainDeferred(d3) d1.callback('hey') I wont go into detail here as this post is already long enough. Those are 3 classes of behavior arising from chainDeferred being very simplistic. Comments welcome, of course. Once again, runnable code is at http://github.com/fluidinfo/chainDeferredExamples Terry

1 0

[Twisted-Python] How to bisect Twisted?
by Peter Westlake 12 Aug '10

12 Aug '10

In the Git version of the Twisted source code, there are tags for releases, e.g. the revision with SHA id 85a6a0f9e3c4ea22d1d72d4ca8eb0c08ee99ca85 corresponds to 8.0.0. This is good, but there aren't any intermediate changes on that branch (tag?), just the releases! So there isn't much for "git bisect" to work with. I'm trying to trace a bug that wasn't in 2.5.0 but is in 8.0.0; what arguments should I give to bisect? Apologies if this is obvious to anyone who knows how to use git, but I'm just a beginner. Thanks, Peter.

3 4

[Twisted-Python] global exception handler
by Zoran Bosnjak 12 Aug '10

12 Aug '10

Hello all, is there a possibility to have a global (application wide) exception handler in twisted application? I would like to stop application in case of any unhandled error in deferred callbacks. If application is run from twistd ..., is it correct to stop application like this? def quit(failure): log.err(str(failure)) reactor.callWhenRunning(reactor.stop) But the main problem is how to get this method called if exception is not handled by explicitly doing d.addErrback for each case. Zoran

2 1

[Twisted-Python] Using cProfile instead of hotshot with twistd
by Jason J. W. Williams 12 Aug '10

12 Aug '10

Hi, I'm having an issue using twistd with --profiler=cprofile. I was using hotshot, but there's a bug in loading big profile logs with it so I'm trying cProfile instead. The issue is that with --profiler=cprofile, the --profile=<filename> flag seems to be ignored. That is no profile file gets created. Any help is greatly appreciated. -J

3 5

[Twisted-Python] Running tests in parallel with trial
by Free Ekanayaka 10 Aug '10

10 Aug '10

Hi, is there a way to tell trial run groups of tests in parallel in different processes? Something like the "-jN" option in zope-testrunner. Or alternatively, what's the recommended way to make better use of multi-core CPUs when running a test suite with trial? Thanks, Free

3 3

[Twisted-Python] xmlrpc protocoll instanze per connection?
by Pet 09 Aug '10

09 Aug '10

Hello, I've started learning twisted by implementing linereceiver, where for each new connection a new instance of MyProtocol is created: class MyFactory(protocol.ServerFactory): protocol = MyProtocol def __init__(self): self.db = adbapi.ConnectionPool() class MyService(internet.TCPServer): def __init__(self): print "Starting Service..." internet.TCPServer.__init__(self, PORT, MyFactory()) def main(): reactor.listenTCP(settings.PORT, MyFactory()) reactor.run() if __name__ == '__main__': main() Now, for XMLRPC this looks for me like Protocol instance is reused for all connection. If it is so, can I change this behavior, so I have one instance of Protocol per connection? Thanks for help! Pet class MyXMLRPCFactory(server.Site): def __init__(self): self.db = adbapi.ConnectionPool() self.r = MyXMLRPCProtokoll(self) server.Site.__init__(self,self.r) class MyXMLRPCService(internet.TCPServer): def __init__(self): internet.TCPServer.__init__(self, settings.XMLRPC_PORT, MyXMLRPCFactory()) def main(): from twisted.internet import reactor r = MyXMLRPCProtokoll() reactor.listenTCP(settings.XMLRPC_PORT, server.Site(r)) reactor.run() if __name__ == '__main__': main()

4 13

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 08 Aug '10

08 Aug '10

1 0

Re: [Twisted-Python] Question about XMLRPC, Sync and ASync
by Andrew Francis 08 Aug '10

08 Aug '10

Hi Colin: > Message: 1 > Date: Fri, 06 Aug 2010 11:02:33 -0700 > From: Colin Fox <greenenergy(a)gmail.com> > Subject: [Twisted-Python] Question about XMLRPC, Sync and > ASync > To: twisted-python(a)twistedmatrix.com > Message-ID: <4C5C4E39.6080807(a)gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > What I'm trying to do now has me a little stumped -- I have > two servers, one of which answers XMLRPC requests, and it calls another > one using the twisted basic.LineReceiver system. I am a little confused about the basic.LineReceiver because I thought that was used for creating server protocols? > So I have an incoming XMLRPC request, and I need to issue a > command to the other server and wait for it to come back before I > respond to the xmlrpc call. So I'm trying to do an async call inside a > synchronous call. I'm a bit rusty but lets use this simple example of a XMLRPC server making another XMLRPC request, processing the data and returning it: class MyXMLRpcServer(xmlrpc.XMLRPC): def __init__(self): self.service = xmlrpc.Proxy("http://www.dude.com") def xmlrpc_someFunction(self, *argv, **kwargs): .... return self.service.callRemote(someFunction).\ addCallback(processResult) self.service.callRemote returns a deferred. Returning a deferred tells the reactor that someFunction() is not finished processing and to call processResult when there is data from www.dude.com eventually processResult(data) is called def processResult(data) """ do something with data """ return result This should return a result to the XMLRPC client. Cheers, Andrew

2 1

[Twisted-Python] Question about XMLRPC, Sync and ASync
by Colin Fox 07 Aug '10

07 Aug '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everyone. I'm relatively new to Twisted, though I have used parts of it for a while. What I'm trying to do now has me a little stumped -- I have two servers, one of which answers XMLRPC requests, and it calls another one using the twisted basic.LineReceiver system. So I have an incoming XMLRPC request, and I need to issue a command to the other server and wait for it to come back before I respond to the xmlrpc call. So I'm trying to do an async call inside a synchronous call. I think that deferreds will play a part here, but I guess I must be missing something, as I'm not sure how to proceed. Can I return a deferred from my XMLRPC handler? And if so, how do I fire the callback when the result comes in to the LineReceiver? Thanks, Colin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJMXE45AAoJEKVzdTf0DlfQ+vIH/RJFfIlVfUAkNXFER12eoY7E TgtXYP4nLBMsBbaPUy7LROrfCl2gdxPdy/q2DI7JHueG2eUznW5IdG4Zipe6RFS/ xCAUebs7E2Cdkc0s+i+6zCvCZQvmUV20tunxyheuwc26PxPT0HEZGpGJhhKOGqGL tYGr3OXAn7CEiAxwX3XMs4nMRVLAalMnKgsDLIRx4jZkaitvAQb/ufuJWmzf2+u2 dgbPREllAze3LrGsPFjCUdJToAE3AVzYIMbGQr/n/TsUK3v+hbAky2Xu8xuAkUgF 4y/5THaPw2EFKJsZNFhSY7b6hHbO+gr0VzRkx92sPKy4GxnxY7Br80n6A76LBoc= =mRxr -----END PGP SIGNATURE-----

2 1

[Twisted-Python] strange bug with... something :)
by Сергей Магафуров 06 Aug '10

06 Aug '10

Example code 1 {{{ class Example(object): def __init__(self, connection): self.connection = connection self.counter = 0 self.requests = {} def generate_request_id(self): self.counter += 1 return self.counter def request(self, data, timeout=10): deferred = defer.Deferred() request_id = self.generate_request_id() self.requests[request_id] = deferred # ATTENTION: I want to see current traceback when timeout occurs # therefore raise and catch here try: raise failure.TimeoutError() except failure.TimeoutError: failure = failure.Failure() # charge timer timer = reactor.callLater(timeout, self._on_timeout, request_id, failure) # send request self.connection.send_request(request_id, data) return deferred def _on_timeout(self, request_id, failure): deferred = self.requests.pop(request_id, None) if deferred is not None: deferred.errback(failure) def on_response_received(self, request_id, response): deferred = self.requests.pop(request_id, None) if deferred is not None: deferred.callback(response) }}} Disadvantage 1 of this code: We construct failure.Failure() always although timeout may not occur at all. If failure.Failure() was lazy it will be cool. Disadvantage 2 (bug in twisted?) of this code: This code doesn't work in some cases and I don't know why. In some cases deferred.callback(response) does nothing although called. But this code magically works properly in the same cases: Example code 2 (one difference is just "lazy" failure construction using special FailureFactory class) {{{ class FailureFactory(object): def __init__(self, arg): try: raise arg except: exc_info = sys.exc_info() self.exc_info = exc_info def __call__(self): exc_type, exc_value, tb = self.exc_info return Failure(exc_value, exc_type, tb) class Example(object): def __init__(self, connection): self.connection = connection self.counter = 0 self.requests = {} def generate_request_id(self): self.counter += 1 return self.counter def request(self, data, timeout=10): deferred = defer.Deferred() request_id = self.generate_request_id() self.requests[request_id] = deferred # ATTENTION: I want to see current traceback when timeout occurs # therefore raise and catch here failure_factory = FailureFactory(failure.TimeoutError()) # charge timer timer = reactor.callLater(timeout, self._on_timeout, request_id, failure_factory) # send request self.connection.send_request(request_id, data) return deferred def _on_timeout(self, request_id, failure_factory): deferred = self.requests.pop(request_id, None) if deferred is not None: deferred.errback(failure_factory()) def on_response_received(self, request_id, response): deferred = self.requests.pop(request_id, None) if deferred is not None: deferred.callback(response) }}} Example 2 works but Example 1 does not. What is wrong with example 1? Bug in twisted? I have posted this message as ticket but was redirected to mailing-list http://twistedmatrix.com/trac/ticket/4628

2 1