[Twisted-Python] Using deferreds when writing across unreliable network
I have been trying out twisted(2.5 for python 2.4 XPSP2) over the past few weeks and have some code running connecting with some mobile devices over GPRS. It works. But I am having problems with deferreds - which I find tricky to understand. I have setup a tcpserver which the mobile devices connect to and I send answers back and occasionally unsolicited messages out. I have made an assumption that I should use a defer to write back across the network. Maybe I am wrong. I have a function in which I set up the transport.write to send a message out. This I make a deferred and return the deferred and add my callBack and errBack functions. I can see them when I print out the callbacks so I know they are there. As far as I can see from my logs, the deferred runs but neither the callback nor the errback are fired. Possibly an error occurs that stops them but I have not yet found a way to pin this down. I see there is a setDebugging option but I have not yet worked out how I would use this. So before I start trying any more incorrect assumptions, what I would like to confirm is that I should expect my callback or errback functions to be called automatically when my deferred has executed and returned. If successful, it would return fairly promptly. If it fails it could take a number of minutes to fail. But that is another problem. So should my callback or errback fire without me having to initiate any action other than setting up the deferred. And should I be using a deferred at all for this situation. If I know that the basis of what I am doing is correct and I am still having problems I can try and put together a small bit of code that reproduces the problem on the assumption that my code is probably wrong. Thanks for any pointers. I'm afraid that everytime I read the documentation on deferreds and go through the examples on using them I come to different conclusions on what I should be doing. So any help would be much appreciated. One example I looked at was the getPage example. Apart from doing a read it seems to be much along the lines of what I want to do. I am writing using straight TCP rather than HTTP. But that example fires the callback and errback without any problems. So it makes me think that my deferreds have something wrong with them. John Aherne
"John Aherne" <johnaherne@rocs.co.uk> writes: Hi John I have no concrete solution to what you describe, but I can try and explain how Deferreds work.
So should my callback or errback fire without me having to initiate any action other than setting up the deferred. And should I be using a deferred at all for this situation.
First, a Deferred is an amazingly powerful concept, but the implementation is surpricingly simple. I once posted a simple explaination for my own project: http://article.gmane.org/gmane.comp.cryptography.viff.devel/26 The basic idea is that a Deferred is a container for event handling functions, called callbacks in Twisted. Python has function pointers, so it is easy to make such a class: class MiniDeferred: def __init__(self): self.callbacks = [] self.value = None def addCallback(self, func, *extra_args): self.callbacks.append((func, extra_args)) def callback(self, initial_value): self.value = initial_value for func, extra_args in self.callbacks: self.value = func(self.value, *extra_args) return self.value def __str__(self): if self.value is None: return "MiniDeferred (no value yet)" else: return "MiniDeferred (value: %s)" % (self.value,) This class skips the error handling (errbacks), but I believe it illustrates the core ideas of a Deferred. You can use it like this:
x = MiniDeferred() print x MiniDeferred (no value yet) x.addCallback(lambda value: value**3) x.callback(-3) print x MiniDeferred (value: -27)
The crucial thing is that "someone" or "something" must call the callback method on your Deferred. The general rule to follow is that it is the code that creates a Deferrred which has the reponsibility of calling the callback method at the correct time. So when you use the getPage function you get a Deferred back. Behind the scenes the getPage function has setup things in such a way that the callback method of the returned Deferred is called when the page has arrived -- you must not invole callback yourself in this case. But if you create a Deferred yourself, then it is up to you to make a call to the callback method when ready. And example could be this mini implementation of the DeferredList class: class MiniDeferredList(MiniDeferred): def __init__(self, deferreds): MiniDeferred.__init__(self) self.results = [None] * len(deferreds) self.missing_results = len(deferreds) for index, deferred in enumerate(deferreds): deferred.addCallback(self._callback_fired, index) def _callback_fired(self, result, index): self.results[index] = result self.missing_results -= 1 if self.missing_results == 0: self.callback(self.results) return result Here the _callback_fired method invokes the callback method when all deferreds have fired. I hope this helps a bit -- it helped me to look at the implementation in defer.py and see that it was less scary than I had imagined :-) -- Martin Geisler VIFF (Virtual Ideal Functionality Framework) brings easy and efficient SMPC (Secure Multiparty Computation) to Python. See: http://viff.dk/.
participants (2)
-
John Aherne -
Martin Geisler