Mailman 3 June 2008 - Twisted

[Twisted-Python] Returning a DeferredList?
by Nathan 01 Jul '08

01 Jul '08

I regularly use the return values of deferreds to simulate synchronous behaviour like this: def handle_db_result(result): if ({some logic stuff}): return True else: return False def check_db(): d = {some db query that returns a deferred} d.addCallback(handle_db_result) return d def main_func(): if check_db(): # This eventually returns True or False # Do some stuff else: # Do some different stuff So now I'm trying to do the same thing with DeferredList's, but apparently the semantics aren't the same. Here's some actual runnable sample code trying to use the DeferredList: from twisted.internet import reactor, defer def callback_function(results): return "(--> This is the value that I want to get into retval! <--)" def make_a_list(): deferred1 = defer.Deferred() deferred2 = defer.Deferred() dl = defer.DeferredList([deferred1, deferred2]) dl.addCallback(callback_function) deferred1.callback('one') deferred2.callback('two') return dl # The crucial line! def test(): retval = make_a_list() print "retval is:", retval reactor.callWhenRunning(test) reactor.callLater(2.0, reactor.stop) reactor.run() The actual output of the above script is: retval is: <DeferredList at 0x7a30a8 current result: '(--> This is the value that I want to get into retval! <--)'> Through some trial and error I found that if I change that "crucial line" above to "return dl.result", then it appears to work how I like. Is that what I should be doing, or did I just get lucky in that particular instance?? Also, I notice that if one of the deferred's errbacked, then the DeferredList seems to errback as well. It shouldn't do that according to http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto7 ... so I filed a bug (#3323). ~ Nathan

1 0

[Twisted-Python] Issues with creating switchboards with twisted.words.protocols.msn
by Hermann Kaser 30 Jun '08

30 Jun '08

Hello all, I'm trying to write an MSN client but I've run into a problem: for some reason when I create a Switchboard and try to invite someone it sends the join command and the invitation command all together and the MSN server doesn't answer any of them. Here's the relevant bits of the code I have: class Switchboard(msn.SwitchboardClient): def connectionMade(self): self.key = self.factory.key self.sessionID = self.factory.sessionID self.userHandle = self.factory.userHandle self.invite = self.factory.invite msn.SwitchboardClient.connectionMade(self) d = self.inviteUser(self.invite) d.addCallback(self.Invited) def Invited(self, *args): print 'Running reply callback' And this is the ngrep stream after the XFR request # send transfer request T 192.168.1.36:39816 -> 207.46.110.133:1863 [AP] CHG 6 NLN..XFR 7 SB.. # get transfer response T 207.46.110.133:1863 -> 192.168.1.36:39816 [AP] XFR 7 SB 64.4.37.58:1863 CKI 712606054.15830117.2511186.. T 207.46.110.133:1863 -> 192.168.1.36:39816 [AP] CHG 6 NLN 0.. T 207.46.110.133:1863 -> 192.168.1.36:39816 [AP] ILN 6 NLN [redacted](a)hotmail.com [redacted screenname] 1073741856.. T 192.168.1.36:44867 -> 64.4.37.58:1863 [AP] USR 1 [my email] 712606054.15830117.2511186..CAL 2 [email of who i want to invite].. If I try to do the same process with pidgin for example, the stream looks exactly the same except the USR and CAL command are on separate packets, and the server responds to the USR command before the client sends the CAL command. Can anyone help me out figuring this out? I know you can send more than one command per packet, as seen in the first packet of the ngrep stream I posted, so I don't see why it wouldn't work to send the USR and CAL command together, but I suspect that not all commands may be stuck together. I stuck a time.sleep(20) between the calling msn.SwitchboardClient.connectionMade(self) and self.inviteUser(self.invite) to see if it was a timing issue but what happens is that the USR command gets buffered, and only when the script tries to send the CAL command does the socket actually write it out. Any help on this would be greatly appreciated, I've been digging around the documentation and the code but I can't seem to figure out why it's buffering the USR command and I don't know enough about the MSN protocol to know which commands need a unique packet and which ones can go with others. Regards, Hermann Käser http://theragingche.com/ http://semicir.cl/user/hermzz

3 3

[Twisted-Python] Implementing startup sanity-checks for twistd services.
by Tim Allen 30 Jun '08

30 Jun '08

The system I'm working on at the moment is talking to MySQL, and in order to get reasonable behaviour, I need to make sure the "SQL_MODE" variable is set on both the database connection, and on the stored procedures I'll be calling. If this is not set up correctly, I expect to be plagued with silent data corruption issues. I can force the SQL_MODE per-connection when I connect to the database, but I need to double-check the procedures whenever I start up. Ideally, my system would follow a startup procedure something like this: - Read configuration - Connect to the configured database. - Check the SQL_MODE associated with any stored procedures. - If everything checks out OK, continue the startup process. - Otherwise, log a noisy error and shut down. Of course, I'd prefer to do things the Twisted Way if at all possible. I've had some ideas about how to implement this, but none of them seem terribly useful: If I launch my queries at startup, and only add my service to the master Application object if they return successfully, then by the time I try to start my service twistd will have (I assume) already dropped privileges and chroot'd and so forth. If I attach my service to the master Application object at startup *then* launch my queries, then I might wind up accepting queries before my sanity checks have completed, defeating the point. Also, if my service has started and my sanity-checks fail, how can I cleanly shut down twistd from inside? If my service is just one of possibly many running inside the master Application object, it seems a bit rude for me to call reactor.stop(); on the other hand, if I just stop my service then twistd keeps happily running - confusing operations staff who often use 'ps' to check whether systems are up. Is there a better way to do this?

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 29 Jun '08

29 Jun '08

1 0

Re: [Twisted-Python] Re-working a synchronous iterator to use Twisted
by Terry Jones 29 Jun '08

29 Jun '08

Following the massive interest in my earlier postings on this thread, I'm following up to myself again :-) Here's what I was trying to do: > In case it wasn't clear before, you're pulling "results" (e.g., from a > search engine) in off the web. Each results pages comes with an indicator > to tell you whether there are more results. I wanted to write a function > (see processResults below) that, when called, would call the process > function below on each result, all done asynchronously. I posted some cumbersome code to roughly do that. I've since been thinking about this on and off, with help from Esteve Fernandez, and we've made the code quite a bit simpler. I think there's a general pattern here that's worth thinking about. Roughly: the above need is like the Twisted analogy of using iterators in regular synchronous programming. By that I mean that the normal pattern of Twisted usage is: a single event is anticipated (by the programmer), it occurs once, and its result is passed down a call/errback chain. That's roughly like a single function call in synchronous code. But if you are expecting a sequence of external events to occur and you want to asynchronously pass their results in turn down a call/errback chain. The need to do this in synchronous code can be filled with a simple iterator. But doing this asynchronously (when the fetch of the next batch of results might take a while) doesn't seem to fit easily into the single-shot asynchronous Twisted paradigm. I thought about modifying defer.py to allow a callback chain to be called multiple times (and to have the "normal" single-shot chain be a special case). But that was clearly going to get messy. BTW, I find defer.py is really elegant. After more thinking about how to make my previously posted code simpler, Esteve and I came up with what you'll find at http://python.pastebin.com/f7df56752 (code) and http://python.pastebin.com/f1e582264 (simple tests) The idea is that you provide a result fetcher function to the TwIterator class. This function will be called repeatedly, as needed, to get more results. It returns a deferred whose callback it should call with a list of next results (which may be empty), a bool to indicate whether to re-call the function, and a dict of args to pass to it next time. The TwIterator class provides you with a list() method that you can use almost like an iterator: @inlineCallbacks def printer(results): for x in results: print (yield x) fetcher.list().addCallback(printer) This is in some sense like a general asynchronous iterator for Twisted. The printer function receives an iterator, each element of which is a deferred, and when that deferred fires it produces the next result. The test code gives 4 simple example result-fetching functions, and calls them all asynchronously. If you run it you'll see the results coming out in a somewhat random order. I wont go into more detail, given that no-one responded to the first two postings. It's still possible that I'm trying to solve a problem that can already be done by some standard Twisted module. I don't know enough about Twisted to know for sure. Terry

1 0

[Twisted-Python] LoopingCall question
by Mike Preshman 28 Jun '08

28 Jun '08

Hello, I am trying to run three deferred LoopingCall chains in parallel. Here is my code below. I am trying to figure out what is the correct way of exiting the polling after I meet some condition that the polling satisfies. Right now I am throwing an exception which is being caught by an errback. I wait for all three deferred chains to complete via a DeferredList with (defer.gatherResults) and then I add a callback to that in order to stop the reactor. class PollingException(twisted.python.failure.Failure): def __init__(self, var): self.var = var def poll(*args): print "poll", args raise PollingException(args[1]) def errorHandle(failure): print "poll exited" return True def stopReactor(result): print "about to stop reactor" reactor.stop() t1 = task.LoopingCall(poll, "A") d1 = t1.start(1, 3) d1.addErrback(errorHandle) t2 = task.LoopingCall(poll, "B") d2 = t2.start(2, 3) d2.addErrback(errorHandle) t3 = task.LoopingCall(poll, "C") d3 = t3.start(3, 3) d3.addErrback(errorHandle) dL = defer.gatherResults([d1, d2, d3]) dL.addCallback(lambda _: reactor.stop()) reactor.run() However, when I run the code and I try to stop the reactor, I get the following exception which I find puzzling. poll ('A',) poll exited poll ('B',) poll exited poll ('C',) poll exited Unhandled error in Deferred: Traceback (most recent call last): File "twisted_poll.py", line 36, in <module> dL.addCallback(lambda _: reactor.stop()) File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 191, in addCallback callbackKeywords=kw) File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 182, in addCallbacks self._runCallbacks() --- <exception caught here> --- File "c:\python25\lib\site-packages\twisted\internet\defer.py", line 317, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted_poll.py", line 36, in <lambda> dL.addCallback(lambda _: reactor.stop()) File "c:\python25\lib\site-packages\twisted\internet\base.py", line 342, in stop raise RuntimeError, "can't stop reactor that isn't running" exceptions.RuntimeError: can't stop reactor that isn't running Thanks very much, -Mike

4 5

[Twisted-Python] twisted threading with a long running process
by Doug Farrell 27 Jun '08

27 Jun '08

Hi all, I've built a Twisted server that handles Python logging messages sent from the logging.handlers.SocketHandler system. Right now my protocol implementation handles logging the received logRecord to a file and entering it into an in memory model for statistical purposes. I'm thinking of off-loading this activity to a separate thread by having the protocol just send the logRecord it receives into queue that is serviced by a separate thread. My question is whether using the twisted.internet.threads.deferToThread() method is a good way to kick off this other thread. The other thread will be running for as long as the Twisted server runs so it can handle logRecords in the queue. Would just starting a threading.Thread based class be a better solution, giving access to queue inside the class to the protocol? Thanks, Doug

2 1

[Twisted-Python] FW: reject all open connections
by Irae Hueck Costa 27 Jun '08

27 Jun '08

From: iraehc(a)hotmail.com To: twisted-python(a)twistedmatrix.com Subject: reject all open connections Date: Sat, 21 Jun 2008 22:26:37 +0200 hi all! i am sorry, its the first time i am asking something twice but its important for me to get a question. thank you!: i just want to clean my reactor - all readers and writer should looseConnection() but how can i do this? reactor.removeAll() dont do it. the best solutions i found was a horrible workaround: a thread is tarted, which constantly calls ``[conn.loseConnection() for conn in reactor.removeAll()]`` is there a better solution? thank you Conheça já o Windows Live Spaces, o site de relacionamentos do Messenger! Crie já o seu! _________________________________________________________________ Conheça o Windows Live Spaces, a rede de relacionamentos do Messenger! http://www.amigosdomessenger.com.br/

2 1

[Twisted-Python] Example for a new protocol over TCP
by Matthieu Brucher 26 Jun '08

26 Jun '08

Hi, I'm new to twisted and I'm trying to implement a set of protocols (already available in Java, JXSE). I lamost know how I will set up the different services that will communicate through the reactor, but I'm starting from the beginning, so here is my real question. How can I set up a factory (using a TCP protocol) so that when I want to send a message to someone, it connects though TCP to the other computer, they exchange a welcome message, and only then the message is sent though the TCP protocol ? Of course, it is not as simple as this, but if someone has a pointer, I will be able to fill the missing pieces in my design ;) Thanks, Matthieu -- French PhD student Website : http://matthieu-brucher.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher

2 3

[Twisted-Python] Question about using a deferred
by Doug Farrell 25 Jun '08

25 Jun '08

Hi all, I've written a twisted server that handles logging messags from the logging.handlers.SocketHandler class. The protocol I've written handles the messages and then calls its own instance of a logger to log the message to. I'm wondering if the file I/O the logger is doing is something I hand off to a deferred, or is it quick enough to not to worry about? Thanks! Doug

1 0