[Twisted-Python] Learning Twisted
Sorry for the length of this post but I learn best by trying to explain what I'm learning (even if nobody's listening). Twisted seems very cool but also *huge* and unlike any framework I've used before so I thought I'd post my first experiences with it in the hopes that I could be corrected where needed (and maybe even help people other new like me). Here's the deal: I want to write an IMAP4 client and Twisted's IMAP4Client [1] looks much more featureful than Python's imaplib. Since I'm writing a client, I started out by reading the "Writing a TCP client" HOWTO [2] where I learned, by looking at the ircLogBot example, that I needed to define a subclass of IMAP4Client and a subclass of ClientFactory. This is what I came up with: from twisted.internet import reactor, protocol from twisted.protocols import imap4 class MyIMAP4Client(imap4.IMAP4Client): def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client f = MyIMAP4ClientFactory() reactor.connectTCP("server", 143, f) reactor.run() Running this works (I see "connectionMade" printed to the console). IMAP4Client has a login method so I modified my connectionMade method to look like this: def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" self.login("user", "password") But I couldn't tell if it worked or not. That's when I read that login returns a "deferred whose callback is invoked if login is successful". So I read the "Using Deferreds" HOWTO [3] and learned that I needed to pass in a function to addCallback on the deferred returned by calling login. So connectionMade now looks like this: def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" d = self.login("user", "password") d.addCallback(loginCallback) loginCallback looks like this: def loginCallback(d): print "loginCallback:", d Note that this is *not* a method of the MyIMAP4Client class. But it worked! I got this printed to the console: connectionMade loginCallback: ([], 'OK LOGIN Ok.') I don't know, however, what the tuple represents. What's that empty list? In a perfect world, would this be explained in the IMAP4Client documentation? (I'm assuming that every callback would be different.) I thought it sucked that my loginCallback function wasn't a member of my class but I thought I'd see what error I'd get if I made it one so I changed the two methods to look like this: def loginCallback(self, d): print "loginCallback:", d def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" d = self.login("user", "password") d.addCallback(self.loginCallback) And that worked, too! At this point, I don't understand *why* this works but it's awfully nice that it does. ("Using Deferreds" shows one example of a callback method but doesn't discuss it in any way.) Next, I wanted to try actually doing something useful. So I called self.select from loginCallback so that I could see how many messages exist in my INBOX. def loginCallback(self, d): print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): print "selectCallback:", d This is the output I got for selectCallback: selectCallback: {'EXISTS': 323, 'PERMANENTFLAGS': ('$MDNSent', 'NonJunk', '\\*', '\\Draft', '\\Answered', '\\Flagged', '\\Deleted', '\\Seen'), 'READ-WRITE': 1, 'FLAGS': ('$MDNSent', 'NonJunk', '\\Draft', '\\Answered', '\\Flagged', '\\Delete d', '\\Seen', '\\Recent'), 'UIDVALIDITY': 1076206465, 'RECENT': 0} Apparently, the parameter to the select callback is a dict. The documentation for select actually mentions some of the keys. It looks like I could use the EXISTS key to get how many messages exist in my INBOX: def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] Now all I have to do is logout (up until now, I've been terminating the script with Ctrl-C): def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): sys.exit(0) This isn't correct, however, since the call to sys.exit prints an exception to the console and, more distressingly, does *not* exit the script. At this point I realize that I'm in some sort of event loop (started by calling reactor.run) and need to tell this loop to stop. So I look at the "Reactor basics" HOWTO [4] and find my way to IReactorCore [5] which documents a stop method: def logoutCallback(self, d): reactor.stop() This works wonderfully although I have no idea if the connection to the server is actually being gracefully closed. So I override connectionLost to find out: def connectionLost(self, reason): imap4.IMAP4Client.connectionLost(self) print "connectionLost" Now "connectionLost" is the last thing I see on my console before the script exits. Nice. I also override both sendLine and lineReceived so that I can see the conversation with the IMAP server to see if I'm actually logging out before closing the connection: def sendLine(self, line): imap4.IMAP4Client.sendLine(self, line) print line def lineReceived(self, line): imap4.IMAP4Client.lineReceived(self, line) print line Perfect. Just for reference, here's the program I ended up with: from twisted.internet import reactor, protocol from twisted.protocols import imap4 debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def connectionMade(self): imap4.IMAP4Client.connectionMade(self) if debug: print "connectionMade" d = self.login("user", "password") d.addCallback(self.loginCallback) def loginCallback(self, d): de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): reactor.stop() def connectionLost(self, reason): imap4.IMAP4Client.connectionLost(self) if debug: print "connectionLost" def sendLine(self, line): imap4.IMAP4Client.sendLine(self, line) if debug: print line def lineReceived(self, line): imap4.IMAP4Client.lineReceived(self, line) if debug: print line class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client f = MyIMAP4ClientFactory() reactor.connectTCP("server", 143, f) reactor.run() So how'd I do for a Twisted newbie? Am I on the path to enlightenment? Are there any obvious errors in what I've interpreted so far? Is this a "good" implementation for this feature or is their a more Twisted-ish approach I need to strive for? Am I correct in assuming that the IMAP4Client documentation needs some buffing up? (Maybe I can help with that.) I didn't look at any source code (other than in the HOWTOs) in implementing this so I think that's a good sign. But as I mentioned above, I'm not clear on how I know what the signature for my callbacks should be or what the parameters to those callbacks mean in some cases. Also, it's not obvious to me how protocol implementors decide when to have users override a method versus having them use callbacks. I know I'm working on client code but knowing this might help me know how to use their code. Is it a matter of personal preference? The ircLogBot example didn't have any callbacks using deferreds but it looks like I had no choice when using IMAP4Client. Thanks for reading this far! -- Jason [1] http://twistedmatrix.com/documents/current/api/twisted.protocols.imap4.IMAP4... [2] http://twistedmatrix.com/documents/current/howto/clients [3] http://twistedmatrix.com/documents/current/howto/defer.html [4] http://twistedmatrix.com/documents/current/howto/reactor-basics.html [5] http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.2.0/api/twisted...
Jason Diamond wrote:
Thank you for sending this post. I wish every new user would give such great feedback!
Although I'm not an expert with the IMAP code, everything seemed right except this last part:
If possible, you should try to run Twisted code under twistd, which means that you'd remove that code from your Python module, and put it into a separate .tac file that read something like: from twisted.application.internet import TCPClient from twisted.application.service import Application, IServiceCollection application = Application("my client application") f = MyIMAP4ClientFactory() client = TCPClient("server", 143, f) client.setServiceParent(application) Even if you do have a need to put your code into its own script - and the twistd support for clients is admittedly weaker than that for servers - it's generally a good idea not to put it in the same script as your code.
Yes. Pretty much every aspect of the documentation needs help. It would be greatly appreciated!
It's not purely preference, but there is some overlap. Generally, what Deferreds are used for are when you've got some operation that happens once, with one result: you want to delete a message, for example, and get a notification when that deletion operation has completed. Methods on objects are used when you are doing something more persistent, for example, connecting a client to a server - there is a potential for success or failure of that one operation, true, but the more interesting thing is the continuous stream of data being received from or sent to the other side of the connection.
Glyph Lefkowitz wrote:
Also, the "request" vs "arbitrary events" distinction should be clarified: Deferreds (i.e., callbacks) are used when you want to expose an API that allows a request to be made for some operation to occur, and they'll get triggered when the operation is complete (or has errored). Methods to be overridden are there for when there's no 1:1 mapping of requests to responses, when these events can happen even without expecting them. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
Christopher Armstrong wrote:
Ah! This makes perfect sense to me. So I can think of overridable methods as events that didn't require any action on my part to fire. However, the ircLogBot example [1], implies that calling self.join will eventually invoke self.joined as if self.joined was the permanent callback for calling self.join. I like your explanation for the distinction between deferreds and methods but this seems to be blurring the two. What happens if an error occurred while trying to join a channel? (I'm not trying to pick anybody--I'm just trying to make sense of the examples that I'm seeing.) -- Jason [1] http://twistedmatrix.com/documents/current/howto/clients#auto5
Jason Diamond wrote:
Aah, but that taps into the deep mystery of IRC. The server can send you a 'joined' notification *even if you didn't previously request to join a channel*. Also there is no particular way to hook up a join request and a join notification; if you request to join #foobar 3 times, and then you only get one notification of having joined, it's not clear which request gets the callback; commands are not tagged. Of course this varies between IRC servers, too. Better-specified protocols will tend to have better-specified interfaces that make use of Deferreds more.
Glyph Lefkowitz wrote:
Thanks for the tip, I started reading the "Twisted from Scratch" tutorial [1] and see why this is useful now. For my simple client explorations, though, it's spitting out a lot more output than I would expect. But still very informative! -- Jason [1] http://twistedmatrix.com/documents/current/howto/tutorial
Jason Diamond wrote:
I'd like to repeat what Glyph said. This is great feedback, I'd love to = see more of it :)
The above works (as you noticed ;) but isn't quite as good as hooking = into the "serverGreeting" method, which is called after the IMAP4 server = sends its initial message (I realize this isn't well documented).
The callback value here is something of an implementation detail. = Deferreds (somewhat informally) often fall into one of two categories: = those which will eventually be called back with an interesting value = which is necessary for further computations, and those which will = eventually be called back with a not-so-interesting value which only = serves to indicate that the desired operation has completed. login() = falls into the second category.
Yep. More documentation would be great. The actual signature for = callbacks is easy. It's always a callable that takes one argument (by = the way, this is why your code worked with both free functions and = methods - it doesn't matter what kind of callable you use, as long as it = takes one argument). Further documentation about the specifics of what = that one parameter means is definitely needed, though.
[snip]
Thanks for reading this far!
Thanks for writing! Jp
Jp Calderone wrote:
Cool! This makes perfect sense and I'll look for methods like serverGreeting from now on. Here's my updated test: from twisted.internet import reactor, protocol from twisted.protocols import imap4 server = "xxx" username = "yyy" password = "zzz" debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def serverGreeting(self, caps): if debug: print "serverGreeting:", caps imap4.IMAP4Client.serverGreeting(self, caps) d = self.login(username, password) d.addCallback(self.loginCallback) def loginCallback(self, d): if debug: print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): if debug: print "selectCallback:", d print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): if debug: print "logoutCallback:", d reactor.stop() def connectionLost(self, reason): if debug: print "connectionLost:", reason imap4.IMAP4Client.connectionLost(self) def sendLine(self, line): if debug: print "sendLine:", line imap4.IMAP4Client.sendLine(self, line) def lineReceived(self, line): if debug: print "lineReceived:", line imap4.IMAP4Client.lineReceived(self, line) class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client if __name__ == "__main__": f = MyIMAP4ClientFactory() reactor.connectTCP(server, 143, f) reactor.run()
OK, this makes sense. But if the login callback doesn't have any useful information, why isn't it None (like the logout callback)? Or is that the implementation detail you're referring to that I should just forget about?
I also just realized that a method attribute retrieved from an instance, even though it's declared as taking in two arguments (the first being self), returns a function that only has one argument--self is bound to the instance you used to grab the method. -- Jason
Jason Diamond wrote:
Thank you for sending this post. I wish every new user would give such great feedback!
Although I'm not an expert with the IMAP code, everything seemed right except this last part:
If possible, you should try to run Twisted code under twistd, which means that you'd remove that code from your Python module, and put it into a separate .tac file that read something like: from twisted.application.internet import TCPClient from twisted.application.service import Application, IServiceCollection application = Application("my client application") f = MyIMAP4ClientFactory() client = TCPClient("server", 143, f) client.setServiceParent(application) Even if you do have a need to put your code into its own script - and the twistd support for clients is admittedly weaker than that for servers - it's generally a good idea not to put it in the same script as your code.
Yes. Pretty much every aspect of the documentation needs help. It would be greatly appreciated!
It's not purely preference, but there is some overlap. Generally, what Deferreds are used for are when you've got some operation that happens once, with one result: you want to delete a message, for example, and get a notification when that deletion operation has completed. Methods on objects are used when you are doing something more persistent, for example, connecting a client to a server - there is a potential for success or failure of that one operation, true, but the more interesting thing is the continuous stream of data being received from or sent to the other side of the connection.
Glyph Lefkowitz wrote:
Also, the "request" vs "arbitrary events" distinction should be clarified: Deferreds (i.e., callbacks) are used when you want to expose an API that allows a request to be made for some operation to occur, and they'll get triggered when the operation is complete (or has errored). Methods to be overridden are there for when there's no 1:1 mapping of requests to responses, when these events can happen even without expecting them. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/
Christopher Armstrong wrote:
Ah! This makes perfect sense to me. So I can think of overridable methods as events that didn't require any action on my part to fire. However, the ircLogBot example [1], implies that calling self.join will eventually invoke self.joined as if self.joined was the permanent callback for calling self.join. I like your explanation for the distinction between deferreds and methods but this seems to be blurring the two. What happens if an error occurred while trying to join a channel? (I'm not trying to pick anybody--I'm just trying to make sense of the examples that I'm seeing.) -- Jason [1] http://twistedmatrix.com/documents/current/howto/clients#auto5
Jason Diamond wrote:
Aah, but that taps into the deep mystery of IRC. The server can send you a 'joined' notification *even if you didn't previously request to join a channel*. Also there is no particular way to hook up a join request and a join notification; if you request to join #foobar 3 times, and then you only get one notification of having joined, it's not clear which request gets the callback; commands are not tagged. Of course this varies between IRC servers, too. Better-specified protocols will tend to have better-specified interfaces that make use of Deferreds more.
Glyph Lefkowitz wrote:
Thanks for the tip, I started reading the "Twisted from Scratch" tutorial [1] and see why this is useful now. For my simple client explorations, though, it's spitting out a lot more output than I would expect. But still very informative! -- Jason [1] http://twistedmatrix.com/documents/current/howto/tutorial
Jason Diamond wrote:
I'd like to repeat what Glyph said. This is great feedback, I'd love to = see more of it :)
The above works (as you noticed ;) but isn't quite as good as hooking = into the "serverGreeting" method, which is called after the IMAP4 server = sends its initial message (I realize this isn't well documented).
The callback value here is something of an implementation detail. = Deferreds (somewhat informally) often fall into one of two categories: = those which will eventually be called back with an interesting value = which is necessary for further computations, and those which will = eventually be called back with a not-so-interesting value which only = serves to indicate that the desired operation has completed. login() = falls into the second category.
Yep. More documentation would be great. The actual signature for = callbacks is easy. It's always a callable that takes one argument (by = the way, this is why your code worked with both free functions and = methods - it doesn't matter what kind of callable you use, as long as it = takes one argument). Further documentation about the specifics of what = that one parameter means is definitely needed, though.
[snip]
Thanks for reading this far!
Thanks for writing! Jp
Jp Calderone wrote:
Cool! This makes perfect sense and I'll look for methods like serverGreeting from now on. Here's my updated test: from twisted.internet import reactor, protocol from twisted.protocols import imap4 server = "xxx" username = "yyy" password = "zzz" debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def serverGreeting(self, caps): if debug: print "serverGreeting:", caps imap4.IMAP4Client.serverGreeting(self, caps) d = self.login(username, password) d.addCallback(self.loginCallback) def loginCallback(self, d): if debug: print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): if debug: print "selectCallback:", d print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): if debug: print "logoutCallback:", d reactor.stop() def connectionLost(self, reason): if debug: print "connectionLost:", reason imap4.IMAP4Client.connectionLost(self) def sendLine(self, line): if debug: print "sendLine:", line imap4.IMAP4Client.sendLine(self, line) def lineReceived(self, line): if debug: print "lineReceived:", line imap4.IMAP4Client.lineReceived(self, line) class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client if __name__ == "__main__": f = MyIMAP4ClientFactory() reactor.connectTCP(server, 143, f) reactor.run()
OK, this makes sense. But if the login callback doesn't have any useful information, why isn't it None (like the logout callback)? Or is that the implementation detail you're referring to that I should just forget about?
I also just realized that a method attribute retrieved from an instance, even though it's declared as taking in two arguments (the first being self), returns a function that only has one argument--self is bound to the instance you used to grab the method. -- Jason
participants (4)
-
Christopher Armstrong -
Glyph Lefkowitz -
Jason Diamond -
Jp Calderone