Mailman 3 May 2004 - Twisted

[Twisted-Python] patch attached--for 0 delay callLater
by Bill la Forge May 19, 2004

May 19, 2004

The attached notes.txt file contains a diff -u patch. I consign the copyright to Glyph. I've tested it best I can, but could someone run it through the regression tests? I'd like that assurance before updating my web pages. ;-) Jp, I added one of your ideas anyway. But the bisect_left kept giving strange errors. Bill la Forge http://www.geocities.com/laforge49/ Yahoo! India Matrimony: Find your partner online. --- other/Twisted/Twisted-1.2.0/twisted/internet/base.py 2004-02-03 02:51:… [View More]

1 0

[Twisted-Python] How to get the IP address of a client using XML-RPC
by Daniel Newton May 18, 2004

May 18, 2004

2 1

[Twisted-Python] thread migration and bsddb
by Bill la Forge May 18, 2004

May 18, 2004

The 040518 release of CompStrm supports the movement of computational streams to a server thread and back to the main thread. See http://compstrm.sourceforge.net/thread.html Thread migration makes it very easy to do blocking I/O without comprimising the main thread. Unfortunatly, this is still not good enough when working with bsddb, as cursors are NOT thread free. To reuse a cursor, you must be on the SAME thread it was created on. ;-( A moderatly reasonable solution seems to be to have … [View More]

2 2

[Twisted-Python] A problem with runUntilCurrent
by Bill la Forge May 18, 2004

May 18, 2004

Glyph, I couldn't resist digging into the code you sent. I noticed that you changed the code to use callLater. But there's a problem with that, I suspect... In base.runUntilCurrent, there's this while loop... while self._pendingTimedCalls and (self._pendingTimedCalls[-1].time <= now): call = self._pendingTimedCalls.pop() . . . This means that when you do a callLater with a delay of 0, runUntilCurrent will immediately call the delayed function, which then calls … [View More]

5 13

[Twisted-Python] More IMAP4Client/Deferred fun
by Jason Diamond May 18, 2004

May 18, 2004

Hi. I'm now trying to move beyond the simple exercise I did yesterday and turn my IMAP4 "client" into something that could be reusable from other classes. This is my current script: >>> from twisted.internet import reactor, protocol from twisted.protocols import imap4 server = "xxx" username = "yyy" password = "zzz" debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def serverGreeting(self, caps): if debug: print "serverGreeting:", caps imap4.IMAP4Client.… [View More]serverGreeting(self, caps) d = self.login(username, password) d.addCallback(self.loginCallback) def loginCallback(self, d): if debug: print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): if debug: print "selectCallback:", d print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): if debug: print "logoutCallback:", d reactor.stop() def connectionLost(self, reason): if debug: print "connectionLost:", reason imap4.IMAP4Client.connectionLost(self) def sendLine(self, line): if debug: print "sendLine:", line imap4.IMAP4Client.sendLine(self, line) def lineReceived(self, line): if debug: print "lineReceived:", line imap4.IMAP4Client.lineReceived(self, line) class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client if __name__ == "__main__": f = MyIMAP4ClientFactory() reactor.connectTCP(server, 143, f) reactor.run() <<< (Since this is a learning exercise, I'm trying to keep things simple by including everything in one script.) The problem with the above is that it prints to stdout in selectCallback and then stops the reactor in logoutCallback. I want to use this class from a Resource's render method. But instead of printing to stdout, I want to "return" the value so that the render method can use that value while outputting HTML. It's probably also not a good idea to stop the reactor while serving a request. The reactor will already be started by the time render gets invoked on my resorce so I won't be calling reactor.run, right? But I will need to connect to my IMAP server. So I tried modifying the script to connect *after* the reactor started: >>> if __name__ == "__main__": def connect(f): reactor.connectTCP(server, 143, f) f = MyIMAP4ClientFactory() reactor.callLater(0, connect, f) reactor.run() <<< This works just like before but feels weird to me. Is that the correct usage for reactor.callLater? This was the only way I could figure out how to invoke a functon *after* calling reactor.run. But this might be a moot point since reactor.connectTCP probably just adds some events to the queue and doesn't do anything until the reactor starts anyways, right? Looking at the documentation for connectTCP, I found out that it returns an IConnector implementation but I'm not saving it to a variable and doing anything with it so I'm not sure what it would buy me and none of the methods on it look helpful here. Anyways, I need to *not* print to the console: >>> def selectCallback(self, d): if debug: print "selectCallback:", d self.factory.count = d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) <<< I'm now saving the count on the factory that created the protocol instead of printing it directly to stdout. At some point, I'm going to have to notify somebody that I've retrieved the count so I figure I'm going to need a Deferred for that. I modify the factory to hold on to a Deferred for me and set it up to call a function to use the count: >>> if __name__ == "__main__": def printCount(count): print count reactor.stop() f = MyIMAP4ClientFactory() f.deferred = defer.Deferred() f.deferred.addCallback(printCount) reactor.connectTCP(server, 143, f) reactor.run() <<< But now I need to tell the Deferred to actually make that callback so I modify logoutCallback and connectionLost: >>> def logoutCallback(self, d): if debug: print "logoutCallback:", d def connectionLost(self, reason): if debug: print "connectionLost:", reason imap4.IMAP4Client.connectionLost(self) self.factory.deferred.callback(self.factory.count) <<< Now logoutCallback is no longer stopping the reactor. And connectionLost is using the factory's deferred object to inform some function what the actual count is. This works but it seems messy. First of all, it seems like it would be nice if my module could expose a free function to do all this for me: >>> def getCount(): f = MyIMAP4ClientFactory() f.deferred = defer.Deferred() reactor.connectTCP(server, 143, f) return f.deferred if __name__ == "__main__": def printCount(count): print count reactor.stop() getCount().addCallback(printCount) reactor.run() <<< That's the best I can do. It works but I still don't feel all that stoked on it. What am I doing wrong? Is using my ClientFactory to hold data like this the correct thing to do? In this case, there will only be one Protocol instance for this ClientFactory, right? Just for easy reference, here's the complete script in all its glory: >>> from twisted.internet import defer, reactor, protocol from twisted.protocols import imap4 server = "xxx" username = "yyy" password = "zzz" debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def serverGreeting(self, caps): if debug: print "serverGreeting:", caps imap4.IMAP4Client.serverGreeting(self, caps) d = self.login(username, password) d.addCallback(self.loginCallback) def loginCallback(self, d): if debug: print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): if debug: print "selectCallback:", d self.factory.count = d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): if debug: print "logoutCallback:", d def connectionLost(self, reason): if debug: print "connectionLost:", reason imap4.IMAP4Client.connectionLost(self) self.factory.deferred.callback(self.factory.count) def sendLine(self, line): if debug: print "sendLine:", line imap4.IMAP4Client.sendLine(self, line) def lineReceived(self, line): if debug: print "lineReceived:", line imap4.IMAP4Client.lineReceived(self, line) class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client def getCount(): f = MyIMAP4ClientFactory() f.deferred = defer.Deferred() reactor.connectTCP(server, 143, f) return f.deferred if __name__ == "__main__": def printCount(count): print count reactor.stop() getCount().addCallback(printCount) reactor.run() <<< Any suggestions for improvements or corrections to my thinking would be greatly appreciated! Thanks. -- Jason [View Less]

1 0

[Twisted-Python] How do I run a twisted server directly
by Alexander May May 17, 2004

May 17, 2004

I am going step by step through the Twisted From Scratch documentation. At step labeled twistd, there is this listing: http://twistedmatrix.com/documents/current/howto/listings/finger/finger11.py At the end it uses: application = service.Application('finger', uid=1, gid=1) factory = FingerFactory(moshez='Happy and well') internet.TCPServer(79, factory).setServiceParent( service.IServiceCollection(application)) Where as the previous code uses: reactor.listenTCP(1079, FingerFactory(… [View More]

8 17

[Twisted-Python] integrating CompStrm//adding background processing
by Bill la Forge May 17, 2004

May 17, 2004

I've been working on integrating Compstrm ( http://compstrm.sourceforge.net ) and, while integration was pretty easy, it got harder when I wanted to speed things up. Basicly, compstrm uses yields to impliment a kind of light-weight threads. So I needed to add background processing to the main reactor loop. Here's the code I finally came up with: from twisted.internet import reactor def _runUntilCurrentNew(): if reactor.poll: p=reactor.poll reactor.poll=None p(… [View More]

4 9

[Twisted-Python] Learning Twisted
by Jason Diamond May 16, 2004

May 16, 2004

Sorry for the length of this post but I learn best by trying to explain what I'm learning (even if nobody's listening). Twisted seems very cool but also *huge* and unlike any framework I've used before so I thought I'd post my first experiences with it in the hopes that I could be corrected where needed (and maybe even help people other new like me). Here's the deal: I want to write an IMAP4 client and Twisted's IMAP4Client [1] looks much more featureful than Python's imaplib. Since I'm … [View More]writing a client, I started out by reading the "Writing a TCP client" HOWTO [2] where I learned, by looking at the ircLogBot example, that I needed to define a subclass of IMAP4Client and a subclass of ClientFactory. This is what I came up with: from twisted.internet import reactor, protocol from twisted.protocols import imap4 class MyIMAP4Client(imap4.IMAP4Client): def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client f = MyIMAP4ClientFactory() reactor.connectTCP("server", 143, f) reactor.run() Running this works (I see "connectionMade" printed to the console). IMAP4Client has a login method so I modified my connectionMade method to look like this: def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" self.login("user", "password") But I couldn't tell if it worked or not. That's when I read that login returns a "deferred whose callback is invoked if login is successful". So I read the "Using Deferreds" HOWTO [3] and learned that I needed to pass in a function to addCallback on the deferred returned by calling login. So connectionMade now looks like this: def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" d = self.login("user", "password") d.addCallback(loginCallback) loginCallback looks like this: def loginCallback(d): print "loginCallback:", d Note that this is *not* a method of the MyIMAP4Client class. But it worked! I got this printed to the console: connectionMade loginCallback: ([], 'OK LOGIN Ok.') I don't know, however, what the tuple represents. What's that empty list? In a perfect world, would this be explained in the IMAP4Client documentation? (I'm assuming that every callback would be different.) I thought it sucked that my loginCallback function wasn't a member of my class but I thought I'd see what error I'd get if I made it one so I changed the two methods to look like this: def loginCallback(self, d): print "loginCallback:", d def connectionMade(self): imap4.IMAP4Client.connectionMade(self) print "connectionMade" d = self.login("user", "password") d.addCallback(self.loginCallback) And that worked, too! At this point, I don't understand *why* this works but it's awfully nice that it does. ("Using Deferreds" shows one example of a callback method but doesn't discuss it in any way.) Next, I wanted to try actually doing something useful. So I called self.select from loginCallback so that I could see how many messages exist in my INBOX. def loginCallback(self, d): print "loginCallback:", d de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): print "selectCallback:", d This is the output I got for selectCallback: selectCallback: {'EXISTS': 323, 'PERMANENTFLAGS': ('$MDNSent', 'NonJunk', '\\*', '\\Draft', '\\Answered', '\\Flagged', '\\Deleted', '\\Seen'), 'READ-WRITE': 1, 'FLAGS': ('$MDNSent', 'NonJunk', '\\Draft', '\\Answered', '\\Flagged', '\\Delete d', '\\Seen', '\\Recent'), 'UIDVALIDITY': 1076206465, 'RECENT': 0} Apparently, the parameter to the select callback is a dict. The documentation for select actually mentions some of the keys. It looks like I could use the EXISTS key to get how many messages exist in my INBOX: def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] Now all I have to do is logout (up until now, I've been terminating the script with Ctrl-C): def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): sys.exit(0) This isn't correct, however, since the call to sys.exit prints an exception to the console and, more distressingly, does *not* exit the script. At this point I realize that I'm in some sort of event loop (started by calling reactor.run) and need to tell this loop to stop. So I look at the "Reactor basics" HOWTO [4] and find my way to IReactorCore [5] which documents a stop method: def logoutCallback(self, d): reactor.stop() This works wonderfully although I have no idea if the connection to the server is actually being gracefully closed. So I override connectionLost to find out: def connectionLost(self, reason): imap4.IMAP4Client.connectionLost(self) print "connectionLost" Now "connectionLost" is the last thing I see on my console before the script exits. Nice. I also override both sendLine and lineReceived so that I can see the conversation with the IMAP server to see if I'm actually logging out before closing the connection: def sendLine(self, line): imap4.IMAP4Client.sendLine(self, line) print line def lineReceived(self, line): imap4.IMAP4Client.lineReceived(self, line) print line Perfect. Just for reference, here's the program I ended up with: from twisted.internet import reactor, protocol from twisted.protocols import imap4 debug = 0 class MyIMAP4Client(imap4.IMAP4Client): def connectionMade(self): imap4.IMAP4Client.connectionMade(self) if debug: print "connectionMade" d = self.login("user", "password") d.addCallback(self.loginCallback) def loginCallback(self, d): de = self.select("INBOX") de.addCallback(self.selectCallback) def selectCallback(self, d): print "I have %d messages in my INBOX." % d["EXISTS"] de = self.logout() de.addCallback(self.logoutCallback) def logoutCallback(self, d): reactor.stop() def connectionLost(self, reason): imap4.IMAP4Client.connectionLost(self) if debug: print "connectionLost" def sendLine(self, line): imap4.IMAP4Client.sendLine(self, line) if debug: print line def lineReceived(self, line): imap4.IMAP4Client.lineReceived(self, line) if debug: print line class MyIMAP4ClientFactory(protocol.ClientFactory): protocol = MyIMAP4Client f = MyIMAP4ClientFactory() reactor.connectTCP("server", 143, f) reactor.run() So how'd I do for a Twisted newbie? Am I on the path to enlightenment? Are there any obvious errors in what I've interpreted so far? Is this a "good" implementation for this feature or is their a more Twisted-ish approach I need to strive for? Am I correct in assuming that the IMAP4Client documentation needs some buffing up? (Maybe I can help with that.) I didn't look at any source code (other than in the HOWTOs) in implementing this so I think that's a good sign. But as I mentioned above, I'm not clear on how I know what the signature for my callbacks should be or what the parameters to those callbacks mean in some cases. Also, it's not obvious to me how protocol implementors decide when to have users override a method versus having them use callbacks. I know I'm working on client code but knowing this might help me know how to use their code. Is it a matter of personal preference? The ircLogBot example didn't have any callbacks using deferreds but it looks like I had no choice when using IMAP4Client. Thanks for reading this far! -- Jason [1] http://twistedmatrix.com/documents/current/api/twisted.protocols.imap4.IMAP… [2] http://twistedmatrix.com/documents/current/howto/clients [3] http://twistedmatrix.com/documents/current/howto/defer.html [4] http://twistedmatrix.com/documents/current/howto/reactor-basics.html [5] http://twistedmatrix.com/documents/TwistedDocs/TwistedDocs-1.2.0/api/twiste… [View Less]

4 7

[Twisted-Python] Custom app using twisted
by Duncan McGreggor May 16, 2004

May 16, 2004

Don't know if this belongs on the list or not... I've just written a python monitoring program for baby-sitting a bunch of servers (I wanted something easier to setup and maintain than Nagios)... but the inevitable has happened, and I want to make a go-cart run like a ferrari. I was thinking of using python's interface to 'select' a la Medusa, but as soon as I saw 'import asyncore', I thought "There's got to be a better way..." Enter twisted. I would like to be able to run n monitoring … [View More]

2 2

[Twisted-Python] Simple app pickle issue when stopping
by Alexander May May 15, 2004

May 15, 2004

Still trying to learn/understand twisted. Mind exploding/getting frustrated. I created this very simple app: #!/usr/bin/python from twisted.application import internet, service from twisted.web import resource, server class SimpleResource(resource.Resource): def __init__(self): resource.Resource.__init__(self) self.isLeaf=1 def render_GET(self, request): return 'simple' application = service.Application('server') serviceCollection = service.IServiceCollection(application) … [View More]internet.TCPServer(8080,server.Site(SimpleResource())).setServiceParent(serv iceCollection) Then I run "twistd -y scratch.py" and it starts up correctly. Great. I can go to http://localhost:8080 and see the web page "simple". Wonderful. I hit control^C to stop the server and I get pickling errors. Not good. Call stack listed below: IF I go into the python interactive environment and import sratch (and pickle), I can instantiate SimpleResrouce and pickle it with no problem. Help! Alex Call stack: C:\Documents and Settings\Alexander May\My Documents\maygold-dev\central-server>twistd -y scratch.py 2004/05/15 14:51 Eastern Daylight Time [-] Log opened. 2004/05/15 14:51 Eastern Daylight Time [-] twistd 1.2.0 (C:\Python23\python.exe 2.3.3) starting up 2004/05/15 14:51 Eastern Daylight Time [-] reactor class: twisted.internet.default.SelectReactor 2004/05/15 14:51 Eastern Daylight Time [-] twisted.web.server.Site starting on 8080 2004/05/15 14:51 Eastern Daylight Time [-] Starting factory <twisted.web.server.Site instance at 0x00C52B70> 2004/05/15 14:51 Eastern Daylight Time [HTTPChannel,0,127.0.0.1] 127.0.0.1 - - [15/May/2004:18:51:26 +0000] "GET / HTTP 1.1" 200 6 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" 2004/05/15 14:51 Eastern Daylight Time [-] (Port 8080 Closed) 2004/05/15 14:51 Eastern Daylight Time [-] Stopping factory <twisted.web.server.Site instance at 0x00C52B70> 2004/05/15 14:51 Eastern Daylight Time [-] Saving server application to server-shutdown.tap... 2004/05/15 14:51 Eastern Daylight Time [-] Traceback (most recent call last): File "C:\Python23\Lib\site-packages\twisted\internet\defer.py", line 227, in callback self._startRunCallbacks(result) File "C:\Python23\Lib\site-packages\twisted\internet\defer.py", line 288, in _startRunCallbacks self._runCallbacks() File "C:\Python23\Lib\site-packages\twisted\internet\defer.py", line 313, in _runCallbacks self.result = callback(self.result, *args, **kw) File "C:\Python23\Lib\site-packages\twisted\internet\base.py", line 317, in _cbContinueSystemEvent self._continueSystemEvent(eventType) --- <exception caught here> --- File "C:\Python23\Lib\site-packages\twisted\internet\base.py", line 325, in _continueSystemEvent callable(*args, **kw) File "C:\Python23\Lib\site-packages\twisted\persisted\sob.py", line 135, in save self._saveTemp(filename, passphrase, dumpFunc) File "C:\Python23\Lib\site-packages\twisted\persisted\sob.py", line 103, in _saveTemp dumpFunc(self.original, f) File "C:\Python23\Lib\site-packages\twisted\persisted\sob.py", line 119, in dumpFunc pickle.dump(obj, file, 1) cPickle.PicklingError: Can't pickle __builtin__.SimpleResource: attribute lookup __builtin__.SimpleResource faied 2004/05/15 14:51 Eastern Daylight Time [-] Main loop terminated. 2004/05/15 14:51 Eastern Daylight Time [-] Server Shut Down. [View Less]

3 4