Mailman 3 October 2004 - Twisted

[Twisted-Python] Case for an Async MySQL Driver?
by L. Daniel Burr 29 Oct '04

29 Oct '04

Seun Osewa wrote: >Here are some ideas for an async MySQL 'driver' for Twisted: [snip description of API] >This format allows the user to send an unlimited number of queries to >be run together, thus limiting the number of callbacks necessary for a >program requiring many small interactions with the database. "begin" >and "end" will count as an example. This minimalistic API also allows >more advanced (twisted.enterprise style) interfaces to be built on top >of it. I think that this is a good idea, but I think it would be far better if you described your API in terms of the twisted.enterprise.adbapi API which exists today. Example: from twisted.enterprise import adbapi dbpool = adbapi.NonBlockingConnectionPool( MySQLProtocolFactory, connectionInfo, cp_min=1, cp_max=5 ) queries = ('SELECT 1 FROM DUAL', 'UPDATE Foo WHERE bar = baz') for query in queries: d = dbpool.runQuery(query) d.addErrback(_onFailure) d.addCallback(_onSuccess) The effort needed to make this proposed asynchronous db protocol work with adbapi would be pretty minor, I think. Tell you what: If somebody implements this MySQL async protocol, I'll create a patch to make adbapi work with it. Sound good? Naz

1 0

[Twisted-Python] FTPClient and fetching a file from an FTP server
by Gaspard, Bradley S 28 Oct '04

28 Oct '04

I've looked at the ftpclient.py example and have also found an example of moving a file to an ftp server on this list ... but not how to fetch a file from the server. From the examples and the API it looks like the retrieveFile( ) method and a callback are required but I have not yet put the pieces together. I am very much a newbie and would be grateful for any pointers on how to do this.

1 0

[Twisted-Python] IMAP fetchUID speed?
by Nick Arnett 27 Oct '04

27 Oct '04

I'm trying to figure out what's happening in an IMAP implementation. The server at the moment has a very big backlog -- perhaps 80K messages. I was expecting that if I did this: d.addCallback(lambda x: self.proto.fetchUID('1:10')) things would move along quickly... but there's a huge delay somewhere. I think it's in that callback, since I've added breakpoints in handleuids() and they're not being reached (as they are when I hit a smaller mailbox on a different server). Thanks for any pointers... I'm brand-new to Twisted, but sure like what I'm seeing in it. Nick P.S. I'm using the same sort of approach that John Goerzen's "Python Network Programming" book shows: > class IMAPClient(IMAP4Client): > def connectionMade(self): > IMAPLogic(self) > > class IMAPFactory(protocol.ClientFactory): > protocol = IMAPClient > def __init__(self, username, password): > self.username = username > self.password = password > > def clientConnectionFailed(self, connector, reason): > LOGGER.error("Client connection failed:", reason) > reactor.stop() > > class IMAPLogic(mail_archive_retriever): > def __init__(self, proto, getting_archives = 0): > self.parser = email.Parser.Parser() > self.request_topic_id = None > self.message_bucket = messagedb.message_inserter() > self.getting_archives = getting_archives > > # Twisted stuff > self.proto = proto > self.factory = proto.factory > d = self.proto.login(self.factory.username, self.factory.password) # d is a Deferred > d.addCallback(lambda x: self.proto.select('INBOX')) # lambda here keeps the output of the previous callback from going into this call > d.addCallback(lambda x: self.proto.fetchUID('1:*')) > d.addCallback(self.handleuids) > d.addCallback(self.deletemessages) > d.addCallback(self.logout) > d.addCallback(self.stopreactor) > d.addErrback(self.errorhappened)

2 1

[Twisted-Python] twisted run out of file descriptors
by Alec Matusis 27 Oct '04

27 Oct '04

Hi I am running twisted on 2.4.21 Kernel. My chat server has been puking and dying the last couple days and it seemed pretty clear that it was dying when it was reaching 1024 open file descriptors. I noticed the default ulimit -n was 1024, so I added a 'ulimit -n 4096' before starting the chat server. That has apparently allowed it to surpass 1024 descriptors (it has 1263 at the moment) but now it is failing to call select (see sample errors below). There is this value FD_SETSIZE in the C headers that determines the maximum number of sockets a proc can select() on, and it's set to 1024. It it possible to increase this limit, short of recompiling the Kernel? Sample Errors: ----- File "/usr/local/encap/Twisted-1.2.0/lib/python2.3/site-packages/twisted/ internet/default.py", line 487, in doSelect [], timeout) exceptions.ValueError: filedescriptor out of range in select() 2004/10/25 16:35 PDT [-] Malformed file descriptor found. Preening lists. 2004/10/25 16:35 PDT [-] bad descriptor Thanks Alec matusis

8 13

[Twisted-Python] CFReactor
by James Y Knight 27 Oct '04

27 Oct '04

Itamar just added half-closing support to Twisted, and I foolishly volunteered to look at adding support to cfreactor for him. However, that was before I looked at it. Now that I have, I have little idea what it's actually doing, so I'm afraid that I'm unable to actually implement this change. The "SelectableSocketWrapper" looks like a horrible kludge, and I'm not sure why it's necessary. Why does it override the wrapped object's "connectionLost"? Why do you have to wrap&override Selectable.start/stopReading/Writing, instead of doing that stuff in reactor.add/removeReader/Writer? What is simulate?? Essentially, I just don't understand why it is so different from the other reactors. Either hints or else a working half-close implementation for cfreactor would be appreciated. James

2 4

[Twisted-Python] Case for an Async MySQL Driver?
by Seun Osewa 27 Oct '04

27 Oct '04

Hi, I'm new to Twisted, and I was just thinking about how, while 'twisted.enterprise.adbapi' is is a really neat idea, it's sort of a stop-gap solution pending the time async database drivers can be integrated into Twisted. It's not 'right' to have 1 python thread and 1 mysql server thread/process for each connection. How about an event-driven MySQL protocol implementation that fits into the Twisted Framework? (I came accross a MySQL protocol specification online once but I think I've lost it). If this sounds sensible, I'll be interested with helping out with implementation but I won't be able to do actual coding for some time. Regards, Seun.

5 5

[Twisted-Python] reactor.stop() won't, threads and Queue to blame?
by Brett Viren 26 Oct '04

26 Oct '04

Hi, I can't seem to make reactor.stop() actually stop the reactor and allow my program to exit. I hope someone can help. What I'm actually trying to do is implement an execution pipeline so I can serialize certain jobs but still run them in a thread so the main loop can go about its business. I do this with a class, CommandQueue, that has a Queue.Queue and a method that can be called via reactor.callInThread() that slurps the queue and runs any callables sent down it. Each callable's result is obtained from a Deferred. This thread loops until I set CommandQueue.stop=True. (Maybe someone knows a better way?) To try to triger the shutdown, I add a callback to the Deferred associated with the final callable sent down the queue. If I instead call shutdown function via reactor.callLater() then the reactor will actually stop. I don't know if it matters, but /bin/ps only shows one instance of python running. Usually I see one instance per thread. The code below and its output shows the problem. #!/usr/bin/env python2.3 from twisted.python import threadable threadable.init(1) from twisted.internet import reactor,defer from Queue import Queue,Empty class CommandQueue: '''Queue up commands for serial calling. One must call the drain() method to start reading the internal queue. Most likely one wants to call this in a thread.''' def __init__(self): "Create a CommandQueue" self.queue = Queue() self.stop = False return def __call__(self,meth,*a,**k): '''Call meth(*a,**k) when it reaches end of queue. Returns a Deferred that will pass the return of meth.''' deferred = defer.Deferred() self.queue.put((deferred,meth,a,k)) return deferred def drain(self): 'Drain the command queue until CommandQueue.stop is True' while not self.stop: try: d,meth,a,k = self.queue.get(True,1) except Empty: print " queue empty" continue print "calling %s(%s,%s)"%(meth.__name__,str(a),str(k)) d.callback(meth(*a,**k)) print "callback done" print "drain closing" return 0 def test1(): import time cq = CommandQueue() reactor.callInThread(cq.drain) def shutdown(x=None): print "Stopping CommandQueue" cq.stop = True print "Stopping reactory" reactor.stop() print "reactor.stop()'ed" def burp(x): for n in range(0,x): time.sleep(1) print x,n return x def chirp(x): print "okay:",x return None def ouch(x): print "bad:",x return x last = 3 for n in range(0,last): print "dispatching",n d = cq(burp,n).addCallbacks(chirp,ouch) if last-n == 1: d.addCallbacks(shutdown,ouch) if __name__ == '__main__': print "running test1" test1() print "end test1" reactor.run() print "reactor exitted" #=-------------- end ------------=# Running this produces the following: [i386]bviren@aviator:test> ./test-commandqueue.py running test1 dispatching 0 dispatching 1 dispatching 2 end test1 calling burp((0,),{}) okay: 0 callback done calling burp((1,),{}) 1 0 okay: 1 callback done calling burp((2,),{}) 2 0 2 1 okay: 2 Stopping CommandQueue Stopping reactory reactor.stop()'ed callback done drain closing At which point I have to hit Control-C and finally get: reactor exitted Here is the mod to use reactor.callLater(): ... last = 3 for n in range(0,last): print "dispatching",n d = cq(burp,n).addCallbacks(chirp,ouch) #if last-n == 1: # d.addCallbacks(shutdown,ouch) reactor.callLater(10,shutdown) ... So, any ideas as to what I'm doing wrong? Thanks, -Brett.

4 7

[Twisted-Python] an alternative to epoll, perhaps? --> libevent
by Jonathan Simms 24 Oct '04

24 Oct '04

I was talking with hagbard, who is good friends with the guy who wrote libevent (and is listed in the ssh(1) man page, Niels Provos), and he (hagbard) suggested that it might be useful in writing a reactor. Apparently it provides a portable compatability layer on top of whatever native event-notification system there is. just thought it was worth mentioning. -Jonathan

1 0

[Twisted-Python] ANN: deferred howto/tutorial
by Jonathan Simms 24 Oct '04

24 Oct '04

Hello all, I was sitting at Jury Duty today, trying to keep myself from the inevitable thoughts of suicide, and I started thinking about how to implement deferreds in C#. I started out by thinking about the logic one would need to use to implement deferreds in python. I wrote out in my notebook an implementation and when I got back home, I was suprised to discover that I wasn't terribly far off. I reflected upon this, and about what had confused me for so long about deferreds in general. ...and I was really stumped... ;) Deferreds are simple and elegant. After rewriting trial and dealing with all of its crazy exception handling, I realised that a lot of that code could have been made more succinct (if not more understandable) had it used deferreds to do the error handling and decision making. I think the thing that confuses most newbies is that deferreds to them seem like this BLACK BOX. It's this mysterious thing that is going to CALL YOUR CODE, and the rules about chaining and errback and callback chains are quite intimidating if you don't have a chunk of familiar looking code to transition you from python to Deferredy-Looking-Kool-Aid-Land. So what i did was write a tutorial on my deferred enlightenment...(pun intended?) I'd appreciate feedback, good or bad. You can find it in svn:HEAD, at doc/core/howto/tutorial/deferred_tutorial.xhtml Cheers! Jonathan D. Simms slyphon AT twistedmatrix DOT com

7 15

[Twisted-Python] two deferred_tutorial patches
by Lee Harr 24 Oct '04

24 Oct '04

The first catches a bit more of the output from deferred_ex3.py which makes a bit more sense: Index: doc/core/howto/tutorial/deferred_tutorial.xhtml =================================================================== --- doc/core/howto/tutorial/deferred_tutorial.xhtml (revision 12146) +++ doc/core/howto/tutorial/deferred_tutorial.xhtml (working copy) @@ -157,6 +157,9 @@ File "./deferred_ex2.py", line 35, in failAtHandlingResult raise RuntimeError, "whoops! we encountered an error" exceptions.RuntimeError: whoops! we encountered an error + +callback 3 + got result: okay, continue on </pre> <h2>addBoth: the deferred version of finally</h2> Actually, now that I look at it ... the output in the text is just from the wrong program: Index: doc/core/howto/tutorial/deferred_tutorial.xhtml =================================================================== --- doc/core/howto/tutorial/deferred_tutorial.xhtml (revision 12146) +++ doc/core/howto/tutorial/deferred_tutorial.xhtml (working copy) @@ -147,16 +147,22 @@ <pre> callback 1 got result: success +callback 2 + got result: yay! handleResult was successful! about to raise exception errback we got an exception: Traceback (most recent call last): ---- <exception caught here> --- - File "/home/slyphon/Projects/Twisted/trunk/twisted/internet/defer.py", line -326, in _runCallbacks - self.result = callback(self.result, *args, **kw) - File "./deferred_ex2.py", line 35, in failAtHandlingResult + File "deferred_ex3.py", line 94, in ? + behindTheScenes("success") +--- <exception caught here> --- + File "deferred_ex3.py", line 57, in behindTheScenes + result = failAtHandlingResult(result) + File "deferred_ex3.py", line 34, in failAtHandlingResult raise RuntimeError, "whoops! we encountered an error" exceptions.RuntimeError: whoops! we encountered an error + +callback 3 + got result: okay, continue on </pre> <h2>addBoth: the deferred version of finally</h2> And the second one makes the tutorial look right when loaded from the filesystem. I have a feeling I am supposed to start a webserver or process these files somehow, but this was easier for me... Index: doc/core/howto/tutorial/deferred_tutorial.xhtml =================================================================== --- doc/core/howto/tutorial/deferred_tutorial.xhtml (revision 12146) +++ doc/core/howto/tutorial/deferred_tutorial.xhtml (working copy) @@ -1,6 +1,7 @@ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <html> <head> <title>Deferreds are beautiful! (A Tutorial)</title> +<link type="text/css" rel="stylesheet" href="../stylesheet.css" /> </head> <body> @@ -27,7 +28,7 @@ First, a simple example so that we have something to talk about: -<a href="../listings/deferred/deferred_ex.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex.py" class="py-listing">deferred_ex.py</a> And the output: (since both methods in the example produce the same output, it will only be shown once.) @@ -56,7 +57,7 @@ returned the deferred needs to alert the callback chain that an error has occurred. -<a href="../listings/deferred/deferred_ex1a.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex1a.py" class="py-listing">deferred_ex1a.py</a> <pre> errback @@ -80,7 +81,7 @@ Now let's see what happens when our callback raises an exception -<a href="../listings/deferred/deferred_ex1b.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex1b.py" class="py-listing">deferred_ex1b.py</a> And the output: (note, tracebacks will be edited slightly to conserve space) @@ -112,7 +113,7 @@ errback in the chain. If the chain is started off with a failure, the first method to be called will be the first errback. -<a href="../listings/deferred/deferred_ex2.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex2.py" class="py-listing">deferred_ex2.py</a> <pre> callback 1 @@ -142,7 +143,7 @@ failure or raise an exception) the chain continues on with the next callback. -<a href="../listings/deferred/deferred_ex3.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex3.py" class="py-listing">deferred_ex3.py</a> <pre> callback 1 @@ -168,7 +169,7 @@ returning the value that it received to allow processing the chain to continue, but appearing transparent in terms of the result. -<a href="../listings/deferred/deferred_ex4.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex4.py" class="py-listing">deferred_ex4.py</a> <pre> callback 1 @@ -205,7 +206,7 @@ addCallback followed by an addErrback. -<a href="../listings/deferred/deferred_ex5.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex5.py" class="py-listing">deferred_ex5.py</a> <pre> callback 1 @@ -227,7 +228,7 @@ processing in response to the error, we would have used a passthru, and returned the failure we received, as we see in this next example: -<a href="../listings/deferred/deferred_ex6.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex6.py" class="py-listing">deferred_ex6.py</a> <pre> callback 1 @@ -258,7 +259,7 @@ beforehand and then executes it, the other executes the chain as it is being constructed. This is because deferreds are stateful. -<a href="../listings/deferred/deferred_ex7.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex7.py" class="py-listing">deferred_ex7.py</a> <pre> callback 1 @@ -309,7 +310,7 @@ processing chain will become the result that gets passed to the next callback of the primary deferreds processing chain -<a href="../listings/deferred/deferred_ex8.py" class="py-listing"></a> +<a href="../listings/deferred/deferred_ex8.py" class="py-listing">deferred_ex8.py</a> <pre> callback 1 Thanks for your time. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/

2 1