Mailman 3 February 2006 - Twisted

[Twisted-Python] Threading and twisted
by David Hirschfield 06 Feb '06

06 Feb '06

I have some old server code that runs in multiple threads, and I'm in the process of converting the server to a twisted xmlrpc server. The code basically has multiple worker threads all pulling from a thread-safe queue of requests to process. Threads pull requests at will from the queue and process them, placing results in a thread-safe object which is used to deliver results to clients. The request processing will often block doing I/O or sleep while waiting for processes to complete, which is why they run in threads. My question is: can I keep the threaded system as-is and just use the xmlrpc system to add requests to the queue and pull results from it to deliver to clients, as long as the xmlrpc calls maintain thread safety when dealing with the shared queue and results object? Or, is it generally unsafe to use non-twisted threading in conjunction with a twisted app? Should I be using twisted's own thread pool and runInThread() system to do the request processing work? It probably won't be too much work to change over the current code to use twisted's threading framework...but I'm just wondering where I might run into trouble if I use the current setup instead. Am I endangering the xmlrpc server's ability to respond to requests if I have threads that aren't under twisted's control? Thanks for your help, -Dave -- Presenting: mediocre nebula.

2 1

[Twisted-Python] Re: Making DeferredList look like Deferred
by Lenny G Arbage 06 Feb '06

06 Feb '06

Actually, I suppose that, if code == False: # assumes failed results have # (False, failure) pairs, and # only returns the first of # these. return res could better be, if code == False: raise failure.DefaultException(result) (so that the errback gets the full list). Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

[Twisted-Python] Help: how to run something at shutdown time
by David Hirschfield 06 Feb '06

06 Feb '06

I'm running an xmlrpc server using twisted xmlrpc. I use reactor.callWhenRunning() to do some setup when the server is started, but I need to also do some special shutdown steps when the server is stopped. I don't see any callWhenStopped() or something to that effect...so is there a way to call specific methods when shutdown happens? Any help is appreciated, -David -- Presenting: mediocre nebula.

2 2

[Twisted-Python] Re: Making DeferredList look like Deferred
by Lenny G Arbage 06 Feb '06

06 Feb '06

Thomas HERVE therve at free.fr wrote: > The behaviour you want is provided by the > option 'consumeErrors' of DeferredList > (you can see why here : > http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto8). Aha, that's the piece I was missing! Thank you for pointing it out. Now, though, this leads me to one more question: what if I want to wait until all the Deferreds have fired (either err or callback), and only then call the callback if all callbacks were fired, otherwise call the errback? To accomplish this, I'm currently just 'wrapping' the DeferredList in a regular Deferred. It's callback is always called on completion, so when it fires I look at all the results given to the callback, and if any have failed, raise an exception. Otherwise I return a result. This feels a little bit clunky, but I don't have a good suggestion for improving it -- 'fireOnErrbackAfterAllCompleted=True'? Thanks, Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] mx.DateTime jellyable?
by Gabe Rudy 06 Feb '06

06 Feb '06

I see the support of the python datetime class, and read through the pb-copyable howto, but am still curious if there is a way to make a module classes jellyable (specifically mx.DateTime) without writting a wrapper class? My postgres database wrapper returns mx.DateTime and a lib on the client side expects mx.DateTime so it makes sense to keep it in that format. I saw a message back in 2004 about working on the ability to register external serialization functions for new types. Is that in place now? I have seen some chatter in previous posts about mx.DateTime but could not find a definitive answer. i.e. "don't hack it, make it work without mx.DateTime", or "philosophically, what your doing would be insecure and not cross-language compatible (which pb is), hence your question is void.", or hopefully "check out xxx for external serialization" Thanks --gabe

1 0

[Twisted-Python] Re: Making DeferredList look like Deferred
by Lenny G Arbage 06 Feb '06

06 Feb '06

Thomas HERVE therve at free.fr wrote: >> def iterfail(err, i): >> print " %i failed" % i > > The problem is here. You add an errback to your defer, > but you stop the error propagation by not returning > err. Either do 'return err' or not > 'addErrback(iterfail)'. Thomas, thanks for the response. I thought that might be it, but no avail. If I 'return err' in iterfail, I get: in testdlist deferred will fail in 2s deferred will succeed in 2s 0 failed failed: testdlist Unhandled error in Deferred: Traceback (most recent call last): Failure: <type 'bool'>: True 1 succeeded If I instead comment out 'addErrback(iterfail,i)', I get: in testdlist deferred will fail in 2s deferred will succeed in 2s failed: testdlist Unhandled error in Deferred: Traceback (most recent call last): Failure: <type 'bool'>: True 1 succeeded I don't know why the error is unhandled in either case -- the DeferredList has an errback set. (If I raise a new error in iterfail, the same result occurs, naturally). It also doesn't seem to matter whether I add 'fireOnOneErrBack' to the DeferredList constructor or not. Any ideas? Does this code behave differently on your machine? Thanks, Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] Re: [Twisted-commits] r15836 - Start adding setMetadata to twisted.vfs, including the SFTP adapter and osfs backend.
by Jean-Paul Calderone 06 Feb '06

06 Feb '06

On Mon, 06 Feb 2006 02:22:43 -0700, Andrew Bennetts <spiv(a)wolfwood.twistedmatrix.com> wrote: >Author: spiv >Date: Mon Feb 6 02:22:43 2006 >New Revision: 15836 > >Modified: > trunk/LICENSE > trunk/twisted/vfs/adapters/sftp.py > trunk/twisted/vfs/backends/osfs.py > trunk/twisted/vfs/ivfs.py > trunk/twisted/vfs/test/test_sftp.py >Log: >Start adding setMetadata to twisted.vfs, including the SFTP adapter and osfs backend. > >This work is contributed by Canonical Ltd -- I've updated the >list of copyright holders in LICENSE to reflect this. > This commit introduced a new failure on the Win32 buildslave. test_setAttrs fails like so: [ERROR]: twisted.vfs.test.test_sftp.SFTPAdapterOSFSTest.test_setAttrs File "c:\buildslave\win32-select\W32-full2.4-select\Twisted\twisted\vfs\test\test_sftp.py", line 136, in test_setAttrs self.sftp.setAttrs('/ned', {'mtime': mtime}) File "c:\buildslave\win32-select\W32-full2.4-select\Twisted\twisted\vfs\adapters\sftp.py", line 190, in setAttrs node.setMetadata(attrs) File "c:\buildslave\win32-select\W32-full2.4-select\Twisted\twisted\vfs\backends\osfs.py", line 57, in setMetadata os.utime(self.realPath, (atime, mtime)) exceptions.OSError: [Errno 13] Permission denied: 'twisted.vfs.test.test_sftp\\SFTPAdapterOSFSTest\\test_setAttrs\\ynojwi\\temp\\ned'

1 0

[Twisted-Python] Re: Making DeferredList look like Deferred
by Lenny G Arbage 06 Feb '06

06 Feb '06

Jean-Paul Calderone exarkun at divmod.com Fri Feb 3 14:08:54 MST 2006 wrote: > > ...However, if > you pass a true value for fireOnOneErrback when > constructing your DeferredList, any Deferred which > errbacks will cause the DeferredList to errback; in > this case it will never callback. I think I must be missing something, but I can't spot it. Here's the smallest example I've come up with: import random from twisted.internet import reactor, defer from twisted.python import failure def error(err, msg): print "failed: %s" % msg reactor.stop() def success(result, msg): print "succeeded: %s" % msg reactor.stop() def defer2seconds(prob=0.5): d = defer.Deferred() if random.random() < prob: print " deferred will succeed in 2s" reactor.callLater(2, d.callback, True) else: print " deferred will fail in 2s" reactor.callLater(2, d.errback, True) return d def deferlist2seconds(prob=0.25, num=2): def itersuccess(result, i): print " %i succeeded" % i return result def iterfail(err, i): print " %i failed" % i dlist = [] for i in range(num): d = defer2seconds(prob) d.addCallback(itersuccess, i) d.addErrback(iterfail, i) dlist.append(d) dl = defer.DeferredList(dlist, fireOnOneErrback=True) return dl def testdlist(): print "in testdlist" #d = defer2seconds(prob=0.5) d = deferlist2seconds(prob=0.5) d.addCallback(success, "testdlist") d.addErrback(error, "testdlist") if __name__ == "__main__": testdlist() reactor.run() The action here is really simple, with each deferred randomly choosing to succeed or fail. The testdlist function can call either the simple, single-deferred test or the multi-deferredlist test. The goal here is that the caller doesn't care what the underlying function call does, in terms of Deferred v. DeferredList. When calling the simple test, the results are as expected: in testdlist deferred will fail in 2 seconds failed: testdlist in testdlist deferred will succeed in 2 seconds succeeded: testdlist But when calling the DeferredList test, it never gets into the DeferredList's errback. in testdlist deferred will succeed in 2 seconds deferred will succeed in 2 seconds 0 succeeded 1 succeeded succeeded: testdlist in testdlist deferred will fail in 2 seconds deferred will succeed in 2 seconds 0 failed 1 succeeded succeeded: testdlist Why isn't the DeferredList's errback being called in this case? It seems that no matter what I do (remove the individual Deferreds' errbacks, change fireOnOneErrback, etc), the DeferredList's callback only is called. This doesn't sound like what you'd expect. What am I doing wrong? What must I do to get the DeferredList's errback to fire when any single Deferred in the list has its errback called? Tested using twisted 2.1. Thanks, Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] GUI application with Conch
by Brendan Simon 06 Feb '06

06 Feb '06

Hi, I want to write a gui application (wxPython) that runs on MSW and talks to a Linux server that has an application that only runs on Posix based systems. The current gui app runs on the Linux server and executes the commands directly using popen, but I want to make it flexible enough so that MSW users can run a local gui which will execute the commands on the Linux server and display the results locally. My question is one of architectural design. How is it best to structure the program so that it can use a remote shell connect (eg. ssh, or possibly other transports such as rsh, or rsh over a vpn, etc) as well as using direct calls to the utilities when running directly on the server ??? I've managed to get the sshsimpleclient.py going and talking to my linux server with my login and rsa keys :) I notice that this sample is different to the documentation mentioned here: http://twistedmatrix.com/projects/conch/documentation/howto/conch_client.ht… The sshsimpleclient.py does not have the following statements: factory = protocol.ClientFactory() factory.protocol = ClientTransport reactor.connectTCP('localhost', 22, factory) reactor.run() Is one better than the other? Is one method better going forward into the future? Also, the sshsimpleclient.py seems to have the closed() method called when issuing a command (eg. CatChannel, TrueChannel, etc). How can I keep a channel open to run commands when I want ?? Cheers, Brendan.

1 0

[Twisted-Python] xmlrpc and deferred result doubts
by Gustavo Rahal 05 Feb '06

05 Feb '06

Hi All I have a doubt about how to factor the an xmlrpc server code. Basicly what happens is: def xmlrpc_runCmd(self, hostname, command): # I call a method that returns a deferred and because of that I # don't have anything to return to the xmlrpc client. runComm = conn.runCommand(hostname, command) # the line below doesn't make much sense because the output needs to # return to the xmlrpc client runComm.addCallback(self.printOutput) return ???? # return what?!?!?!? What should I do? How should I factor the code? Thanks Gustavo

2 1