Mailman 3 February 2009 - Twisted

[Twisted-Python] Why do I get an AlreadyCalledError?
by Conrad Winchester 28 Feb '09

28 Feb '09

Hi all, I am writing a restful twisted (8.1.0) server and am trying to return useful error statuses from requests. The problem is that even following the documentation extremely carefully I am getting an AlreadyCalled Error every time I generate a failure. Here is my main class (leaving out code that I feel is not relevant to the question: from xml.dom import minidom from twisted.web import server, resource from FWGException import FWGException class DeferredPostHandler(resource.Resource,minidom.Document): NOT_IMPLEMENTED = 501 BAD_REQUEST = 400 def writeError(self, failure, request, *args, **kw): failure.trap(FWGException) request.setResponseCode(self.errorCode) request.write("Error handling request: %s" % failure.getErrorMessage()) request.finish() def writeResult(self, result, request, *args, **kw): self.writexml(request) request.finish() def render_GET(self, request): self.process=self.db.runInteraction(self.processForm,request) self.process.addCallbacks(self.writeResult, self.writeError, [request], {}, [request], {}) return server.NOT_DONE_YET I have a handler that extends this from DeferredPostHandler import DeferredPostHandler from twisted.web import http from twisted.python.failure import Failure from FWGException import FWGException class CollectionAdminHandler(DeferredPostHandler): def __init__(self, task, db, config): DeferredPostHandler.__init__(self,db) self.config = config self.task = task self.errorCode = http.OK def processForm(self, cursor, request): if self.task == 'add': self.addCollection(cursor, request) elif self.task == 'delete': self.deleteCollection(cursor, request) else: self.errorCode = DeferredPostHandler.NOT_IMPLEMENTED fail = Failure(FWGException("Unsupported Task")) self.process.errback(fail) def addCollection(self, cursor, request): fail = None if self.isValidAddRequest(request): pass else: self.errorCode = DeferredPostHandler.BAD_REQUEST fail = Failure(FWGException("Missing Parameter")) if fail: self.process.errback(fail); def deleteCollection(self, cursor, request): print 'Delete Collection' def isValidAddRequest(self, request): return 'title' in request.args and 'category' in request.args and 'date' in request.args Unfortunately whenever I call self.process.errback(fail) I get the following stack trace in my log Traceback (most recent call last): File "startserver.py", line 23, in <module> reactor.run() File "/Library/Python/2.5/site-packages/Twisted-8.1.0-py2.5- macosx-10.5-i386.egg/twisted/internet/base.py", line 1048, in run self.mainLoop() File "/Library/Python/2.5/site-packages/Twisted-8.1.0-py2.5- macosx-10.5-i386.egg/twisted/internet/base.py", line 1057, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/Library/Python/2.5/site-packages/Twisted-8.1.0-py2.5- macosx-10.5-i386.egg/twisted/internet/base.py", line 677, in runUntilCurrent f(*a, **kw) File "/Library/Python/2.5/site-packages/Twisted-8.1.0-py2.5- macosx-10.5-i386.egg/twisted/internet/defer.py", line 243, in callback self._startRunCallbacks(result) File "/Library/Python/2.5/site-packages/Twisted-8.1.0-py2.5- macosx-10.5-i386.egg/twisted/internet/defer.py", line 298, in _startRunCallbacks raise AlreadyCalledError twisted.internet.defer.AlreadyCalledError: I can not understand why this is happening :-( Can anybody help? BTW I am using POST instead of PUT because my client can't do PUT Conrad Winchester

2 3

[Twisted-Python] server client in one code
by khawar hasham 26 Feb '09

26 Feb '09

I am trying to use twisted to build an application that can act as server as well as client at the same time. I need this application to send and receive messages back and forth between two of its instances. I used the reactor.run() to start listen and connectTCP. My problem is that I need to run server or client module from other program. Moreover, I cannot put the calling module on hold because reactor.run() stops every other execution of your code. To make it work, I started the reactor.run in a separate thread with installSignalHanlder=0. I have got the success to transfer messages between two instances if one is acting as server and other as client. However, I could not send messages if both instances started as servers and then start sending the messages to each other. I tried to google around but could not find the solution. Can you help me in this? I can not start server as independent module. I have to start it from other python script that will use my module to act as server as well as client to send and receive the messages.

1 0

Re: [Twisted-Python] What to do when a service fails to start, also, deferred and startService
by Terry Jones 26 Feb '09

26 Feb '09

Following up on myself.... Here's an updated/working version of the MultiService subclass I proposed the other day: class GuardedMultiService(service.MultiService): def __init__(self, guardService): service.MultiService.__init__(self) self.guardService = guardService self.startDeferred = defer.Deferred() self.stopDeferred = defer.Deferred() def startService(self): def startOthers(_): log.msg("Guard service startup succeeded. Starting other services.") service.MultiService.startService(self) def initFailed(f): log.msg("Guard service startup failed.") return f d = defer.maybeDeferred(self.guardService.startService) d.addCallbacks(startOthers, initFailed) d.chainDeferred(self.startDeferred) def stopService(self): d = service.MultiService.stopService(self) d.addCallback(lambda _: self.guardService.stopService()) d.chainDeferred(self.stopDeferred) return self.stopDeferred This is pretty much as I described earlier. I've added self.startDeferred and self.stopDeferred deferreds. These give the programmer using this class (who, I imagine, is probably doing so in the context of writing a Twistd plugin - i.e., writing the makeService method in a class implementing IServiceMaker and IPlugin) the chance to add callbacks to the chain of events that happen when the service is started and stopped. Their use is optional. It's easy to see how to make guardService default to None, which would make it possible to merge the above into MultiService itself. If there's any interest at all in this, I'm happy to open a ticket, run tests, submit a patch, etc. Terry

1 0

[Twisted-Python] Development Process Failure
by Jean-Paul Calderone 23 Feb '09

23 Feb '09

Hey all, Recently I've been bothered by a systematic shortcoming of the Twisted development process. When changes are made in response to a review, they are generally made in a way which is difficult to inspect. The value of a review is lost if the valid points raised by it are not addressed before the changes are applied to trunk. Often, there's no problem here, but it sometimes happens that /not/ all of the review points are addressed before a ticket is re-submitted for review. In these cases, one of two things happens. The re-reviewer might go back to the previous review and verify that all of the points raised in it have been addressed. This is a time consuming process, though it can be made easier if the author responds to each review point (and even easier if a changeset is linked in each response). Or, the re-reviewer might assume that the author addressed all points and just look over the branch for anything missed in the previous review. In either of these cases, time is being wasted, but in the latter case, time is being wasted /and/ bad code is being added to trunk. I would like to consider how we might address this shortcoming of the development process. One possibility is to explicitly adjust the review guidelines and direct reviewers always to verify that previous review points have actually been addressed. What ideas do other people have? Jean-Paul

3 4

[Twisted-Python] Debugging Twisted
by Luper Rouch 23 Feb '09

23 Feb '09

I would like to investigate on the conch error I mentionned in my previous message ("KeyError when an errback is triggered within a conch TCP tunnel"). My problem is Twisted catches the exceptions and the debugger won't stop on them, how could I avoid that ? Is there a "debug" reactor ?

3 6

[Twisted-Python] KeyError when an errback is triggered within a conch TCP tunnel
by Luper Rouch 23 Feb '09

23 Feb '09

(posting it a second time with the example code inlined as pipermail did not like the attachment) Sorry I have not been able to find a better title, please see the attached example which may be more descriptive. Here is what it does: * set up an SSH connection * open a TCP forwarding channel through this connection * do something through the tunnel that will trigger an errback (in the example, try to do an XMLRPC call with no one listening on the tunnel's end) This scenario gives me the following traceback (tested with Twisted 8.1.0 and 8.2.0): -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/selectreactor.py", line 156, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/posixbase.py", line 193, in _disconnectSelectable selectable.connectionLost(f) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/tcp.py", line 520, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/forwarding.py", line 129, in connectionLost self.channel.loseConnection() File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/channel.py", line 253, in loseConnection self.conn.sendClose(self) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/connection.py", line 491, in sendClose self.channelsToRemoteChannel[channel])) exceptions.KeyError: <twisted.conch.ssh.forwarding.SSHListenClientForwardingChannel instance at 0x972048c> -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- It works well in the other cases, i.e.: * the errback is triggered but the tunnel is not used (comment line 57, uncomment line 59) * the errback is not triggered (start the XMLRPC server by uncommenting line 100), using the tunnel or not You get the error only when the tunnel is used AND the errback is triggered. To run the example execute it with an user name that can login using ssh on your host, e.g. 'python test.py someuser' and enter its password. Cheers, Luper Example code: -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- #!/usr/bin/env python import sys from getpass import getpass from twisted.python import log from twisted.conch.ssh import connection, transport, userauth, keys, \ forwarding, channel from twisted.conch.ssh.common import NS from twisted.internet import defer, protocol, reactor from twisted.web import server, xmlrpc from twisted.web.xmlrpc import Proxy conn = None def connect(user, host, port=22): return reactor.connectTCP(host, port, Factory(user, Transport)) class Factory(protocol.ClientFactory): def __init__(self, user, protocol): self.user = user self.protocol = protocol def buildProtocol(self, addr): p = self.protocol(self.user) p.factory = self return p class Transport(transport.SSHClientTransport): def __init__(self, user): self.user = user def verifyHostKey(self, pubkey, fingerprint): return defer.succeed(1) def connectionSecure(self): self.requestService(UserAuth(self.user, Connection())) class UserAuth(userauth.SSHUserAuthClient): def getPassword(self, prompt=None): return defer.succeed(getpass("password: ")) class Connection(connection.SSHConnection): def __init__(self, *args, **kwargs): connection.SSHConnection.__init__(self, *args, **kwargs) self.forwarding_sockets = [] def serviceStarted(self): # Create a tunnel from 12345 to 54321 self.forward_port(12345, 54321) # Do something through the tunnel p = Proxy('http://localhost:12345/') # passing through the tunnel, # triggers the KeyError #p = Proxy('http://localhost:54321/') # not using the tunnel, works print "ping...", d = p.callRemote("ping") def stop_ok(response): print response conn.disconnect() reactor.stop() def stop_error(error): print "error", error conn.disconnect() reactor.stop() d.addCallback(stop_ok) d.addErrback(stop_error) def serviceStopped(self): # Stop forwarding sockets for socket in self.forwarding_sockets: socket.stopListening() def forward_port(self, local_port, remote_port): print "forwarding %d => %d" % (local_port, remote_port) socket = reactor.listenTCP(local_port, forwarding.SSHListenForwardingFactory(self, ("localhost", remote_port), forwarding.SSHListenClientForwardingChannel)) self.forwarding_sockets.append(socket) class DummyServer(xmlrpc.XMLRPC): def xmlrpc_ping(self): return "pong" if __name__ == "__main__": if len(sys.argv) != 2: print "usage: test.py user" sys.exit(1) #log.startLogging(sys.stdout) # If the server is not running, stop_error() errback is triggered and we # have the KeyError when passing through the tunnel #reactor.listenTCP(54321, server.Site(DummyServer())) conn = connect(sys.argv[1], "localhost") reactor.run() -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

2 1

[Twisted-Python] monitoring twisted services / processes
by Michele - 23 Feb '09

23 Feb '09

Hi, background: I'm currently working on a PoC with twisted, Python, to prove the technology as an alternative to more established enterprise choices (java app servers, etc..). The idea is to build lightweight services to process atomic "chunks" of business processes, that can scale horizontally as much as needed proven that the common persistence layer (a db) scales. The idea is already working in a non python approach, so the attempt is to prove that it's quicker and easier to maintain if built with twisted/python. So, the question is: if I have N number of processes running in a M number of machines, given that there are no network restriction, and that at least http and hhtps are always available, how these services would be efficiently monitored? is there a twisted application/plugin/framework to do so? are you just create 'polling' requests that an existing monitoring tool (i.e. Nagios) parses and interprets? How would you build a "console" to manage services status? In a nutshell, what I'm trying to find out if there is already around a "container-like" twisted based application that can be used to manage/monitor twisted applications provided that they adhere to a given interface Thanks for your help Micc

2 1

Re: [Twisted-Python] What to do when a service fails to start, also, deferred and startService
by Terry Jones 22 Feb '09

22 Feb '09

Hi again Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On 28 Nov, 03:38 pm, terry(a)jon.es wrote: >> Thanks for the detailed reply. glyph> No problem, sorry it took so long to get back to this; I definitely glyph> left the conversation halfway through. And this followup is even slower, sorry. Here's a summary of the conversation I'd like to continue: >> So in my case I want to indicate to twistd that the service that my >> class creates a makeService method to create, but which I do not set in >> motion, has failed to start and that twistd should exit, or do something >> other than cheerfully tell me that there's been an Unhandled Error. >> Does that make more sense? Sorry, I should have said I was using >> twistd. glyph> Yes. And I think that this is a good use-case for making IService a glyph> bit deeper than it is. glyph> My previous messages were basically saying "you can't do what you glyph> want with IService". i.e. you can't write your own code to do glyph> something clever and somehow have dependencies and notifications to glyph> users of "twistd" fall out of that. But that shouldn't be taken as glyph> an indication that twistd itself shouldn't be improved. glyph> It's not that you've failed to understand something - you have glyph> correctly identified that there is nothing to understand :). You glyph> need to go find that thing ("whatever it is" ;-)) in twistd that's glyph> invoking the very first call to IService.startService (and glyph> privilegedStartService as well) and propose a concrete way to make glyph> it smarter. OK, here's a concrete proposal. Let's forget about twistd for the time being and just address the central things I'm trying to do: 1. I want a service to be able to initialize itself using code that receives deferreds. Right now, startService is not expected to return anything. I can go ahead and call a deferred-returning function in startService but I wont know that it's done unless I set a flag somewhere in a callback, and the rest of the methods in my service have to check that flag to see if the initialization is complete. In case it's not clear, I'm imagining the case of a MultiService, one Service of which performs the start and stop for the overall service. 2. Because the other Services in the MultiService depend on the successful execution of the initialization, I'd rather they didn't get started at all if the initialization fails. 3. It would be nice to get my hands on a deferred that had errbacked as a result of an unsuccessful start or stop. You say: "making IService a bit deeper than it is" and then "you can't do what you want with IService". And Esteve Fernandez has already pointed me to this: > "duuuuuude. don't. inherit. multiservice." But it turns out you can do exactly what I want with a very simple subclass (or rewriting of) MultiService. BTW, this code is not tested - I want to float the idea before doing more. Here's the simple version: class GuardedMultiService(service.MultiService): implements(service.IServiceCollection) def __init__(self, initService): service.MultiService.__init__(self) self.initService = initService def startService(self): d = defer.maybeDeferred(self.initService.startService) d.addCallback(lambda _: service.MultiService.startService) def stopService(self): d = service.MultiService.stopService() d.addCallback(lambda _: self.initService.stopService) return d This, I believe, solves 1 and 2 above. It has the nice property that all the regular services add to this class will not be started until after the possibly deferred-returning initService.startService is done. So there's no need for any checking later to see that you really are initialized. And to solve 3, you can add (all defaulting to None) args to __init__ to specify two errback functions and their arguments, and use addErrback to attach these to the deferreds in startService and stopService. If we also make initService have a default of None, the result will be backward compatible with the existing MultiService, and could just replace it. The new version simply gives a way to deal with 1, 2 and 3, if you need it. BTW, in your original reply you wrote glyph> This might seem a bit inconsistent, since stopService uses the glyph> return of a Deferred. However, this is for a very specific reason, glyph> not a generalized error-handling case: you may need to prevent the glyph> *rest* of the system (specifically, the reactor) from completely glyph> shutting down My #2 above is like the flip side of this: if the initialization of a service fails, you may need to prevent the *rest* of the system from coming up, and (if you can't fix things in the errback) you might want to stop the reactor. I say this specifically with regard to services launched by twistd. Terry

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 21 Feb '09

21 Feb '09

1 0

[Twisted-Python] Twisted Sprint in February
by Jean-Paul Calderone 21 Feb '09

21 Feb '09

Hello All, I'd like to organize another Twisted sprint in Cambridge, MA in February. As with previous sprints, this is an all day event, most likely hosted at the Divmod office in Porter Square, for Twisted hacking. The two topics which have been raised thus far for this sprint are documentation - in particular, looking at Twisted documentation from a high-level and coming up with a coherent overall strategy for improving it - and deprecation (looking for old, bad APIs in Twisted which we should get rid of). Of course, other topics are welcome as well. Which weekend in February would work best for people interested in attending? Jean-Paul

6 13

Twisted February 2009