Mailman 3 February 2009 - Twisted

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 08 Feb '09

08 Feb '09

1 0

[Twisted-Python] Perspective Broker and thread safe!
by B. B. 07 Feb '09

07 Feb '09

Hello, I am relative new to twisted, though I have manage to make a simple client server solution ( using perspective broker ) that I am now try to refactor a little. The solution is now something like : Using perspective broker, clients connects to the server ( setting up a remote reference to each other ). Via a PB call, each client can request the server do some processing. The result of the processing is distributed to each connected client - and the client asking for the request gets a simple status report! This solution works quite well, the only bottleneck is that all the "thread-safe" processing on the server is done in the mainloop of the server reactor. I want to do concurrent processing on the server! The function doing the processing on the server, also distribute the result to all the connected clients, and returns the result to the klient asking for the request. My question is : If I just "defer" that function "toThread" ( like threads.deferToThread( myServerProcessingFunction ) ). Do I have potential problems, when several threads try to distribute its result to the connecting clients??? If I have a problem: Can I make a quick fix as follows: Within each thread ( started with the threads.deferToThread function ), I do the distributing to the clients by using the "reactor.callFromThread" ( like reactor.callFromThread( distributeToAllClient , myResult ) where the function distributeToAllClient are using the remoteReference for each client to send the "myResult" ) Or is the only solution, to let the myServerProcessingFunction returns the "myResult" to be distributed to all connected clients, and do the distribution in the reactor mainthread??? Thank you! Brian

3 4

[Twisted-Python] Problems with signalProcess on Windows
by Brian Granger 06 Feb '09

06 Feb '09

Hi, [take a deep breath and try to get over how much I dislike Windows] So, I am using Twisted's external process capabilities in a cross-platform setting. Almost everything is working now in Windows. But, when I try to stop the external program (using signalProcess("INT")) that I starting using reactor.spawnProcess, I get the following traceback: self.process_transport.signalProcess(sig) File "C:\Python25\lib\site-packages\twisted\internet\_dumbwin32proc.py ", line 247, in signalProcess win32process.TerminateProcess(self.hProcess, 1) pywintypes.error: (5, 'TerminateProcess', 'Access is denied.') The odd thing is that the process stops just fine. That is, it tells me that it has received SIGINT and is shutting down. Any thoughts on why this would happen? I have no idea where to even begin looking. Thanks in advanced. Cheers, Brian

2 2

[Twisted-Python] Maximum Recursion Depth Error with Twisted 8.2/Stackless 2.6.1
by Andrew Francis 06 Feb '09

06 Feb '09

Hi Colleagues: I recently updated to Stackless 2.6.1 and Twisted 8.2. I executed the following programme that works fine under Stackless 2.5.2 and Twisted (including 8.2) and I received the following error: traceback (most recent call last): File "ToyProcessor5.py", line 60, in twistedReactor reactor.run() File "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" , line 1048, in run self.mainLoop() --- <exception caught here> --- File "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" , line 1057, in mainLoop self.runUntilCurrent() File "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" , line 707, in runUntilCurrent log.deferr() File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", l ine 153, in err _stuff = failure.Failure() File "/usr/local/lib/python2.6/site-packages/twisted/python/failure.py ", line 265, in __init__ parentCs = reflect.allYourBase(self.type) ", line 542, in allYourBase accumulateBases(classObj, l, baseClass) File "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded Similarily written programmes fail in the same fashion. What is suspicious is that there is no recursion in the offending section. Even if I create one worker tasklet, I get the same error. def twistedReactor(): l = task.LoopingCall(stackless.schedule) l.start(.01) reactor.run() however, if I change l = task.LoopingCall(stackless.schedule) to l = task.LoopingCall(tick) and tick is def tick(): stackless.schedule() the programme works. Although a work around, I would like to find the real problem. What particular worries me is when I created small test examples, I was not able to recreate the problem. Something else is going on.... What I would appreciate is some hints as to what may be happening. - I normally don't expect "Maximum Recursion Depth" errors in Stackless. - What is the reflect.allYourBases stuff? - What is log.deferr? (I don't recall seeing that method). If I get few clues, it would make it easier for me to write new tests and zero in on the problem. Hopefully the problem is with my code rather than Stackless 2.6.1 and/or Twisted 8.2 I have included some sample code. Unfortunately ToyProcessor5.py is a bit large (I have newer code that is smaller but requires more files) Cheers, Andrew

3 6

[Twisted-Python] most efficient new connection rate limiting?
by Alec Matusis 05 Feb '09

05 Feb '09

We need to limit the new connection rate from a given IP, such that for example 10.10.1.10 cannot connect more often than 10 times per minute. This is a high-volume TCP Twisted server, with about 500 new distinct IP connections per second in the normal state (and we run 8 of these on each 8 core server, so ~4000 new conns/sec per box). I am trying to find the least CPU intensive approach for this. 1) Create an dictionary {ip1:count1, ip2: count2, .} in the server, and check the counts for each incoming connection. Disconnect with transport.loseConnection() if the threshold for ip:count is exceeded. Reset this dictionary to empty dict {} every minute with reactor.callLater timer. 2) Use some Twisted rate limiter API that I am not familiar with? 3) Use iptables rate-limiting module like so: iptables -I INPUT -p tcp --dport 8888 -i eth0 -m state --state NEW -m recent --set iptables -I INPUT -p tcp --dport 8888 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP Which one of these approaches would you recommend?

2 1

[Twisted-Python] Twisted Conch SSH timeout
by ray terrill 05 Feb '09

05 Feb '09

I've got a Twisted Conch/SSH script that I inherited which I'm using to run commands remotely on some servers. I'd like to add a timeout for servers that are down, etc (the script currently hangs on these), but I'm not sure where to add that functionality. Any help would be appreciated. Thanks, -Ray =========Script Follows========= #!/usr/bin/python from twisted.conch import error from twisted.conch.ssh import transport, connection, keys, userauth, channel, common from twisted.internet import defer, protocol, reactor #a class to store the data returned from the ssh client code class SSHData(): data = '' class ClientCommandTransport(transport.SSHClientTransport): def __init__(self, username, password, command): self.username = username self.password = password self.command = command def verifyHostKey(self, pubKey, fingerprint): # in a real app, you should verify that the fingerprint matches # the one you expected to get from this server return defer.succeed(True) def connectionSecure(self): self.requestService( PasswordAuth(self.username, self.password, ClientConnection(self.command))) class PasswordAuth(userauth.SSHUserAuthClient): def __init__(self, user, password, connection): userauth.SSHUserAuthClient.__init__(self, user, connection) self.password = password def getPassword(self, prompt=None): return defer.succeed(self.password) class ClientConnection(connection.SSHConnection): def __init__(self, cmd, *args, **kwargs): connection.SSHConnection.__init__(self) self.command = cmd def serviceStarted(self): self.openChannel(CommandChannel(self.command, conn=self)) class CommandChannel(channel.SSHChannel): name = 'session' def __init__(self, command, *args, **kwargs): channel.SSHChannel.__init__(self, *args, **kwargs) self.command = command def channelOpen(self, data): self.conn.sendRequest( self, 'exec', common.NS(self.command), wantReply=True).addCallback( self._gotResponse) def _gotResponse(self, _): self.conn.sendEOF(self) def dataReceived(self, data): SSHData.data = data def closed(self): reactor.stop() class ClientCommandFactory(protocol.ClientFactory): def __init__(self, username, password, command): self.username = username self.password = password self.command = command self.deferred = defer.Deferred() def buildProtocol(self, addr): protocol = ClientCommandTransport( self.username, self.password, self.command) return protocol def testConnect(username, password, command, server): testFactory = ClientCommandFactory(username, password, command) reactor.connectTCP(server, 22, testFactory) return testFactory.deferred def handleSuccess(result, port): print "Connected to port %i" % port reactor.stop() def handleFailure(failure, port): print "Error connecting to port %i: %s" % ( port, failure.getErrorMessage()) reactor.stop() if __name__ == "__main__": import sys, getpass server = sys.argv[1] command = sys.argv[2] username = raw_input("Username: ") password = getpass.getpass("Password: ") connecting = testConnect(username, password, command, server) #connecting.addCallback(handleSuccess, server) #connecting.addErrback(handleFailure, server) connecting.setTimeout(1, handleError) reactor.run() print "The data I got back from the ssh client is " + SSHData.data

2 3

[Twisted-Python] installation help
by Arie Lakeman 04 Feb '09

04 Feb '09

Hi, I've taken an interest in event driven programming, I came across twisted and liked the look of it, I've tried downloading the .dmg from twistedmatrix/trac unfortunately after trying multiple installations including direct installation to both the folder where python is found (under system/frameworks...) and by changing the sys.path to another directory I fail to be able to import twisted in the interactive prompt. I have a hunch I may be missing OpenSSL (occassional errors) but I don't know enough about this (after googling) to be able to resolve this. Essentially I'm interested in writing local event-driven code, whereby I write specific objects that are capable of interaction in a more natural manner than Python would permit by threading and classes, so I'm perfectly happy with any solution that allows me to only locally program event-driven code. I would really appreciate any help whatsoever on how to resolve this installation problem! thanks, Arie

2 2

[Twisted-Python] redefine signal handlers
by Gabriel Rossetti 04 Feb '09

04 Feb '09

Hello everyone, I would like to run some cleanup code when my Twisted app receives a signal (SIGINT/SIGBREAK/SIGTERM). I saw that "_SignalReactorMixin" sets the handlers and that "ReactorBase" defines the default handlers : def sigInt(self, *args): """Handle a SIGINT interrupt. """ log.msg("Received SIGINT, shutting down.") self.callFromThread(self.stop) def sigBreak(self, *args): """Handle a SIGBREAK interrupt. """ log.msg("Received SIGBREAK, shutting down.") self.callFromThread(self.stop) def sigTerm(self, *args): """Handle a SIGTERM interrupt. """ log.msg("Received SIGTERM, shutting down.") self.callFromThread(self.stop) My question is how can I redefine them, other than monkey patching or inheriting the reactor and over-riding them (which I'd rather not do since some of my code uses the windows reactor when on windows since I was having problems with windows event)? Is there such a mechanism, something like setDefaultSig("SIGINT, mySigIntHandler)? Thank you, Gabriel

3 3

[Twisted-Python] RE: [Stackless] Maximum Recursion Depth Error with Twisted 8.2/Stackless 2.6.1
by Andrew Francis 04 Feb '09

04 Feb '09

Hi Kristjan: Thanks for the explanation and the effort. Tomorrow, I'll hunt down a few things on my end. If you find ToyProcessor5 too convoluted, I'll post the code for another prototype that should be easier to follow. Cheers, Andrew --- On Tue, 2/3/09, Kristján Valur Jónsson <kristjan(a)ccpgames.com> wrote: > From: Kristján Valur Jónsson <kristjan(a)ccpgames.com> > Subject: RE: [Stackless] Maximum Recursion Depth Error with Twisted 8.2/Stackless 2.6.1 > To: "Andrew Francis" <andrewfr_ice(a)yahoo.com>, "twisted-python(a)twistedmatrix.com" <twisted-python(a)twistedmatrix.com> > Cc: "stackless(a)stackless.com" <stackless(a)stackless.com> > Date: Tuesday, February 3, 2009, 1:49 AM > Stackless does stack copying if it senses that the C stack > has grown to deep. But it still tries to maintain the > recursion depth limit set in "sys". There is > probably a bug somewhere in the bookkeeping of this. > I'll try trawling through the code for the > corresponding stuff If I find the time today. > K > > -----Original Message----- > From: stackless-bounces(a)stackless.com > [mailto:stackless-bounces@stackless.com] On Behalf Of Andrew > Francis > Sent: 2. febrúar 2009 22:48 > To: twisted-python(a)twistedmatrix.com > Cc: stackless(a)stackless.com > Subject: [Stackless] Maximum Recursion Depth Error with > Twisted 8.2/Stackless 2.6.1 > > Hi Colleagues: > > I recently updated to Stackless 2.6.1 and Twisted 8.2. I > executed the following programme that works fine under > Stackless 2.5.2 and Twisted (including 8.2) and I received > the following error: > > traceback (most recent call last): > File "ToyProcessor5.py", line 60, in > twistedReactor > reactor.run() > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 1048, in run > self.mainLoop() > --- <exception caught here> --- > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 1057, in mainLoop > self.runUntilCurrent() > File > "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py" > , line 707, in runUntilCurrent > log.deferr() > File > "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", > l ine 153, in err > _stuff = failure.Failure() > File > "/usr/local/lib/python2.6/site-packages/twisted/python/failure.py > ", line 265, in __init__ > parentCs = reflect.allYourBase(self.type) > ", line 542, in allYourBase > accumulateBases(classObj, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > File > "/usr/local/lib/python2.6/site-packages/twisted/python/reflect.py", > line 550, in accumulateBases > accumulateBases(base, l, baseClass) > exceptions.RuntimeError: maximum recursion depth > exceeded > > Similarily written programmes fail in the same fashion. > What is suspicious is that there is no recursion in the > offending section. Even if I create one worker tasklet, I > get the same error. > > def twistedReactor(): > l = task.LoopingCall(stackless.schedule) > l.start(.01) > reactor.run() > > however, if I change > > l = task.LoopingCall(stackless.schedule) > > to > > l = task.LoopingCall(tick) > > and tick is > > def tick(): > stackless.schedule() > > the programme works. Although a work around, I would like > to find the real problem. > > What particular worries me is when I created small test > examples, I was not able to recreate the problem. Something > else is going on.... > > What I would appreciate is some hints as to what may be > happening. > > - I normally don't expect "Maximum Recursion > Depth" errors in Stackless. > - What is the reflect.allYourBases stuff? > - What is log.deferr? (I don't recall seeing that > method). > > If I get few clues, it would make it easier for me to write > new tests and zero in on the problem. Hopefully the problem > is with my code rather than Stackless 2.6.1 and/or Twisted > 8.2 > > I have included some sample code. Unfortunately > ToyProcessor5.py is a bit large (I have newer code that is > smaller but requires more files) > > Cheers, > Andrew

1 0

[Twisted-Python] How can I disconnect ssl connection from the client side?
by Atsuo Ishimoto 02 Feb '09

02 Feb '09

Hello, I'm writing a smtp-client application with twisted.mail. Testing the application, I noticed that timeout doesn't work in some case if client started starttls session. If smtp-server stops responding after DATA command was sent from client, calling transport.loseConnection() doesn't close connection. I found ticket # 2827 reports connector.disconnect() doesn't work for ssl connection if the server doesn't send any data. Is there a workaround to disconnect ssl connection? Regards,

3 7