Mailman 3 March 2006 - Twisted

[Twisted-Python] Degrading under load
by Yitzchak Gale 08 Mar '06

08 Mar '06

I need to set up a TCP service (on a linux box) that will get something like a few hunderd connections per minute at peak load. For each connection, I do some XML processing, and possibly send a query to another nearby machine and get a respone. Seems to me that twisted should be able to handle that. But what happens when I get the occasional burst of connections, lets say tens of connections within one second? What I need is: o Every client gets a socket connection promptly, so no danger of TCP timeout. o Under medium load, clients will have to wait a bit longer for the response. o Under heavy load, some clients will get a "busy" response (defined in the protocol I am implementing) and immediate socket close. What is the best way to do that in twisted? I envision one of the following architectures: A. Just use twisted in the usual way. Watch twisted's event queue for heavy load. B. Two processes: one to dish out connections and one to queue requests and process them. C. Three processes: one to dish out connections, one to queue requests and watch for load, and one to process the requests. Which of these do I need to use to get the desired effect under load? Or is there some better way? Thanks, Yitz

2 1

[Twisted-Python] ATTN z3p: Errors in Conch tests
by Jonathan Lange 07 Mar '06

07 Mar '06

z3p, I've switch the conch tests to use Deferreds rather than spinUntil and spinWhile (See branches/deprecated-trial-1509 and http://www.twistedmatrix.com/trac/ticket/1509). However, two tests in test_conch are failing when run on Linux (they pass on OS X). The code looks fine and other reviewers can't seem to spot the problem. Why is it breaking? I'd appreciate a response as soon as possible, even a negative one. cheers, jml

2 1

[Twisted-Python] ClientCreator error handling
by Mike Christey 07 Mar '06

07 Mar '06

Hi, I don't know if this is right place to ask this kind of question but I'm really stuck at this problem and documentation seems not help (or better, I'm dumb). For me who uses try/except on daily basis the twisted error handling seems to be voodoo. All I want to do is add 'connection refused', 'dns lookup er ror' and similiar handling to the sshsimpleclient.py script. I tried something like: def errorCallback(failure): print "Err" [...] # other code d = defer.Deferred() d.addErrback(errorCallback) protocol.ClientCreator(reactor, SimpleTransport).connectTCP(HOST, 22) reactor.run() but that's probably have nothing to do with way it should be done.. Anyone could help me please? Concerned about your privacy? Instantly send FREE secure email, no account required http://www.hushmail.com/send?l=480 Get the best prices on SSL certificates from Hushmail https://www.hushssl.com?l=485

2 1

[Twisted-Python] Missing feature in detecting stalled PB connections?
by Luc Stepniewski 07 Mar '06

07 Mar '06

While using pb.PBClientFactory() with reactor.connectTCP(), I noticed that the specified timeout parameter is only for testing if there is no connection, not if the remote PB replies correctly. I noticed this when one of my remote PB instance was stuck. It was accepting requests, but did not answer anything at all. So the connection is established, and stays indefinitely! You can verify this by yourself by just putting netcat ("nc -vv -l -p 9999" for example) instead of a PB instance. The PB on the other side will connect, and then, nothing else. No timeout that netcat didn't reply (not even correctly). I know I could add a timer (callLater) that would wake up and kill the connection after some time (because I know one transaction will not exceed some precise time), but it's really really trashy, and one not always knows what will be the duration of the transaction. Is there really no other alternative to detect the correct connection to another PB instance? I attached an example code that would show this behaviour. Thanks, Luc -- Luc Stepniewski <luc.stepniewski(a)adelux.fr> <sip:724766@fwd.pulver.com> Adelux - Securite, Linux Public key: <http://lstep.free.fr/pubkey.txt> Key BC0E3C2A fingerprint = A4FA466C68D27E46B427 07D083ED6340BC0E3C2A

3 5

[Twisted-Python] API reference
by Diwaker Gupta 05 Mar '06

05 Mar '06

Hi everyone, The new website seems to be coming along nicely. However, I find myself missing the old API reference a lot. The previous Javadoc style reference was very easy to navigate, jump across things, and provided a lot more useful information in the same screen space compared to the current format (where everything is on its own HTML page, no frames) It seems the pages are still being generated by epydoc -- some styling is missing, and the frames page is missing? Is this something being worked on/considered? Or is the current format of API reference final? Thanks, Diwaker -- Web/Blog/Gallery: http://floatingsun.net/blog

4 4

[Twisted-Python] SQLAlchemy with Twisted
by Simon Hedberg 05 Mar '06

05 Mar '06

Hi, There were some talk on the list about this a while ago and Ed Suominen had some code up for review. What's the status on this now? I haven't tested SQLAlchemy yet but it seems quite nice on paper, and it would feel good to replace my own orm hack with something that has more development efforts behind it. And I guess I'm not the only one in this situation.. TIA /Simon

1 0

Re: [Twisted-Python] beginner's question to OSCAR & twisted
by Andreas Kostyrka 05 Mar '06

05 Mar '06

Ok, twisted/docs/words/examples contains oscardemo which works well enough. If you are interested I've written on the base of it an AIM <-> IRC (xchat) bridge. Andreas Am Sonntag, 5. März 2006 18:42 schrieb Tobi Lewitt: > Hello, my name is Tobi and i'm a student from germany. > I have some experience with python, but i think i'm still bad at all. > I have heard about twisted, and i am thinking for months to build an > own ICQ/OSCAR-bot, so i downloaded Twisted-Words and looked at this > OSCAR-file. but i dont know really what to do with it. > > I think it would already help me if you give me perhaps a hint how to > use twisted to connect with a ICQ-server, if i once know i think i > would be alble to understand the rest on my own. > > i hope my questions aren't too stubid for that list, tobi. > ______________________________________________________________ > Verschicken Sie romantische, coole und witzige Bilder per SMS! > Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193 > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Andreas Kostyrka -- Andreas Kostyrka

1 0

[Twisted-Python] beginner's question to OSCAR & twisted
by Tobi Lewitt 05 Mar '06

05 Mar '06

Hello, my name is Tobi and i'm a student from germany. I have some experience with python, but i think i'm still bad at all. I have heard about twisted, and i am thinking for months to build an own ICQ/OSCAR-bot, so i downloaded Twisted-Words and looked at this OSCAR-file. but i dont know really what to do with it. I think it would already help me if you give me perhaps a hint how to use twisted to connect with a ICQ-server, if i once know i think i would be alble to understand the rest on my own. i hope my questions aren't too stubid for that list, tobi. ______________________________________________________________ Verschicken Sie romantische, coole und witzige Bilder per SMS! Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193

2 1

[Twisted-Python] sample tac that needs a critique
by Don Smith 04 Mar '06

04 Mar '06

Okay, I've made some huge progress, I think with my Telnet connection stuff. I'm curious what the Twisted pros think of this..syntax, if it is correct, etc. Here is what I'm going for, I want to have this application running, which provides an XML-RPC interface for 'controlling' it. The server, right now, can start Telnet clients at the request of XML-RPC clients. This seems to work, however, I'd also like to be able to stop said Telnet clients, or see which ones are running, etc. I can't wrap my brain around how to accomplish those bits of functionality. I've included the .tac file below for reference. Thanks so much for your assistance! -Don #Try to build a Twisted Connector Service the twisted way from twisted.application import internet, service from twisted.web import resource, server, static, xmlrpc import re, time from EventTypes import Event connectiontypes = {'TelnetConnection':'A telnet client connection.', 'FileTailConnection':'Facilitates tailing a log file'} class Connection: def __init__(self,name): # Constructor self.name=name #print "Connection name: %s" % self.name self.status=False def setStatus (self,status): self.status=status #################### # TCP Client Stuff #################### from twisted.internet.protocol import Protocol, ReconnectingClientFactory from twisted.conch.telnet import Telnet ########################## # Telnet Clients ########################## class TelnetClient(Telnet): "TelnetClient class." def connectionMade(self): #print "connection made to %s" % self.factory.name #if the cmd variable was set in the calling factory, lets iterate over it. if self.factory.cmd: for cmd in self.factory.cmd: if "sleep" in cmd: #print re.compile ('(?P<sleeptime>(\d+))').search(cmd).group('sleeptime') sleeptime = int(re.compile ('(?P<sleeptime>(\d+))').search(cmd).group('sleeptime')) time.sleep(sleeptime) else: self.write(cmd) def write(self, data): #print data Telnet._write(self, data+"\r\n") def dataReceived(self, data): print "received:", data #always send data to the eventQueue self.factory.eventQueue.insert(Event(data, self.factory.name)) #check if we should send any commands based on data received if self.factory.expect: #check if any of the expect dictionary keys are in data for key in self.factory.expect.keys(): if key in data: for cmd in self.factory.expect[key]: if "sleep" in cmd: sleeptime = int(re.compile ('(?P<sleeptime>(\d+))').search(cmd).group('sleeptime')) time.sleep(sleeptime) else: self.write(cmd) class TelnetConnection(Connection, ReconnectingClientFactory): "Telnets to host:port and executes cmd. cmd" protocol = TelnetClient def __init__(self, name, eventQueue, cmd=None, expect=None): Connection.__init__(self, name) self.cmd = cmd self.expect = expect self.eventQueue = eventQueue def clientConnectionFailed(self, connector, reason): print 'connection failed:', reason.getErrorMessage() def clientConnectionLost(self, connector, reason): print 'connection lost:', reason.getErrorMessage() class ManagerService(service.Service): def __init__(self): pass # some XML-RPC function calls for twisted server def stop(self): return 'Shutdown initiated' def getConnectionTypes(self): """Return connection types""" return connectiontypes def xmlrpc_getActiveConnections(self): """Return all active connections""" def getConnectionStatus(self, connection): return "connection status" def getResource(self): """Return sum of arguments.""" r = resource.Resource() x = xmlrpc.XMLRPC() x.xmlrpc_stop = self.stop x.xmlrpc_getConnectionStatus = self.getConnectionStatus x.xmlrpc_startTelnet = self.startTelnet r.putChild('RPC2', x) return r def startTelnet(self, host, port, name, queue, command, script): from twisted.internet import reactor from EventQueue import Queue #Create the Event Queue object self.queue = queue self.queue = Queue() reactor.connectTCP(host, int(port), TelnetConnection(name,self.queue ,command,script)) return "%s started" % name #create an application instance application = service.Application('connectorservice', uid=1, gid=1) m = ManagerService() serviceCollection = service.IServiceCollection(application) internet.TCPServer(7090, server.Site(m.getResource()) ).setServiceParent(serviceCollection)

2 2

[Twisted-Python] Re: [Twisted-commits] r16149 - first complete draft of documentation, one example module, and make some changes to tap.py so that things actually work as documented.
by Stephen Waterbury 03 Mar '06

03 Mar '06

David Reid wrote: > + ... Web2 makes use of TAP plugins, created by the mktap > + commandline utility. This document outlines a few of the > + approaches for creating, configuring, and deploying a Twisted.Web2 > + with mktap.</p> Eh? I must have missed something (not unusual) -- I thought taps were going away ... are they coming back now? Steve

2 1