Mailman 3 March 2009 - Twisted

[Twisted-Python] Authentication on NNTP in Twisted-News
by Benjamin Welton March 31, 2009

March 31, 2009

Hey Everyone, I am wondering if NNTP authentication was ever implemented into twisted-news? I ran across a mailing list discussion from about two years ago talking about implementing it in twisted but I haven't been able to find any info on if it was actually implemented. Thanks, Ben

1 0

[Twisted-Python] Twisted conch - ssh question
by Ameya Lokare March 31, 2009

March 31, 2009

Hello everyone, I'm trying to get the sshsimpleserver.py and sshsimpleclient.py from the examples <http://twistedmatrix.com/projects/conch/documentation/examples/>to talk to each other. Both work fine with the standard linux ssh client/server respectively, but I couldn't get them to communicate with each other. All I want to do is simple data transfers to and fro, over the SSH channel (no commands to be executed). I could get the session established, but couldn't figure out how to do send data across from the client to the server. I tried channel.write, but that didn't appear to work. Sorry if this is too trivial a question, I'm a newbie to both Python and twisted. Any help would be greatly appreciated! -- Thanks & Regards, Ameya

2 3

[Twisted-Python] What kind of throughput can we expect to achieve when using DatagramProtocol (UDP)
by John Draper March 31, 2009

March 31, 2009

Hi, We have an application we intend to release where we intend to have about a million applications sending UDP packets to a Twisted Python server which needs to process incoming data (Approx 128 bytes of text) per packet, which a BACK end system is inserting this data into mySQL tables. Has anyone done some serious "stress testing" of Twisted Python simple server code to see just how much data it can digest at a time. Our server will no doubt be hosted on an OC3 - capable of 150 megabits of throughput with approx a 75 - 80% load. We need to know how many servers we need to put into some kind of a load sharing cluster to be able to handle this very high data before it chokes. Right now, in our proof of performance, we are using UDP, but planning to move to the more reliable TCPIP protocol when we get into production. These data requests will come in fast and violent spurts of data. John

3 2

[Twisted-Python] How to use defer.fail better...
by Jan Bakuwel March 30, 2009

March 30, 2009

Hoi all, I'm writing a twisted/python SMTP server that accepts emails from a MTA using twisted. All works well, except that when I return a defer.fail(None) from processEmail to messageHandler.eomReceived (see below), twisted dumps the trace back of the last exception on stdout. Is there a way to avoid this? I would like to handle the exception gracefully (with an exception handler) and would not like to see any trace back from it in my logs. I've read that using None as the parameter to defer.fail does exactly that... what should I use instead? The other thing I would like to do better is to have a way to return smtp code 421 (service temporarily unavailable) rather than 550 (fatal) in those cases where I would like to indicate a (temporary) failure to receive the email. I've patched smtp.py (replace 550 -> 421) for this but would gladly hear about a way to do this without patching a standard component on my system. thanks, Jan class messageHandler(object): implements(smtp.IMessage) def __init__(self, addressee): self.lines = [] self.noisy = False self.emailMessage = None self.emailAddress = str(addressee) #__init__ def lineReceived(self, line): self.lines.append(line) #lineReceived def eomReceived(self): # message is complete, store it self.lines.append('') # add a trailing newline messageData = '\n'.join(self.lines) emailMessage = message_from_string(messageData) return processEmail(self.emailAddress, emailMessage) #eomReceived def connectionLost(self): log(1, 'Connection lost unexpectedly!') # unexpected loss of connection; don't save del(self.lines) del(self.emailMessage) #connectionLost #messageHandler class localDelivery(object): implements(smtp.IMessageDelivery) def __init__(self): pass #end __init__ def validateFrom(self, helo, originAddress): log (1, 'Incoming email from %s', str(originAddress)) # accept mail from anywhere. To reject an address, raise # smtp.SMTPBadSender here. return originAddress #end validateFrom def receivedHeader(self, helo, origin, recipients): myHostname, clientIP = helo headerValue = "by %s from %s with ESMTP ; %s" % (myHostname, clientIP, smtp.rfc822date()) log(1, '...received: %s', headerValue) # email.Header.Header used for automatic wrapping of long lines return "Received: %s" % Header(headerValue) #end receivedHeader def validateTo(self, user): #if not user.dest.domain in valid_domains: # print "Not accepting mail for %s" % user.dest # raise smtp.SMTPBadRcpt(user) log(1, 'Accepting email for %s', user.dest) return lambda: messageHandler(user.dest) #end validateTo #localDelivery class SMTPFactory(protocol.ServerFactory): def __init__(self): pass #end __init__ def buildProtocol(self, addr): delivery = localDelivery() smtpProtocol = smtp.SMTP(delivery) smtpProtocol.factory = self return smtpProtocol #buildProtocol #SMTPFactory

2 2

Re: [Twisted-Python] RE: how to create state machines?
by Andrew Francis March 30, 2009

March 30, 2009

Message: 2 Date: Mon, 30 Mar 2009 11:10:48 -0400 From: "Doug Farrell" <dfarrell(a)mypublisher.com> Subject: [Twisted-Python] RE: how to create state machines? To: <twisted-python(a)twistedmatrix.com> Message-ID: <318F79422ADC5041A93343721F84474D01A8BD68(a)exchange.mypublisher.local> Content-Type: text/plain; charset="us-ascii" >[Doug Farrell] Thank you very much for your detailed response, You are welcome Doug. >One thing I'm still a little confused by in your reply is how you're >getting the deferred instance. Is there a particular reason your example >does this: >deferred = someOtherFunctionThatReturnsADeferred() Yes. There are Twisted methods that return a Deferred. For instance, client.getPage() or in the case of a XMLRPC client, proxy.callRemote(). def processPage(data): # do stuff with the web page here deferred = client.getPage("http://wwww.google.com") deferred.addCallback(processPage) >rather than this: >deferred = defer.deferred() Sometimes you want to create your own deferred and trigger it with a callback(). Cheers, Andrew

1 0

[Twisted-Python] RE: how to create state machines?
by Doug Farrell March 30, 2009

March 30, 2009

Hi Andrew, > > Hi Doug: > > >This state machine initializes a system, issues an asynchronous > command > >that takes time to complete (say a motor move command) and then waits > >for that command to be done before exiting. In the context of a > >framework that is calling this state machine, the WAITCMD1 is executed > >repeatedly (polling) while the asynchronous command completes. A > system > >can be constructed with lots of little state machines like this and be > >made to look like it is doing many things at once even though the > >execution of the program is single threaded. > > >I understand (pretty much) the Twisted framework is like this and > >implmenting event handlers like connectionMade(), etc., is a state > >machine, but I'm wondering how to implement what I've outlined above > in > >one of Twisted's state event handlers, or inside something like > >callLater(). For example, let's say I want to use Twisted to run a > long > >running daemon process that has an XMLRPC interface. That XMLRPC > >interface is an interface to a state machine inside Twisted that > allows > >the caller to change state, or start new state machines, or get the > >status of a running state machine. In that case I'm thinking I want a > >state machine the runs continuously in the Twisted loop, structured > like > >the example above; co-operatively yielding back to Twisted, but > running > >non-stop. Something like callLater(1, stateMachine), but non-stop, > >without even the 1 second call loop. > > If I understand your correctly, I don't think you need to implement a > state machine to simulate concurrency with Twisted - Twisted does a lot > of that for you. You can think of a Twisted application as a state > machine - the callback being the state and the completion of the > operation and the calling of the callback is the transition. These > callbacks at runtime act like a thread of execution. > > def Initialize(...): > # do something > deferred = someFunctionThatReturnsADeferred() > deferred.addCallback(State2) > > def State2(...): > # do something > deferred = someFunctionThatReturnsADeferred() > deferred.addCallback(State3) > > def State3(someData): > # do something > if someData == 'State4': > deferred = someFunctionThatReturnsADeferred() > func = State4 > elif someData == 'State5': > deferred = someOtherFunctionThatReturnsADeferred() > func = State5 > ... > deferred.addCallback(func) > > if __name__ = "__main__": > initialize() > reactor.run() > > As for the Twisted loop. Well, you don't really see the Twisted loop > since that is hidden in the reactor. Also you should distinguish > between writing a new protocol and using an existing one. > > In the case of XMLRPC, creating the server isn't the problem. > > http://twistedmatrix.com/projects/web/documentation/howto/xmlrpc.html > > Once a XMLRPC server is created, Twisted will take responsibility for > creating new instances (or threads if you want to see it that way). If > you still need a state machine, then the only hiccup I can see is > sharing state machine (if you really need one) between XMLRPC method > invocations. > > Cheers, > Andrew > [Doug Farrell] Thank you very much for your detailed response, I'm kinda getting it and am going to try out a prototype to make sure I do. As you mention, Twisted does take care of a lot of the issues dealing with concurrency. Having written a few threaded applications, I didn't really want to get into starting my state machine in a thread and then having to deal with all the cross thread data safety issues. One thing I'm still a little confused by in your reply is how you're getting the deferred instance. Is there a particular reason your example does this: deferred = someOtherFunctionThatReturnsADeferred() rather than this: deferred = defer.deferred() I just want to understand if there a reason the deferred is being created in the other function. Again, thanks for your help and response! Doug

1 0

[Twisted-Python] Mac-owning Twisted programmer(s) wanted for interactive museum project
by Darran Edmundson March 29, 2009

March 29, 2009

Hi All, We are working on a large Australian museum project due to open in May and expecting more than 300,000 visitors per year. We urgently need a few more highly-skilled python programmers to help get us across the finish line. On the digital side, this 10,000 square foot "Museum of Democracy" features - an interactive timeline made from 14 52" touch screens. - a 270-degree cylindrical theatre of choreographed voxpops - touch interactives that solicit visitor opinions on museum content - RFID-enabled passes that, among other things, provide content for non-English speakers and accessibility to vision impaired visitors. - roughly 45 short films spread over 20 different viewing areas - on the technology side, we are using Twisted and CoreAnimation through the pyObjC bridge. If this sounds interesting and you: - are a savvy python programmer with either extensive Twisted or pyObjC experience, or - are an ObjectiveC guru with CoreAnimation experience and proficient in python and - are experienced in object-oriented software development - can work remotely as part of our globally-distributed team (our team has members in Czech Republic, London, Guernsey, Australia) - have access to a Mac for local development - are committed to unit testing (we use trial and buildbot) - are available for full-time employment from now through the end of May (and perhaps longer). - are comfortable with SVN and Trac then please contact me directly at darran(a)edmstudio.com. As part of your response, please provide - a resume, preferably focused on python and/or osx development - a code sample demonstrating your Twisted, pyObjC or Objc prowess - example of a unit test you've written - your availability - your desired hourly rate Note, we are after Twisted python and/or CoreAnimation gurus, not general python programmers with an interest in Mac. Cheers, Darran P.S. Apologies if the above "job posting" isn't welcome on the list; I looked but couldn't find any restrictions in this regard.

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 29, 2009

March 29, 2009

1 0

FW: [Twisted-Python] strange server crash
by Alec Matusis March 28, 2009

March 28, 2009

One more thing: This server has ECC memory, and it also has a BMC controller that externally logs all hardware errors independently of the memory condition. >From what I understand, ECC memory module has an extra memory chip for data hashes. When data is written into ECC memory, the hash is created and stored in the extra chip, and when it's retrieved, the hash is checked. When a mismatch occurs, even if kernel does not log such error, the BMC controller logs it as # ipmitool sel list 1c10 | 02/19/2009 | 16:29:39 | Memory #0x08 | Uncorrectable ECC | .. I retrieved the system even log (SEL) list from the BMC controller, and are no errors whatsoever. This looks like a Python error or something very basic. It started happening after I slightly changed the code for this particular server. -----Original Message----- From: Alec Matusis [mailto:matusis@yahoo.com] Sent: Tuesday, March 24, 2009 6:26 PM To: 'Twisted general discussion' Subject: RE: [Twisted-Python] strange server crash This server crashed again today, again during maximum load for the day. This time, no errors in the twisted log, and not even a segfault message in /var/log/messages : the pid simply ceased to exist. Once again, this machine runs 8 twisted servers, but this one is slightly different from the others, and the error happened after the code for this server has been slightly modified. I do not think this is the bad RAM anymore, because there's one particular server that keeps crashing on this machine. > -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python- > bounces(a)twistedmatrix.com] On Behalf Of glyph(a)divmod.com > Sent: Monday, March 23, 2009 1:37 AM > To: 'Twisted general discussion' > Subject: RE: [Twisted-Python] strange server crash > > > On 07:25 am, matusis(a)yahoo.com wrote: > >Very strange. I am not using any custom C extensions... > >In the last two days, it has been under larger load, and it has not > >crashed. > >I will update to Python 2.6 soon. > > Have you tested for bad RAM on that server? The error mode is > sufficiently weird and rare to make me suspect cosmic rays. > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

[Twisted-Python] db connections
by Pet March 28, 2009

March 28, 2009

Hi, I've wrote an daemon which does some queries to db and sends response back to client. Do I need to make for every request from client (new instance of MyProtocol) a new connection to DB? Or can I somehow prepare connection, so I could save some time? Maybe make connection in Factory and pass it to Protocol? But what happens if too much clients are connected to server? What is the way to control it? Regards, Pet class MyProtocol(basic.LineReceiver): def __init__(self): print "new connection" self.dbcon = adbapi.ConnectionPool("pyPgSQL.PgSQL", database="data", user='pet', host='local', password='some') class MyFactory(protocol.ServerFactory): protocol = MyProtocol class MyService(internet.TCPServer): def __init__(self): internet.TCPServer.__init__(self,PORT,MyFactory()) def main(): reactor.listenTCP(PORT, MyFactory()) reactor.run() if __name__ == '__main__': main()

4 10