Mailman 3 July 2009 - Twisted

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com July 5, 2009

July 5, 2009

1 0

[Twisted-Python] buildProtocol never called
by travis+ml-twisted＠subspacefield.org July 5, 2009

July 5, 2009

Hello, First let me introduce a piece of software I have written using Twisted: it's called dfd_keeper[1], and is a BSD/pf port of a generic concept I invented called the dynamic firewall daemon[2]. [1] http://www.subspacefield.org/security/dfd_keeper/ [2] http://www.subspacefield.org/security/dfd/ If you like the concept, please promote the idea; I think it's neat but there hasn't exactly been a groundswell of interest in it. In essence, it allows you to netcat or telnet to the daemon process and issue text-like commands that alter your firewall rules. You can think of it as a "shell" for the firewall, and it can do neat things like create firewall rules that expire automatically after a period of time and so on. The problem I'm having is this; I upgraded my firewall from OpenBSD 4.2 or so to OpenBSD 4.5, and now I'm working with newer versions of all the software including Twisted. The program works fine normally, and can run in the background, but if I invoke a daemonize() routine that turns it into a network daemon, it refuses to serve incoming TCP connections. Actually the TCP connection is made, but the software never responds to it. Specifically, my twisted.internet.protocol.Factory instance is created, but buildProtocol is never called. Before I dive deeply into debugging the software, I thought I'd ask if anyone knew off the top of their head any reason why Twisted might not act the same if it was daemonized (no controlling terminal, no stdin/out/err, forked as a background process). Thanks! Travis H. -- Obama Nation | My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email john(a)subspacefield.org to get blacklisted.

3 6

[Twisted-Python] Conch, getting the avatarId, session number, and terminal protocol instance of all connected users.
by asset July 4, 2009

July 4, 2009

Ive been at it for several days now trying to get at the username (avatarId), session id (SSHChannel session # I imagine), and terminal protocol instance (recvline.HistoricRecvLine) of my application's SSH terminal service's clients. Specifically, I'm trying to expose these three things (for all connected users) to my terminal commands which are using the plugin system. I'm not considerably far off from this example for the setup, http://twistedmatrix.com/projects/conch/documentation/examples/demo_recvline .tac, minus the telnet support. For the life of me I can't seem to determine how best to approach this. Ive come as far as getting a list of the terminal recvline.HistoricRecvLine instances and then being able to manipulate the client terminals directly, but I need the usernames from the avatars and session id as well. I imagine this is a pretty common and simple thing, so I'm hoping that somebody can point me in the right direction. Please help if you can. Thx, TWKiel

2 1

[Twisted-Python] Twisted Trial changes working directory
by Gerrat Rickert July 3, 2009

July 3, 2009

I have a (Twisted) program that saves & uses a file in the current working directory. When running tests for the program, Trial changes the working directory, causing this file to be written into the '_trial_temp' directory instead. ...so the program works differently when run via Trial vs. in normal use. There are a number of solutions to this, but I'd just like to understand the reasoning behind this surprising behaviour. The docs say that this: "...allows them to write whatever data they like to disk, and not worry about polluting the current working directory" ...couldn't they just do this without changing the working directory? One of the things I love about python is that things just seem to work like you expect - I'm rarely surprised by it's behaviour. Twisted, OTOH, continually surprises me...and not like getting-a-pony-on-your-10th-birthday type of surprise. Don't get me wrong, I love Twisted...it makes writing networked/asynchronous programs a breeze. ...everything I write now (in Python), just seems to "need" Twisted. I just find Twisted's idiosyncrasies frustrating sometimes. Sorry for the rant, Gerrat

2 2

[Twisted-Python] Perspective broker remote calls to multiple servers seem to run sequentially
by Sean Hollingsworth July 3, 2009

July 3, 2009

Let me start off by warning everyone that I am extremely new to twisted (I've been using it maybe a week) and it's highly likely some things don't work the way I think they should. Or, I'm just using them incorrectly. Some background on what I'm trying to accomplish: I have a client/server system that reads in data values from a file (anywhere between 10k and 1.5M, running every hour) and sends those off to a server for processing. Processing each value takes some variable amount of time and, given that I am doing this every hour, I need it to be as fast as possible, which is why I was drawn to the asynchronous nature of python. I'm using python because everything we do here is in python. I had the app working well with a single server, but I need to scale it out to multiple servers to get the processing time down. At first, I started writing my own distributedish app, which worked (more or less), but had a enough niggly issues that I started looking for something else. This morning I discovered the Perspective Broker and that seemed to be the solution I needed. The problems: Right now, for development and testing, I have three servers running on localhost on three ports. When I run my test code, it seems that each Perspective Broker runs sequentially, rather than concurrently. For example, if i watch the debugging output from the server, I see the following: server 0 processing dataval1 server 0 processing dataval2 server 0 processing dataval3 server 0 processing dataval4 server 1 processing dataval*5* server 1 processing dataval6 server 1 processing dataval7 server 2 processing dataval8 server 2 processing dataval9 server 2 processing dataval10 My understanding is that the perspective brokers would work concurrently. Is that incorrect? My guess is that they should work concurrently and I am just doing something wrong in my code, due to my very, very limited understanding of how they work. Well, how much of twisted works, really. Below is my the relevant code for the client (I've taken out code that just deals with prepping data or debugging). Please keep in mind that this is mostly testing code, while I get a better understanding of how perspective brokers work and was cobbled together from examples and docs found online. CLIENT CODE: import sys, os from twisted.spread import pb from twisted.internet import reactor import Queue MY_PORT = 49981 SERVERS = 3 class Client: def __init__(self, host): self.out_count = 0 self.in_count = 0 self.total = 0 self.in_file = open(sys.argv[1], 'rb') self.out_file = open(sys.argv[2], 'wb') # Create data queues for each server so that we can send data to each server self.queues = [] for server in range(0,SERVERS): self.queues.append(Queue.Queue()) # Populate queues with data pos = 0 for dv in self.in_file: dv = dv.strip() self.queues[pos % len(self.queues)].put(dv) pos += 1 # Conenct to servers print 'Connecting to Servers...' for server in range(0, COUNT): factory = pb.PBClientFactory() reactor.connectTCP(host, server + MY_PORT, factory) factory.getRootObject().addCallbacks(self.connected, self.connect_failure, [self.queues[server]]) def connected(self, perspective, queue): print "connected" while 1: try: dv = queue.get_nowait() except Queue.Empty: break else: perspective.callRemote('process_data', dv).addCallbacks(self.success, self.lookup_failure) def success(self, result): print result dv, answer = result self.in_count += 1 self.out_file.write('%s,%s\n' % (dv, answer)) if(self.in_count == self.out_count): self.out_file.close() reactor.stop() def connect_failure(self, _): print "Remote connect failure" self.out_file.close() reactor.stop() def lookup_failure(self, _): print "Remote lookup failure" self.out_file.close() reactor.stop() Client("127.0.0.1") reactor.run() END CLIENT CODE Any help anyone can provide will be greatly appreciated. I have another question, too, but it can wait till this one is solved, since, for all I know, whatever is causing this problem may be causing the other one. Sean M Hollingsworth

2 2

Re: [Twisted-Python] Need working examples of imap4 client.
by Pywinder Singh July 2, 2009

July 2, 2009

Phil, Much obliged. Was able to isolate and confirm that the real issue is that login is hanging for some reason, and causing the timeout message. I suspect SSL/TLS issues, which at least gives me a decided place to start investigating. Thanks again for the code, it was sanity inducing to say the least. ;) Phil Mayers wrote: > Pywinder Singh wrote: > > > > Ideally, I'd love to see a snipped which is able to log into an imap > > server and gets a list of mailboxes. If the example on the site works > > Here you go: > > #!/usr/bin/python > > from twisted.internet import reactor, protocol, defer > from twisted.mail import imap4 > > # change these... > username = '?' > password = '?' > servername = '?' > > def mailboxes(list): > for flags,sep,mbox in list: > print mbox > > def loggedin(res, proto): > d = proto.list('','*') > d.addCallback(mailboxes) > return d > > def connected(proto): > print "connected", proto > d = proto.login(username, password) > d.addCallback(loggedin, proto) > return d > > def failed(f): > print "failed", f > return f > > def done(_): > reactor.callLater(0, reactor.stop) > > def main(): > c = protocol.ClientCreator(reactor, imap4.IMAP4Client) > d = c.connectTCP(servername, 143) > d.addCallbacks(connected, failed) > d.addBoth(done) > > reactor.callLater(0, main) > reactor.run() > > > This example makes use of deferred chaining i.e. returning a deferred > from a callback handler, so you'll want to understand that. >

2 3

[Twisted-Python] problem on running apache2 + scgi
by Chris July 2, 2009

July 2, 2009

Hi, I was working on a project which use twisted.web2 scgi.I chose apache2 as the front end web server.the apache2 server will receive a http POST message which carry about 200k-1M data.I found that sometimes I get the correct answer but sometimes there is a exception: Traceback (most recent call last): File "//usr/lib64/python2.5/site-packages/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "//usr/lib64/python2.5/site-packages/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib64/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib64/python2.5/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib64/python2.5/site-packages/twisted/internet/tcp.py", line 463, in doRead return self.protocol.dataReceived(data) File "/usr/lib64/python2.5/site-packages/twisted/web2/channel/scgi.py", line 72, in dataReceived self.request.handleContentChunk(data) File "/usr/lib64/python2.5/site-packages/twisted/web2/http.py", line 393, in handleContentChunk self.stream.write(data) File "/usr/lib64/python2.5/site-packages/twisted/web2/http.py", line 331, in write stream.ProducerStream.write(self, data) File "/usr/lib64/python2.5/site-packages/twisted/web2/stream.py", line 667, in write self.producer.pauseProducing() exceptions.AttributeError: SCGIChannelRequest instance has no attribute 'pauseProducing' it seems the error is from the twisted.web2 module,usually when the data is very big,I will get this exception.how to avoid this exception? I have to transfer the data in one request anyway. Any suggestion is helpful.Thank you! Regards Chris

1 0