Mailman 3 October 2007 - Twisted

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 26 Oct '07

26 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 26, 2007 2:05 PM > To: Twisted general discussion > Subject: RE: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 26 Oct 2007 11:41:50 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > > > > [snip] > > > >Could you point me in the direction of a better example? I > assumed that > >I could find what I needed if I dug hard enough in the API > docs, but I > >just kept reaching dead ends. > > The API docs are nice (in places), but nothing really beats > diving into the > source itself. There are some areas of Twisted which use > obscure techniques > to achieve a goal, but for the most part the code isn't too > hard to follow > (and this is improving all the time). If I had the code in > front of me again > I could probably tell you the particular path I traced up > from your code into > conch to find a point where it was clear that the "closed" > method is what you > want to override, but I don't at the moment - but that's all > I did, looked at > the source, figured out what was called by what, step by step > until I found > the feature you were interested in. Now, I've used conch > before, and maybe > even this feature of it, but I didn't consciously remember > the answer until > I saw it in the code again this time. Previous experience > might have guided > me a bit in the right direction, but not much beyond that. > > I might be able to provide some more specific guidance if > you're still not > finding what you need, but probably not until sometime next week. > > Jean-Paul > Thanks, I'll start cracking open the source and let you know how I make out. --Brian

1 0

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 26 Oct '07

26 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 19, 2007 5:07 PM > To: Twisted general discussion > Subject: RE: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 19 Oct 2007 16:53:00 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > > > > [snip] > > > > def openShell(self, trans): > > ep = EchoProtocol() > > ep.makeConnection(trans) > > trans.makeConnection(session.wrapProtocol(ep)) > > > > def eofReceived(self): > > pass > > > > def closed(self): > > pass > > > > Here, you implemented connection set up, invoking makeConnection on > the protocol you instantiated. However, you didn't do > anything similar > for connectionLost. You may want to keep a list of the protocol > instances openShell creates and give them disconnection notification > in closed (which is being called in this example). > > Jean-Paul > I'm trying various things to do what you suggest, but the Twisted API docs are hard for me to follow sometimes. I tried making "eofReceived" and "closed" just print a debug line (after calling my "connectionLost function didn't appear to do anything), and neither appeared to fire when I closed the ssh client. I tried looking up other examples I could possibly follow on the net, but came up only with the same examples that are on the Twisted site. I can't even figure out where "trans" comes from or what it is :( . Could you point me in the direction of a better example? I assumed that I could find what I needed if I dug hard enough in the API docs, but I just kept reaching dead ends. Thanks for all your help! --Brian

2 1

[Twisted-Python] Using reactor to connect to multiple servers
by Vijay Mathew 26 Oct '07

26 Oct '07

Hi all, I am in the process of writing a http client using twisted. I want it to connect to multiple http servers and download more than one page asynchronously, without using threads. How can I use the reactor object to connect to multiple servers and download data like this? thanks, --Vijay

2 1

[Twisted-Python] Twistd problem with redirection
by Roland Hedberg 24 Oct '07

24 Oct '07

Hi! I've encountered a strange problem which I'd like to have some help with. I'm building some software using Twisted and have encountered the following problem. I use twistd 2.5.0 and starts the server using: twistd pyom --logfile=x.log ... Now, the tap file that is run by twistd loads and initiates modules dependent on the configuration. And these modules might load other modules. So, what happens is that, depending (looks like it) on how complex the setup is, the redirect of stdout output for print to the logfile is not universally done. Consistent but depending on the configuration, the output from print for one of modules continues to go to stdout instead of to the logfile. Anyone got a clue as to what is happening and what I can do about it ? -- Roland

2 2

[Twisted-Python] CFReactor redux
by Phil Christensen 24 Oct '07

24 Oct '07

It's been awhile since this was brought up on the mailing list, but I'd like to see if anyone has any real interest in using Twisted with Cocoa on Mac OS X. It was suggested to me that one thing that was needed was a reactor based on ThreadedSelectReactor that would integrate with the Cocoa runtime. I've posted my results here: http://twistedmatrix.com/trac/ticket/1833 This class is extremely trivial, but it makes working with PyObjC/ Twisted apps a bit more straightforward. However, there are significant issues with the entire TSR approach. The primary one is that it's impossible to do any Trial testing, because as far as I can tell, it is impossible to make the CFReactor's run() method return without terminating the entire process. I have a couple questions for the group, namely: 1.) Does anyone with a better grasp of Cocoa/PyObjC development have any ideas on how a suitable reactor might be implemented? 2.) Is there any way to run the unit tests without restarting the reactor between each test? I'm sure this would cause many tests to fail, but at least we'd be able to see if there was any other major issues. I'm not suggesting any changes be made to trial. I'm just looking for any possible way to run the tests manually. 3.) Will anyone use this? I know the core devs are generally uninterested in supporting a Cocoa reactor, but I'm glad to maintain this if it would be used. -phil

3 3

[Twisted-Python] reading serial port asynchronously
by Tom Brown 23 Oct '07

23 Oct '07

Hi, I have an application that reads characters off the serial port in a thread. The problem is, the entire application will hang sometimes. I believe it is a threading problem, but I can't pinpoint why the application hangs. So, I'd like to get away from threads and thought Twisted may help me. I was told Twisted has the ability to interface with serial ports. I don't know where to look for the information on how to do this. Any ideas on where to start looking or pointers on how to accomplish this? Thanks, Tom

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 21 Oct '07

21 Oct '07

Bug summary ______________________ Summary for 2007-10-14 through 2007-10-21 Bugs opened: 4 Bugs closed: 1 Total open bugs: 1043 (+3) |== Type Changes |== Priority Changes |== Component Changes |Defect: +2 |Highest: -1 |Core: +3 |Enhancement: +1 |Normal: +3 |Names: +1 |Lowest: +1 |Web2: -1 New / Reopened Bugs ______________________ ===== Normal ===== [#2850] getHostByName Error when DNS Server return Name Server (opened by wangminghua) defect names http://twistedmatrix.com/trac/ticket/2850 [#2851] No hooks in NetstringReceiver to catch parse errors. (opened by jml) enhancement core http://twistedmatrix.com/trac/ticket/2851 [#2853] findFailure result is used exclusively, ignoring new traceback (opened by Peaker) defect core http://twistedmatrix.com/trac/ticket/2853 ===== Lowest ===== [#2852] spawnProcess doc glitch (opened by alanfranzoni) defect core http://twistedmatrix.com/trac/ticket/2852 Closed Bugs ______________________ ===== Highest ===== [#2304] t.web2.client.http.HTTPClientChannelRequest.connectionLost does nothing (opened by ghazel, closed by therve, fixed) defect web2 http://twistedmatrix.com/trac/ticket/2304 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2853] findFailure result is used exclusively, ignoring new traceback (since 2007-10-17 22:04:16). Mean open ticket age: 544 days, 17:31:29.430966. Median: 446 days, 3:59:09.191273. Standard deviation: 404 days, 0:42:51.794059. Interquartile range: 505 days, 3:59:29. Mean time between ticket creation and ticket resolution: 122 days, 16:37:43.001128. Median: 14 days, 16:22:15. Standard deviation is 226 days, 21:33:13.531588. The interquartile range is 130 days, 2:23:34. Mean time spent in review: 68 days, 20:15:20.554302. Median: 4 days, 9:08:37. Standard deviation: 151 days, 9:40:09.422509. Interquartile range: 31 days, 18:40:54. Mean number of times a ticket is reviewed: 1.96567505721. Median: 1 Standard deviation: 1.61407221677. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 12 unique ticket reporters 7 unique ticket reviewers 2 unique ticket resolvers In the last 24 weeks, 69 unique ticket reporters 16 unique ticket reviewers 13 unique ticket resolvers In the last 48 weeks, 152 unique ticket reporters 18 unique ticket reviewers 25 unique ticket resolvers

1 0

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 19 Oct '07

19 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 19, 2007 4:29 PM > To: Twisted general discussion > Subject: Re: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 19 Oct 2007 16:13:20 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > >I'm looking for what is probably an simple solution I can't > figure out > >on my own. I'm writing an SSH server based on the example on the web > >(using conch). I'm trying to figure out how to detect when > the client > >exists (for example, when I just close out PuTTY), but I > can't get this > >to work right. Looking through the API docs I found > "connectionLost()", > >which I put in my protocol class (EchoProtocol in the > example), but it's > >never getting called. > > > >Can someone please tell me what I'm doing wrong? > > You seem to be on the right track. However, you haven't > supplied enough > details for anyone to be sure why it isn't behaving as you > expect. If you > can post a minimal example of a program which you think > should be working > (ie, giving you connection lost notification) but isn't, then > someone can > probably point you to a solution. > > Jean-Paul > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > Sorry about that. Here is most of what is involved. (Code below) I would think that when I close my PuTTY window, the connectionLost function in the EchoProtocol would be called...but clearly I'm missing some crucial step. --Brian from twisted.cred import portal, checkers from twisted.conch import error, avatar from twisted.conch.checkers import SSHPublicKeyDatabase from twisted.conch.ssh import factory, userauth, connection, keys, session from twisted.internet import reactor, protocol, defer from twisted.python import log from zope.interface import implements import sys import socket import threading log.startLogging(sys.stderr) """Example of running another protocol over an SSH channel. log in with username "user" and password "password". """ class defaultdict(dict): def __init__(self, default=None): dict.__init__(self) self.default = default def __getitem__(self, key): try: return dict.__getitem__(self, key) except KeyError: return self.default class ExampleAvatar(avatar.ConchUser): def __init__(self, username): avatar.ConchUser.__init__(self) self.username = username self.channelLookup.update({'session':session.SSHSession}) class ExampleRealm: implements(portal.IRealm) def requestAvatar(self, avatarId, mind, *interfaces): return interfaces[0], ExampleAvatar(avatarId), lambda: None class EchoProtocol(protocol.Protocol): """this is our example protocol that we will run over SSH """ def connectionMade(self): self.transport.write("Connected!\r\n") self.commandBuffer = "" def connectionLost(self,reason): print "lost:%s" %reason self.serialServerSocket.close() self.alive = False self.serialSocketReadThread.join() publicKey = 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEArzJx8OYOnJmzf4tfBEvLi8DVPrJ3/c9k2I/Az64fxjHf 9imyRJbixtQhlH9lfNjUIx+4LmrJH5QNRsFporcHDKOTwTTYLh5KmRpslkYHRivcJSkbh/C+ BR3utDS555mV' privateKey = """-----BEGIN RSA PRIVATE KEY----- MIIByAIBAAJhAK8ycfDmDpyZs3+LXwRLy4vA1T6yd/3PZNiPwM+uH8Yx3/YpskSW 4sbUIZR/ZXzY1CMfuC5qyR+UDUbBaaK3Bwyjk8E02C4eSpkabJZGB0Yr3CUpG4fw vgUd7rQ0ueeZlQIBIwJgbh+1VZfr7WftK5lu7MHtqE1S1vPWZQYE3+VUn8yJADyb Z4fsZaCrzW9lkIqXkE3GIY+ojdhZhkO1gbG0118sIgphwSWKRxK0mvh6ERxKqIt1 xJEJO74EykXZV4oNJ8sjAjEA3J9r2ZghVhGN6V8DnQrTk24Td0E8hU8AcP0FVP+8 PQm/g/aXf2QQkQT+omdHVEJrAjEAy0pL0EBH6EVS98evDCBtQw22OZT52qXlAwZ2 gyTriKFVoqjeEjt3SZKKqXHSApP/AjBLpF99zcJJZRq2abgYlf9lv1chkrWqDHUu DZttmYJeEfiFBBavVYIF1dOlZT0G8jMCMBc7sOSZodFnAiryP+Qg9otSBjJ3bQML pSTqy7c3a2AScC/YyOwkDaICHnnD3XyjMwIxALRzl0tQEKMXs6hH8ToUdlLROCrP EhQ0wahUTCk1gKA4uPD6TMTChavbh4K63OvbKg== -----END RSA PRIVATE KEY-----""" class InMemoryPublicKeyChecker(SSHPublicKeyDatabase): def checkKey(self, credentials): return credentials.username == 'user' and \ keys.getPublicKeyString(data=publicKey) == credentials.blob class ExampleSession: def __init__(self, avatar): """ We don't use it, but the adapter is passed the avatar as its first argument. """ def getPty(self, term, windowSize, attrs): pass def execCommand(self, proto, cmd): raise Exception("no executing commands") def openShell(self, trans): ep = EchoProtocol() ep.makeConnection(trans) trans.makeConnection(session.wrapProtocol(ep)) def eofReceived(self): pass def closed(self): pass from twisted.python import components components.registerAdapter(ExampleSession, ExampleAvatar, session.ISession) class ExampleFactory(factory.SSHFactory): publicKeys = { 'ssh-rsa': keys.getPublicKeyString(data=publicKey) } privateKeys = { 'ssh-rsa': keys.getPrivateKeyObject(data=privateKey) } services = { 'ssh-userauth': userauth.SSHUserAuthServer, 'ssh-connection': connection.SSHConnection } portal = portal.Portal(ExampleRealm()) passwdDB = checkers.InMemoryUsernamePasswordDatabaseDontUse() passwdDB.addUser('user', 'password') portal.registerChecker(passwdDB) portal.registerChecker(InMemoryPublicKeyChecker()) ExampleFactory.portal = portal if __name__ == '__main__': reactor.listenTCP(5022, ExampleFactory()) reactor.run()

2 1

[Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 19 Oct '07

19 Oct '07

I'm looking for what is probably an simple solution I can't figure out on my own. I'm writing an SSH server based on the example on the web (using conch). I'm trying to figure out how to detect when the client exists (for example, when I just close out PuTTY), but I can't get this to work right. Looking through the API docs I found "connectionLost()", which I put in my protocol class (EchoProtocol in the example), but it's never getting called. Can someone please tell me what I'm doing wrong? Thanks, --Brian

2 1

Re: [Twisted-Python] Re: Content Encoding : gzip ?
by anurag uniyal 19 Oct '07

19 Oct '07

Thanks. So it seems I have to write my own code, there in nothing like I can set some flag. So if that is the case shouldn't I just be gzipping and unzipping data write and read to request object. rgds anurag ----- Original Message ---- From: David Bolen <db3l.net(a)gmail.com> To: twisted-python(a)twistedmatrix.com Sent: Friday, October 19, 2007 1:02:21 AM Subject: [Twisted-Python] Re: Content Encoding : gzip ? anurag uniyal <anuraguniyal(a)yahoo.com> writes: > Is it possible to send and recieve compressed content using twisted.web? > > I have set Accept-encoding to gzip but it doesn't make any difference. > Will I have to cater for this myself? Yes, in twisted.web. In twisted.web2, there's a filter that supports it. Here's a small wrapper class I wrote recently that I've used to handle this in twisted.web, with the goal of minimal changes to existing resource code. I use it by noticing, in the original request handler, the ability to handle gzip encodings, and then wrap the original request in this class (since there's no built-in filtering of the output stream in twisted.web). It's worked fine in my tests to date, and in limited production use, but to be honest it hasn't seen large scale, long term use because my original use case only yielded ~15% compression (multimedia files) and using this negates the ability to know the full length in advance, thus clients can't give projected download times. (My files are large enough that compressing twice is high overhead, and it wouldn't play well with the wrapper approach anyway). In any event, hopefully it'll be a useful starting point for you. Here's the wrapper: - - - - - - - - - - - - - - - - - - - - - - - - - import struct import zlib class GzipRequest(object): """Wrapper for a request that applies a gzip content encoding""" def __init__(self, request, compressLevel=6): self.request = request self.request.setHeader('Content-Encoding', 'gzip') # Borrowed from twisted.web2 gzip filter self.compress = zlib.compressobj(compressLevel, zlib.DEFLATED, -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0) def __getattr__(self, attr): if 'request' in self.__dict__: return getattr(self.request, attr) else: raise AttributeError, attr def __setattr__(self, attr, value): if 'request' in self.__dict__: return setattr(self.request, attr, value) else: self.__dict__[attr] = value def write(self, data): if not self.request.startedWriting: self.crc = zlib.crc32('') self.size = self.csize = 0 # XXX: Zap any length for now since we don't know final size if 'content-length' in self.request.headers: del self.request.headers['content-length'] # Borrow header information from twisted.web2 gzip filter self.request.write('\037\213\010\000' '\0\0\0\0' '\002\377') self.crc = zlib.crc32(data, self.crc) self.size += len(data) cdata = self.compress.compress(data) self.csize += len(cdata) if cdata: self.request.write(cdata) elif self.request.producer: # Simulate another pull even though it hasn't really made it # out to the consumer yet. self.request.producer.resumeProducing() def finish(self): remain = self.compress.flush() self.csize += len(remain) if remain: self.request.write(remain) self.request.write(struct.pack('<LL', self.crc & 0xFFFFFFFFL, self.size & 0xFFFFFFFFL)) self.request.finish() - - - - - - - - - - - - - - - - - - - - - - - - - and here's a sample of using it. This code is from one of my Resource objects - if I were to use it more generally in my site I'd extract that into a mix-in class of some sort - currently there was only one specific resource (the multimedia files) I wanted to support it for: def render_GET(self, request): # (... argument validation ...) accept_encoding = request.getHeader('accept-encoding') if accept_encoding: encodings = accept_encoding.split(',') for encoding in encodings: name = encoding.split(';')[0].strip() if name == 'gzip': request = GzipRequest(request) break # At this point, use 'request' as normal -- David _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

Twisted October 2007