Mailman 3 October 2007 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 05 Aug '21

05 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool 28 Apr '13

28 Apr '13

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] Perspective broker and pipes
by Nitro 05 Nov '07

05 Nov '07

Hello, I wrote a pb application which also uses udp sockets. It works nicely. Now there's the situation where client and server are both located on the same machine. This means I could just connect to localhost and everything works. Unfortunately some firewalls warn about connections. I want to avoid this. So I thought about replacing the transports in pb with pipes instead of sockets if the server and client are both on the same machine. I'd have to do the same for my custom udp solution. Doing it for my custom udp code is easy, but how do I do this with pb? Do I just have to create some kind of reactor.createPipe() thing or is there more to it? Thanks for your help, -Matthias

4 7

[Twisted-Python] Deferreds and Trial -- getting my timing right
by Brendon Colby 01 Nov '07

01 Nov '07

Greetings, I've paused development to bring my unittests up to date, and have hit another snag. I am under the impression that no real work should be done in my factory's init, so I have a bunch of routines that load data from a DB in startFactory() (they all return deferreds from adbapi). I'm not sure when startFactory() runs, but I know it's definitely not running before my tests fire. I'm curious: do I need to move my data loading back into init, save the deferred and return that deferred from setUp to have all my pre-loading complete before my tests run? When does startFactory() actually fire? Thanks, Brendon Colby

2 4

[Twisted-Python] asyncronous access to serial ports under windows
by Tom Brown 29 Oct '07

29 Oct '07

Hi, I've been able to create an application in Linux that reads/writes multiple serial ports asyncronously. The setup code that does this looks like this: ... from twisted.internet.qtreactor import install a = QApplication(argv) install(a) from twisted.internet import reactor from twisted.internet.serialport import SerialPort ... ports, badPorts = getGoodPorts() if not ports: exit(1) data = ConfigData(join(sep, 'etc', 'qa.conf')) dbInfo = copy(data['qadata']) getLogin(dbInfo) w = MainWindow(data, dbInfo, ports) w.show() reactor.addSystemEventTrigger('after', 'shutdown', a.quit) a.connect(a, SIGNAL('lastWindowClosed()'), reactor.stop) for portObj in w.portObjs: SerialPort(portObj.scanner, portObj.port, reactor, baudrate=38400) portObj.sendLine() reactor.run() Where the portObj.scanner is an instance of a descendent of Protocol. Like I said, the above code works under Linux. Then I tried porting this to Windows. The first problem I came across is that the qtreactor.py would not work. I had to subclass QTReactor from Win32Reactor. It runs without errors. However, I am not reading anything off of the serial port. I can see the lights blink on the port when the portObj.sendLine() is called, so I believe I am writing to it ok and data is coming back. The data is just not read by the application. I think it must have something to do with the SerialPort instance not getting an even that data is ready. I suspect this is a Windows issue in that Windows is not signaling an event when data is ready to read on the serial port. Does anybody have any experience with this? Is there a work around? Am I doing something wrong? Thanks, Tom

1 2

[Twisted-Python] IPushProducer - medium volume streaming
by Rutt, Benjamin 29 Oct '07

29 Oct '07

I've got an IPushProducer written with these 3 methods and it works fine - i.e. resumeProducing() is called, then eventually pauseProducing is called() when the producer produces data too fast; when the client disconnects early, the stopProducing method is called, etc.. def pauseProducing(self): def resumeProducing(self): def stopProducing(self): My problem is, the source of the data (a slow non-relational database) that I send back to the client from within resumeProducing generates data kind of slowly. As in, it is quite rare that pauseProducing is ever called in my tests. I feel that the data source generates data too slowly, and have figured out I will block the entire server during the resumeProducing operation (for example, during the first call to resumeProducing, I do not generate any data at all for the first 20 seconds during bootstrapping). At this point, I am thinking about the following ideas of code changes to prevent blockage of the server: 1) forking off a thread so that I will not block other clients from talking to the server. I will not have very many clients using this producer, so this would not result in any large numbers of threads which I know can be a problem. I know how to spawn Python threads and use socket APIs, but it is not clear to me how I would get at the socket descriptor from within the twisted framework (I am using a LineReceiver subclass, which in this case is spawning the IPushProducer). Are there any examples of spawning off a thread from a LineReceiver class and communicating using blocking calls within said thread? 2) returning from resumeProducing after a few seconds of production, even though I could produce more, so I do not block the server. I have tested this, it works, and does let other clients get in, but I still feel it is suboptimal, since my data source is so slow, it still blocks the server during each of those few second intervals. Any recommendations here? Thank you.

3 9

[Twisted-Python] Proxy server in front of two web sites
by Stefan Arentz 29 Oct '07

29 Oct '07

I'm looking at implementing a solution for the following problem: I have a setup with two web servers. Each contains the same directory structure but not the same files. Using a proxy server in front of these two I would like to merge them into one. So the proxy would first hit server A and if A returned a 404 then it would hit B. Is this something that I should implement as a web server or as a proxy server? Is there actually a difference? S.

3 3

[Twisted-Python] Re: [Twisted-web] installed twisted. twisted.web missing?
by Phil Christensen 29 Oct '07

29 Oct '07

I saw this pop up over on -web, but I'd like to ask about this... On Oct 29, 2007, at 3:49 AM, David Reid wrote: > On Oct 28, 2007, at 6:14 PM, aaron smith wrote: >> quick newb question. I installed twisted from source on mac os x >> leopard. When I run: >> >> >>python >> >>import twisted >> >>from twisted.web import http >> >> I get an ImportError: ImportError: No module named web > > Unfortunately there is an incomplete distribution of twisted 2.4 in / > System/Library/Frameworks > > Your source install likely went into /Library/Python/2.5/site- > packages/ which is also unfortunately after /System/Library/ > Frameworks in the PYTHONPATH. > > Put /Library/Python/2.5/site-packages/ at the front of your > PYTHONPATH. This seems like a relatively large issue with respect to Twisted support on Mac OS X. Is it really going to be necessary to set an environment variable anywhere you want to use Twisted? I guess this would be an issue for any of the packages distributed in Extras, but this is the one I have the most opinions about ;-)... Is there some other way to modify your path? I assume .pth files won't work for this, since they only append to the path. -phil

1 1

[Twisted-Python] python2.5
by Sergey Alembekov 29 Oct '07

29 Oct '07

Hi, Will twisted works with python2.5? -- Regards, Sergey Alembekov ALTLinux Team xmpp: rt at jabber.ru

3 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 28 Oct '07

28 Oct '07

Bug summary ______________________ Summary for 2007-10-21 through 2007-10-28 Bugs opened: 4 Bugs closed: 10 Total open bugs: 1037 (-6) |== Type Changes |== Priority Changes |== Component Changes |Defect: -3 |Highest: -2 |Core: -5 |Enhancement: -2 |High: +1 |Ftp: +1 |Normal: -4 |Trial: +0 |Web: -1 New / Reopened Bugs ______________________ ===== Highest ===== [#2855] Add tests for ftp.IFTPShell (opened by therve) enhancement ftp http://twistedmatrix.com/trac/ticket/2855 [#2856] trial verbose color reporter should colorize final "PASSED" and "FAILED" message (opened by exarkun) (CLOSED, fixed) enhancement trial http://twistedmatrix.com/trac/ticket/2856 [#2766] test_sslverify uses deprecated function log.flushErrors (opened by therve) defect core http://twistedmatrix.com/trac/ticket/2766 ===== High ===== [#2854] Reactor claims not be running although reactor.run() has been called (opened by htj) defect core http://twistedmatrix.com/trac/ticket/2854 Closed Bugs ______________________ ===== Highest ===== [#2767] test_ident uses deprecated function log.flushErrors (opened by therve, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2767 [#2685] factor secure random implementation (opened by antoine, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2685 [#2766] test_sslverify uses deprecated function log.flushErrors (opened by therve, closed by therve, ) defect core http://twistedmatrix.com/trac/ticket/2766 [#1220] Should the coding standard mention PEP8 (opened by hypatia, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/1220 [#2856] trial verbose color reporter should colorize final "PASSED" and "FAILED" message (opened by exarkun, closed by therve, fixed) enhancement trial http://twistedmatrix.com/trac/ticket/2856 [#2643] twisted.python.modules does not account for memory/disk inconsistencies when scanning packages (opened by glyph, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2643 ===== Normal ===== [#1909] [PATCH] add content encoding support to xmlrpc (opened by pythonguy, closed by exarkun, invalid) enhancement web http://twistedmatrix.com/trac/ticket/1909 [#2642] Circular reference in tcp.BaseClient (opened by mircea, closed by therve, wontfix) defect core http://twistedmatrix.com/trac/ticket/2642 [#2727] plugin howto refers to "getPlugin" instead of "getPlugins" (opened by exarkun, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2727 [#2692] before-shutdown triggers are fired multiple times (opened by yaubi, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2692 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2855] Add tests for ftp.IFTPShell (since 2007-10-23 05:25:55). Mean open ticket age: 552 days, 22:31:27.885002. Median: 453 days, 13:26:47.401878. Standard deviation: 404 days, 14:59:20.224490. Interquartile range: 504 days, 20:17:18. Mean time between ticket creation and ticket resolution: 123 days, 5:12:33.030303. Median: 15 days, 1:19:40. Standard deviation is 226 days, 23:50:09.969125. The interquartile range is 131 days, 17:01:58. Mean time spent in review: 69 days, 7:13:09.122129. Median: 4 days, 9:08:37. Standard deviation: 153 days, 1:45:54.575417. Interquartile range: 31 days, 18:48:10. Mean number of times a ticket is reviewed: 1.98177676538. Median: 1 Standard deviation: 1.63103040096. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 12 unique ticket reporters 7 unique ticket reviewers 2 unique ticket resolvers In the last 24 weeks, 67 unique ticket reporters 16 unique ticket reviewers 13 unique ticket resolvers In the last 48 weeks, 151 unique ticket reporters 18 unique ticket reviewers 25 unique ticket resolvers

1 0

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 26 Oct '07

26 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 26, 2007 2:05 PM > To: Twisted general discussion > Subject: RE: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 26 Oct 2007 11:41:50 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > > > > [snip] > > > >Could you point me in the direction of a better example? I > assumed that > >I could find what I needed if I dug hard enough in the API > docs, but I > >just kept reaching dead ends. > > The API docs are nice (in places), but nothing really beats > diving into the > source itself. There are some areas of Twisted which use > obscure techniques > to achieve a goal, but for the most part the code isn't too > hard to follow > (and this is improving all the time). If I had the code in > front of me again > I could probably tell you the particular path I traced up > from your code into > conch to find a point where it was clear that the "closed" > method is what you > want to override, but I don't at the moment - but that's all > I did, looked at > the source, figured out what was called by what, step by step > until I found > the feature you were interested in. Now, I've used conch > before, and maybe > even this feature of it, but I didn't consciously remember > the answer until > I saw it in the code again this time. Previous experience > might have guided > me a bit in the right direction, but not much beyond that. > > I might be able to provide some more specific guidance if > you're still not > finding what you need, but probably not until sometime next week. > > Jean-Paul > Thanks, I'll start cracking open the source and let you know how I make out. --Brian

1 0

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 26 Oct '07

26 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 19, 2007 5:07 PM > To: Twisted general discussion > Subject: RE: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 19 Oct 2007 16:53:00 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > > > > [snip] > > > > def openShell(self, trans): > > ep = EchoProtocol() > > ep.makeConnection(trans) > > trans.makeConnection(session.wrapProtocol(ep)) > > > > def eofReceived(self): > > pass > > > > def closed(self): > > pass > > > > Here, you implemented connection set up, invoking makeConnection on > the protocol you instantiated. However, you didn't do > anything similar > for connectionLost. You may want to keep a list of the protocol > instances openShell creates and give them disconnection notification > in closed (which is being called in this example). > > Jean-Paul > I'm trying various things to do what you suggest, but the Twisted API docs are hard for me to follow sometimes. I tried making "eofReceived" and "closed" just print a debug line (after calling my "connectionLost function didn't appear to do anything), and neither appeared to fire when I closed the ssh client. I tried looking up other examples I could possibly follow on the net, but came up only with the same examples that are on the Twisted site. I can't even figure out where "trans" comes from or what it is :( . Could you point me in the direction of a better example? I assumed that I could find what I needed if I dug hard enough in the API docs, but I just kept reaching dead ends. Thanks for all your help! --Brian

2 1

[Twisted-Python] Using reactor to connect to multiple servers
by Vijay Mathew 26 Oct '07

26 Oct '07

Hi all, I am in the process of writing a http client using twisted. I want it to connect to multiple http servers and download more than one page asynchronously, without using threads. How can I use the reactor object to connect to multiple servers and download data like this? thanks, --Vijay

2 1

[Twisted-Python] Twistd problem with redirection
by Roland Hedberg 24 Oct '07

24 Oct '07

Hi! I've encountered a strange problem which I'd like to have some help with. I'm building some software using Twisted and have encountered the following problem. I use twistd 2.5.0 and starts the server using: twistd pyom --logfile=x.log ... Now, the tap file that is run by twistd loads and initiates modules dependent on the configuration. And these modules might load other modules. So, what happens is that, depending (looks like it) on how complex the setup is, the redirect of stdout output for print to the logfile is not universally done. Consistent but depending on the configuration, the output from print for one of modules continues to go to stdout instead of to the logfile. Anyone got a clue as to what is happening and what I can do about it ? -- Roland

2 2

[Twisted-Python] CFReactor redux
by Phil Christensen 24 Oct '07

24 Oct '07

It's been awhile since this was brought up on the mailing list, but I'd like to see if anyone has any real interest in using Twisted with Cocoa on Mac OS X. It was suggested to me that one thing that was needed was a reactor based on ThreadedSelectReactor that would integrate with the Cocoa runtime. I've posted my results here: http://twistedmatrix.com/trac/ticket/1833 This class is extremely trivial, but it makes working with PyObjC/ Twisted apps a bit more straightforward. However, there are significant issues with the entire TSR approach. The primary one is that it's impossible to do any Trial testing, because as far as I can tell, it is impossible to make the CFReactor's run() method return without terminating the entire process. I have a couple questions for the group, namely: 1.) Does anyone with a better grasp of Cocoa/PyObjC development have any ideas on how a suitable reactor might be implemented? 2.) Is there any way to run the unit tests without restarting the reactor between each test? I'm sure this would cause many tests to fail, but at least we'd be able to see if there was any other major issues. I'm not suggesting any changes be made to trial. I'm just looking for any possible way to run the tests manually. 3.) Will anyone use this? I know the core devs are generally uninterested in supporting a Cocoa reactor, but I'm glad to maintain this if it would be used. -phil

3 3

[Twisted-Python] reading serial port asynchronously
by Tom Brown 22 Oct '07

22 Oct '07

Hi, I have an application that reads characters off the serial port in a thread. The problem is, the entire application will hang sometimes. I believe it is a threading problem, but I can't pinpoint why the application hangs. So, I'd like to get away from threads and thought Twisted may help me. I was told Twisted has the ability to interface with serial ports. I don't know where to look for the information on how to do this. Any ideas on where to start looking or pointers on how to accomplish this? Thanks, Tom

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 21 Oct '07

21 Oct '07

Bug summary ______________________ Summary for 2007-10-14 through 2007-10-21 Bugs opened: 4 Bugs closed: 1 Total open bugs: 1043 (+3) |== Type Changes |== Priority Changes |== Component Changes |Defect: +2 |Highest: -1 |Core: +3 |Enhancement: +1 |Normal: +3 |Names: +1 |Lowest: +1 |Web2: -1 New / Reopened Bugs ______________________ ===== Normal ===== [#2850] getHostByName Error when DNS Server return Name Server (opened by wangminghua) defect names http://twistedmatrix.com/trac/ticket/2850 [#2851] No hooks in NetstringReceiver to catch parse errors. (opened by jml) enhancement core http://twistedmatrix.com/trac/ticket/2851 [#2853] findFailure result is used exclusively, ignoring new traceback (opened by Peaker) defect core http://twistedmatrix.com/trac/ticket/2853 ===== Lowest ===== [#2852] spawnProcess doc glitch (opened by alanfranzoni) defect core http://twistedmatrix.com/trac/ticket/2852 Closed Bugs ______________________ ===== Highest ===== [#2304] t.web2.client.http.HTTPClientChannelRequest.connectionLost does nothing (opened by ghazel, closed by therve, fixed) defect web2 http://twistedmatrix.com/trac/ticket/2304 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2853] findFailure result is used exclusively, ignoring new traceback (since 2007-10-17 22:04:16). Mean open ticket age: 544 days, 17:31:29.430966. Median: 446 days, 3:59:09.191273. Standard deviation: 404 days, 0:42:51.794059. Interquartile range: 505 days, 3:59:29. Mean time between ticket creation and ticket resolution: 122 days, 16:37:43.001128. Median: 14 days, 16:22:15. Standard deviation is 226 days, 21:33:13.531588. The interquartile range is 130 days, 2:23:34. Mean time spent in review: 68 days, 20:15:20.554302. Median: 4 days, 9:08:37. Standard deviation: 151 days, 9:40:09.422509. Interquartile range: 31 days, 18:40:54. Mean number of times a ticket is reviewed: 1.96567505721. Median: 1 Standard deviation: 1.61407221677. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 12 unique ticket reporters 7 unique ticket reviewers 2 unique ticket resolvers In the last 24 weeks, 69 unique ticket reporters 16 unique ticket reviewers 13 unique ticket resolvers In the last 48 weeks, 152 unique ticket reporters 18 unique ticket reviewers 25 unique ticket resolvers

1 0

RE: [Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 19 Oct '07

19 Oct '07

> -----Original Message----- > From: twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf > Of Jean-Paul Calderone > Sent: Friday, October 19, 2007 4:29 PM > To: Twisted general discussion > Subject: Re: [Twisted-Python] Need some help with my first > Twisted program > > On Fri, 19 Oct 2007 16:13:20 -0400, "McCann, Brian" > <bmccann(a)andmore.com> wrote: > >I'm looking for what is probably an simple solution I can't > figure out > >on my own. I'm writing an SSH server based on the example on the web > >(using conch). I'm trying to figure out how to detect when > the client > >exists (for example, when I just close out PuTTY), but I > can't get this > >to work right. Looking through the API docs I found > "connectionLost()", > >which I put in my protocol class (EchoProtocol in the > example), but it's > >never getting called. > > > >Can someone please tell me what I'm doing wrong? > > You seem to be on the right track. However, you haven't > supplied enough > details for anyone to be sure why it isn't behaving as you > expect. If you > can post a minimal example of a program which you think > should be working > (ie, giving you connection lost notification) but isn't, then > someone can > probably point you to a solution. > > Jean-Paul > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > Sorry about that. Here is most of what is involved. (Code below) I would think that when I close my PuTTY window, the connectionLost function in the EchoProtocol would be called...but clearly I'm missing some crucial step. --Brian from twisted.cred import portal, checkers from twisted.conch import error, avatar from twisted.conch.checkers import SSHPublicKeyDatabase from twisted.conch.ssh import factory, userauth, connection, keys, session from twisted.internet import reactor, protocol, defer from twisted.python import log from zope.interface import implements import sys import socket import threading log.startLogging(sys.stderr) """Example of running another protocol over an SSH channel. log in with username "user" and password "password". """ class defaultdict(dict): def __init__(self, default=None): dict.__init__(self) self.default = default def __getitem__(self, key): try: return dict.__getitem__(self, key) except KeyError: return self.default class ExampleAvatar(avatar.ConchUser): def __init__(self, username): avatar.ConchUser.__init__(self) self.username = username self.channelLookup.update({'session':session.SSHSession}) class ExampleRealm: implements(portal.IRealm) def requestAvatar(self, avatarId, mind, *interfaces): return interfaces[0], ExampleAvatar(avatarId), lambda: None class EchoProtocol(protocol.Protocol): """this is our example protocol that we will run over SSH """ def connectionMade(self): self.transport.write("Connected!\r\n") self.commandBuffer = "" def connectionLost(self,reason): print "lost:%s" %reason self.serialServerSocket.close() self.alive = False self.serialSocketReadThread.join() publicKey = 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEArzJx8OYOnJmzf4tfBEvLi8DVPrJ3/c9k2I/Az64fxjHf 9imyRJbixtQhlH9lfNjUIx+4LmrJH5QNRsFporcHDKOTwTTYLh5KmRpslkYHRivcJSkbh/C+ BR3utDS555mV' privateKey = """-----BEGIN RSA PRIVATE KEY----- MIIByAIBAAJhAK8ycfDmDpyZs3+LXwRLy4vA1T6yd/3PZNiPwM+uH8Yx3/YpskSW 4sbUIZR/ZXzY1CMfuC5qyR+UDUbBaaK3Bwyjk8E02C4eSpkabJZGB0Yr3CUpG4fw vgUd7rQ0ueeZlQIBIwJgbh+1VZfr7WftK5lu7MHtqE1S1vPWZQYE3+VUn8yJADyb Z4fsZaCrzW9lkIqXkE3GIY+ojdhZhkO1gbG0118sIgphwSWKRxK0mvh6ERxKqIt1 xJEJO74EykXZV4oNJ8sjAjEA3J9r2ZghVhGN6V8DnQrTk24Td0E8hU8AcP0FVP+8 PQm/g/aXf2QQkQT+omdHVEJrAjEAy0pL0EBH6EVS98evDCBtQw22OZT52qXlAwZ2 gyTriKFVoqjeEjt3SZKKqXHSApP/AjBLpF99zcJJZRq2abgYlf9lv1chkrWqDHUu DZttmYJeEfiFBBavVYIF1dOlZT0G8jMCMBc7sOSZodFnAiryP+Qg9otSBjJ3bQML pSTqy7c3a2AScC/YyOwkDaICHnnD3XyjMwIxALRzl0tQEKMXs6hH8ToUdlLROCrP EhQ0wahUTCk1gKA4uPD6TMTChavbh4K63OvbKg== -----END RSA PRIVATE KEY-----""" class InMemoryPublicKeyChecker(SSHPublicKeyDatabase): def checkKey(self, credentials): return credentials.username == 'user' and \ keys.getPublicKeyString(data=publicKey) == credentials.blob class ExampleSession: def __init__(self, avatar): """ We don't use it, but the adapter is passed the avatar as its first argument. """ def getPty(self, term, windowSize, attrs): pass def execCommand(self, proto, cmd): raise Exception("no executing commands") def openShell(self, trans): ep = EchoProtocol() ep.makeConnection(trans) trans.makeConnection(session.wrapProtocol(ep)) def eofReceived(self): pass def closed(self): pass from twisted.python import components components.registerAdapter(ExampleSession, ExampleAvatar, session.ISession) class ExampleFactory(factory.SSHFactory): publicKeys = { 'ssh-rsa': keys.getPublicKeyString(data=publicKey) } privateKeys = { 'ssh-rsa': keys.getPrivateKeyObject(data=privateKey) } services = { 'ssh-userauth': userauth.SSHUserAuthServer, 'ssh-connection': connection.SSHConnection } portal = portal.Portal(ExampleRealm()) passwdDB = checkers.InMemoryUsernamePasswordDatabaseDontUse() passwdDB.addUser('user', 'password') portal.registerChecker(passwdDB) portal.registerChecker(InMemoryPublicKeyChecker()) ExampleFactory.portal = portal if __name__ == '__main__': reactor.listenTCP(5022, ExampleFactory()) reactor.run()

2 1

[Twisted-Python] Need some help with my first Twisted program
by McCann, Brian 19 Oct '07

19 Oct '07

I'm looking for what is probably an simple solution I can't figure out on my own. I'm writing an SSH server based on the example on the web (using conch). I'm trying to figure out how to detect when the client exists (for example, when I just close out PuTTY), but I can't get this to work right. Looking through the API docs I found "connectionLost()", which I put in my protocol class (EchoProtocol in the example), but it's never getting called. Can someone please tell me what I'm doing wrong? Thanks, --Brian

2 1

Re: [Twisted-Python] Re: Content Encoding : gzip ?
by anurag uniyal 19 Oct '07

19 Oct '07

Thanks. So it seems I have to write my own code, there in nothing like I can set some flag. So if that is the case shouldn't I just be gzipping and unzipping data write and read to request object. rgds anurag ----- Original Message ---- From: David Bolen <db3l.net(a)gmail.com> To: twisted-python(a)twistedmatrix.com Sent: Friday, October 19, 2007 1:02:21 AM Subject: [Twisted-Python] Re: Content Encoding : gzip ? anurag uniyal <anuraguniyal(a)yahoo.com> writes: > Is it possible to send and recieve compressed content using twisted.web? > > I have set Accept-encoding to gzip but it doesn't make any difference. > Will I have to cater for this myself? Yes, in twisted.web. In twisted.web2, there's a filter that supports it. Here's a small wrapper class I wrote recently that I've used to handle this in twisted.web, with the goal of minimal changes to existing resource code. I use it by noticing, in the original request handler, the ability to handle gzip encodings, and then wrap the original request in this class (since there's no built-in filtering of the output stream in twisted.web). It's worked fine in my tests to date, and in limited production use, but to be honest it hasn't seen large scale, long term use because my original use case only yielded ~15% compression (multimedia files) and using this negates the ability to know the full length in advance, thus clients can't give projected download times. (My files are large enough that compressing twice is high overhead, and it wouldn't play well with the wrapper approach anyway). In any event, hopefully it'll be a useful starting point for you. Here's the wrapper: - - - - - - - - - - - - - - - - - - - - - - - - - import struct import zlib class GzipRequest(object): """Wrapper for a request that applies a gzip content encoding""" def __init__(self, request, compressLevel=6): self.request = request self.request.setHeader('Content-Encoding', 'gzip') # Borrowed from twisted.web2 gzip filter self.compress = zlib.compressobj(compressLevel, zlib.DEFLATED, -zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0) def __getattr__(self, attr): if 'request' in self.__dict__: return getattr(self.request, attr) else: raise AttributeError, attr def __setattr__(self, attr, value): if 'request' in self.__dict__: return setattr(self.request, attr, value) else: self.__dict__[attr] = value def write(self, data): if not self.request.startedWriting: self.crc = zlib.crc32('') self.size = self.csize = 0 # XXX: Zap any length for now since we don't know final size if 'content-length' in self.request.headers: del self.request.headers['content-length'] # Borrow header information from twisted.web2 gzip filter self.request.write('\037\213\010\000' '\0\0\0\0' '\002\377') self.crc = zlib.crc32(data, self.crc) self.size += len(data) cdata = self.compress.compress(data) self.csize += len(cdata) if cdata: self.request.write(cdata) elif self.request.producer: # Simulate another pull even though it hasn't really made it # out to the consumer yet. self.request.producer.resumeProducing() def finish(self): remain = self.compress.flush() self.csize += len(remain) if remain: self.request.write(remain) self.request.write(struct.pack('<LL', self.crc & 0xFFFFFFFFL, self.size & 0xFFFFFFFFL)) self.request.finish() - - - - - - - - - - - - - - - - - - - - - - - - - and here's a sample of using it. This code is from one of my Resource objects - if I were to use it more generally in my site I'd extract that into a mix-in class of some sort - currently there was only one specific resource (the multimedia files) I wanted to support it for: def render_GET(self, request): # (... argument validation ...) accept_encoding = request.getHeader('accept-encoding') if accept_encoding: encodings = accept_encoding.split(',') for encoding in encodings: name = encoding.split(';')[0].strip() if name == 'gzip': request = GzipRequest(request) break # At this point, use 'request' as normal -- David _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] Should I use asynchronous programming in my own modules?
by Jürgen Strass 19 Oct '07

19 Oct '07

Hello, I'm rather new to twisted and asynchronous programming in general. Overall, I think I've understood the asynchronous programming model and its implications quite well. Nevertheless, there are some remaining questions. To give some example, I'd like to develop my own simplified document format in XML and a corresponding parser. The output of the parser (a specialized document object model) will be traversed and translated into HTML afterwards. This module could be useful outside any twisted application, of course. Instead of generating HTML one could develop a generator that produces LaTeX, for example. But it could also be used to render HTML pages in a twisted web application. The question is this: since parsing and generating large documents could block the reactor in a twisted app, should I use any of twisted's asynchronous programming features in this module (for better integration with twisted) or should I rather develop it in a traditional way and run it in a thread? The question came to my mind, because somewhere I read that long lasting operations in third party modules should be called in a thread. This is clear. I also read that if one has the opportunity to develop an application from scratch, one should rather go for using twisted's asynchronous programming features and divide long lasting operations into small chunks. In principal, this approach is clear to me, but does it also apply for modules which are entirely independent from twisted networking code? And if so, is there any way to decouple them from the twisted library for reuse in other applications? The last question is what criteria I could use to divide long lasting operations into chunks. In almost all books about asynchronous programming I only read that if they're too big, they could block the event loop. Of course, but how big is too big? And what's the measure for it? Milliseconds, number of operations, number of code lines - or what? Doesn't it depend entirely on the application at hand and how reactive it has to be? Moreover, depending on the hardware used, on a Pentium II less chunks can be processed at the same time than on a Athlon 64, for example. And couldn't chunks also be too small, spending more time than necessary in putting them into the reactor's queue, then maybe sorting them and then calling them? In case the overhead involved in scheduling some chunk is bigger than the processing time of the chunk itself, the chunks are too small, aren't they? Thanks in advance for any answers, Jürgen

6 10

[Twisted-Python] Swapping a set of deferreds to diskOrganization: One Piece
by Vincent Bernat 19 Oct '07

19 Oct '07

Hi ! I have a function which returns a deferred but will take many time to really process an item. This function will be feed with a lot of items. For example, the function could be called 1000 times per second and will take 30 minutes to really process each item. However, I don't know the delay. The function could also process the items in real time. When the function takes 30 minutes to really process the item, the reactor will keep a lot of deferreds and huge portions of memory. Therefore, I would like to store myself those items (in a database for example) and wait the function to be ready to process all the items (because the function knows when it will be ready). I could store the items directly in the database and asks the function an unique deferred whose callback will be handled when the function is ready to process my items. But since, the function could be a function that process items in real time (and more faster than a database), I would like to use this "fallback" method only if the reactor has already a lot of deferred for this function. Ideally, in this case, current deferred should be converted to database. In short, I would like to be able to store/flush deferreds in database when I have a lot of them and keep them in memory if I have only a few of them. For example, a naive implementation could be to implement subclass of Deferred and each Deferred instance can store itself to disk if a lot of instances of this subclass already exists. This is not exactly what I want since I need a mechanism that will also flush all instances to disk instead of keeping them in memory. So an instance should be able to replace a lot of instances and recreate them when the function becomes ready. Does such a thing already exists? If it helps, I can guarantee that function will process items in FIFO order. Thanks. -- I WILL NOT AIM FOR THE HEAD I WILL NOT AIM FOR THE HEAD I WILL NOT AIM FOR THE HEAD -+- Bart Simpson on chalkboard in episode 8F13

1 0

[Twisted-Python] Content Encoding : gzip ?
by anurag uniyal 18 Oct '07

18 Oct '07

Is it possible to send and recieve compressed content using twisted.web? I have set Accept-encoding to gzip but it doesn't make any difference. Will I have to cater for this myself? rgds Anurag __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

3 2

Re: [Twisted-Python] [newbie] server script with client functionality
by Evert Rol 16 Oct '07

16 Oct '07

Hi Johann, Thanks for your reply. >> I'm new to Twisted, so I'm not sure where to find information on >> this. > http://twistedmatrix.com/projects/core/documentation/howto/tutorial/ > client.html I had looked at the tutorial, but admittedly not gone all the way, as it didn't seem to address my problem. But it looks like this part of the tutorial actually does. Then again, by that time, I felt a bit daunted by the Twisted framework (Protocols, Services, Factories and what not, all hooked into each other). So I've switched back to the socket module. While less neat, it does have the additional advantage that when exporting my script to other people, I don't have to ask them to install Twisted as well. But perhaps, in time, I will give Twisted another try. I'll certainly play with a few times more, to see if I can get the hang of it. Anyway, thanks again, Evert ps: ah yes, I did mean reactor.run(). I had already gotten somewhat confused with the terminology. > Evert Rol wrote: >> Hi all, >> >> I'm trying to write a server script that can pass the received >> data on >> (acting as a client) to a next server. >> Say, server A receives data from client B, and then acts as client A >> (while keeping in contact with client B) for server C. >> I'm using it to pass data through an extra machine before reaching >> the >> actual machine, where the extra machine serves as a security measure >> (eg, if that gets hacked, that won't bring too may other things >> down). >> >> >> I've been looking at creating a client at the moment that the server >> receives data (in the dataReceived() method), but then I get two >> factory.run() methods > do you mean reactor.run()? Factories usually just implement > buildProtocol, and can be used to keep references to protocols and > other > stuff (see tutorial). >> , and would also need to find a way to stop the client; that doesn't >> seem to be correct. Or perhaps using threads, but then I don't know >> how to pass the data from the server to client. > you don't need threads. >> Best to me would to have both connections active, each on their own >> port, and when data is received, 'something' in the client part gets >> called that sends these data on. >> Alternatively, I could simply call an external program with the data >> as arguments from the server script, but I'd like a all-in-one >> solution if possible. Or perhaps I'm looking at this the wrong way? >> (Although I'd prefer not to try and turn 'server C' into 'client C'.) > what do you mean by turning 'server C' into 'client C'? >> >> Any suggestions or pointers? >> > How many connections from A to C do you need? as many as for B to A? > Since you're in control of both servers it might make sense to keep > some > connections open between A and C, and save some handshake-overhead. > > The finger example comes close to what you need, you just have to make > serverprotocolA and clientprotocolA aware of each other. > > Johann >> Thanks in advance, >> >> Evert >>

1 0

[Twisted-Python] using Twisted with postgresql
by Doris Lam 16 Oct '07

16 Oct '07

Hi, I'm new to Twisted and is currently looking for a way to asynchronously do queries to a postgres database. From what I've seen it looks like there's a project called pgasync (http://jamwt.com/pgasync/) that does this, or I can use Twisted's adbapi and some other synchronous module. I'm wondering if anyone has used pgasync and how it compares to using adbapi and threads? From the pgasync site it seems the last update was in 2005, so I'm not sure if anyone's still maintaining it. If using adbapi's better, can I get some suggestions as to which postgres module I should use? So far I've seen pyPgSQL and psychopg that're both DB API 2 compliant. Any help will be appreciated! Thanks, Doris

4 3

[Twisted-Python] [newbie] server script with client functionality
by Evert Rol 15 Oct '07

15 Oct '07

Hi all, I'm trying to write a server script that can pass the received data on (acting as a client) to a next server. Say, server A receives data from client B, and then acts as client A (while keeping in contact with client B) for server C. I'm using it to pass data through an extra machine before reaching the actual machine, where the extra machine serves as a security measure (eg, if that gets hacked, that won't bring too may other things down). I'm new to Twisted, so I'm not sure where to find information on this. I've been looking at creating a client at the moment that the server receives data (in the dataReceived() method), but then I get two factory.run() methods, and would also need to find a way to stop the client; that doesn't seem to be correct. Or perhaps using threads, but then I don't know how to pass the data from the server to client. Best to me would to have both connections active, each on their own port, and when data is received, 'something' in the client part gets called that sends these data on. Alternatively, I could simply call an external program with the data as arguments from the server script, but I'd like a all-in-one solution if possible. Or perhaps I'm looking at this the wrong way? (Although I'd prefer not to try and turn 'server C' into 'client C'.) Any suggestions or pointers? Thanks in advance, Evert

2 1

[Twisted-Python] trouble with missing exceptions
by markus espenhain 15 Oct '07

15 Oct '07

hi i've written a tcp client - using a derived NetstringReceiver, a ReconnectingClientFactory and just reactor.connectTCP + reactor.run. the problem is that after the connection is made - every exception that occurred seems to be lost and the execution at this point discontinues - also with a try/catch block around it class MyProtocol(basic.NetstringReceiver): ... def onData(self, data): print data try: print 'before' # printed raise Exception('err') except Exception, e: print e # nothing print 'after' # nothing after print 'before' nothing more in onData happens - anyway the app runs and on new data the onData is entered again i poke around 2h or so and i can't remember to had this problem before ... so anybody can give me a hint or maybe something that i miss? thanks markus using twisted 2.4.0

2 2

[Twisted-Python] 2006, 'MySQL server has gone away'
by Werner Thie 15 Oct '07

15 Oct '07

Hi all While using mySQL V 5.0.33 and MySQL-python-1.2.2 with twisted/adbapi with the following connection params DB_DRIVER = "MySQLdb" DB_ARGS = { 'host': 'xxx.xxx.xxx.xxx', 'db': 'db', 'user': 'dbuser', 'passwd': 'dbpasswd' } I noticed (2006, 'MySQL server has gone away') errors, which seem to be not recoverable from an adbapi standpoint. Further analysis and reading the mySQL manual showed, that the mySQL server in versions > 5 seems to close open connections after (default) 8 hours. The reconnect feature of the db is disabled by default and there is no parameter to preset it via MySQL-python. Further digging brought up this link http://sourceforge.net/tracker/index.php?func=detail&aid=1483074&group_id=2… which patches the (also 1.2.2) version such, that a reconnect keyword is introduced in the DB_ARGS. Questions: - are there any adverse effects in applying this patch and setting reconnect: 1 in DB_ARGS? - is there a better, safer way to avoid this nasty error? Thxs, Werner

3 3

[Twisted-Python] Http PUT
by Voznesensky Vladimir 15 Oct '07

15 Oct '07

Dear Twisted authors, I need to realise an HTTP server capable of serving PUT request. There seems to be no standard twisted implementation for it. Is it possible to realise and include it into the standard twisted contribution? I have no solid idea were to start with it. Thanks, VV

8 7

[Twisted-Python] returning strings with twisted.web.xmlrpc
by Gerardo 15 Oct '07

15 Oct '07

How can I return NON-ASCII strings via xmlrpc( p.e. latin1)? Is it possible to read them using python's xmlrpclib? Thanks.

1 0

[Twisted-Python] twisted.names.client.getHostByName Error when DNS Server return Auth NS Server
by wangmm 15 Oct '07

15 Oct '07

There is a bug in the extractRecord function of common.py. And I open a ticket in http://twistedmatrix.com/trac/ticket/2850 ---------------------------------------------------------------------------------------------------- Python Version:2.5 Twisted Version:2.5 Names Version:0.4 When we got a resolver and call getHostByname, perhaps DNS server can return CNAME, Auth NS Server Name and Auth NS Server IP Address. So extractRecord function will extract the Auth NS Server and send the DNS request to the Auth NS Server. But there is a bug in the extractRecord function. Code: # No answers, but maybe there's a hint at who we should be asking about this for r in answers: if r.type == dns.NS: from twisted.names import client r = client.Resolver(servers=[(str( r.payload.name), dns.PORT)]) return r.lookupAddress(str(name) Generally, the Auth NS Server is the Domain name, and it's IP contained in the addition fields. If we get the NS Server by r.payload.name, we will get the DNS Server Domain, so the Domain as server will pass to client.queryUDPas DNS Server Address. UDP.write will complaint that warnings.warn("Please only pass IPs to write(), not hostnames "+addr[0] <http://twistedmatrix.com/trac/changeset/0>, DeprecationWarning? <http://twistedmatrix.com/trac/wiki/DeprecationWarning>, stacklevel=2). In addition, self.socket.sendto(datagram, addr) will call socket.gethostbyname(addr), but this call is block. I think we should use the Auth NS Server IP to replace the Domain. The patch code as fellow: Any comment is welcome! # No answers, but maybe there's a hint at who we should be asking about this for r in answers: if r.type == dns.NS: from twisted.names import client server = str(r.payload.name) for s in answers: if s.type==dns.A and str(s.name)==server : server = socket.inet_ntop(socket.AF_INET, s.payload.address) break r = client.Resolver(servers=[(server, dns.PORT)])

1 0

[Twisted-Python] Regarding "add" method in microdom.lmx
by Raj kumar 15 Oct '07

15 Oct '07

Hi every one, This is raj.. I am new to twisted python. I have created a xml and want to add some content to it but i am unable to edit the xml file.. I tried in this way..... .............................................................. Method(self, X,Y,Z,xmlfile=None) xmlfile = xmlfile or self.xmlfile . . jump = microdom.lmx('jumping') jump.add("jumpat", xjump=X, yjump=Y) save_xml(jump.node,xmlfile) then the xml file is showing <jumping> </jumping> but not the jumpat tag and content ............................................................... if i try in this below manner ...................................... Method(self, X,Y,Z,xmlfile=None) xmlfile = xmlfile or self.xmlfile . . jump = microdom.lmx('jumping') jump.add("jumpat") save_xml(jump.node,xmlfile) then xml file is showing <jumping> <jumpat> </jumpat> </jumping> .................................. Can any one help me how to add those X,Y,Z values in that? I wanted the out put to be stored as below <jumping> <jumpat x=value,y=value,z=value> </jumpat> <jumpat x=value,y=value,z=value> </jumpat> </jumping> Thank you in advance. --------------------------------- Unlimited freedom, unlimited storage. Get it now

1 0

[Twisted-Python] UDP with multiple connections
by Simon Pickles 14 Oct '07

14 Oct '07

Ok, UDP is new to me, so please be gentle.... I've got a server authentication app which needs to accept many client connections, but also dispatch messages to a master server. Am I best using unconnected UDP? do i then have to deal with each received datagram by checking which host is has come from and acting accordingly? thanks Simon

7 13

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 14 Oct '07

14 Oct '07

Bug summary ______________________ Summary for 2007-10-07 through 2007-10-14 Bugs opened: 5 Bugs closed: 3 Total open bugs: 1040 (+2) |== Type Changes |== Priority Changes |== Component Changes |Defect: +0 |Highest: +0 |Core: +1 |Enhancement: +1 |High: -1 |Trial: +1 |Task: +1 |Normal: +2 |Low: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2849] A reentrant addCallbacks on a Deferred adds to mid-chain (opened by Peaker) defect core http://twistedmatrix.com/trac/ticket/2849 ===== Normal ===== [#2845] twisted.internet.thread._putResultInDeferred should be public (opened by therve) enhancement core http://twistedmatrix.com/trac/ticket/2845 [#2847] Get rid of all "API Stability: unstable" markers (opened by radix) task core http://twistedmatrix.com/trac/ticket/2847 [#2848] typos in doc/core/man/tap2{deb,rpm}.1 (opened by exarkun) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/2848 ===== Low ===== [#2846] trial coverage option should include tracing in threads (opened by therve) enhancement trial http://twistedmatrix.com/trac/ticket/2846 Closed Bugs ______________________ ===== Highest ===== [#2457] failed reactor selection is reported poorly by trial and twistd (opened by exarkun, closed by therve, fixed) enhancement core http://twistedmatrix.com/trac/ticket/2457 ===== High ===== [#411 ] Deferreds are good at blowing the stack (opened by etrepum, closed by Peaker, invalid) defect core http://twistedmatrix.com/trac/ticket/411 ===== Normal ===== [#2848] typos in doc/core/man/tap2{deb,rpm}.1 (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/2848 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2849] A reentrant addCallbacks on a Deferred adds to mid-chain (since 2007-10-10 11:25:51). Mean open ticket age: 539 days, 13:55:11.952859. Median: 440 days, 3:14:16.977859. Standard deviation: 403 days, 6:49:39.251191. Interquartile range: 504 days, 16:10:28. Mean time between ticket creation and ticket resolution: 122 days, 14:09:56.608352. Median: 14 days, 16:22:15. Standard deviation is 226 days, 22:06:10.127155. The interquartile range is 130 days, 2:23:34. Mean time spent in review: 67 days, 20:17:30.264549. Median: 4 days, 9:08:37. Standard deviation: 149 days, 7:51:46.848091. Interquartile range: 31 days, 6:31:05. Mean number of times a ticket is reviewed: 1.95150115473. Median: 1 Standard deviation: 1.58950187867. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 10 unique ticket reporters 7 unique ticket reviewers 2 unique ticket resolvers In the last 24 weeks, 70 unique ticket reporters 16 unique ticket reviewers 14 unique ticket resolvers In the last 48 weeks, 152 unique ticket reporters 18 unique ticket reviewers 26 unique ticket resolvers

1 0

[Twisted-Python] Running commands (ssh) from a GUI client
by Raúl Gómez C. 11 Oct '07

11 Oct '07

Hi list, this is my first post :p I'm new to Twisted/Conch and I was wondering how can I modify the sshsimpleclient.py in order to run several commands, on a user request mode, without the need to authenticate just before every command, just like a normal interactive ssh session. This is for a GUI front end I'm writing (PythonCard) that will execute some benchmarks on a remote server. Thanks in advance... Raul

6 13

[Twisted-Python] stdio.StandardIO, ServerProtocol and Services.
by kgi 10 Oct '07

10 Oct '07

Hi all. I'm trying to write a "terminal" component, wrapped in a service.Service, to drop into an application framework so that I can use it to build interactive command-line tools. I've almost, but not quite, got it working. I know about manhole, but I want to write a simple command-based syntax, rather than exposing a python namespace and interpreter. I could also just go with a telnet or ssh interface, but that would require an additional step (the actual telnet or ssh). There's also stdiodemo.py, but that doesn't offer readline-like functionality, which would be a Very Good Thing. (Besides, I'm really curious as to why this approach is not working!) The code below is a minimal stripped-down example of what I've got so far through frantic grepping and glimpseing through the Twisted codebase. (although there are some fragmentary holdovers from various experiments; ignore these). At the moment, the command handler (the lineReceived() method) just echoes the argument with some '+' signs prepended. The termios stuff comes from twisted/conch/stdio.py, via a mailing list post saying something to the effect of, "this is the minimal amount you have to do in order to hook up your terminal to your process's stdio"; I don't have the ML link handy right now, sorry. Note that whereas stdio.py passes a Protocol argument to ServerProtocol, I've created an intermediate class, CLIServerProtocol, which has a protocolFactory attribute. An examination of twisted/conch/insults/insults.py shows that this should be equivalent. For the curious bystanders, HistoricRecvLine is derived as follows: HistoricRecvLine -> RecvLine -> TerminalProtocol ... and TerminalProtocol implements things like connectionMade, etc. Here's the behaviour I see: With the code as is, I can type in lines and get a reply from lineReceived. However, none of the cursor keys or other things for which there is special code in twisted/conch/recvline.py work; I just get control codes all over the terminal. If I uncomment the "reactor.run()", however, the behaviour changes completely, and becomes much closer to what I expect: keypresses like Home, cursor keys, etc, are honoured, as well as the history. However, Ctrl-C merely loses the connection, and the terminal hangs and I have to kill the twistd process externally. The fact that adding reactor.run() (which shouldn't be needed in a Service, right?) implies to me that the reactor isn't starting up properly without it, and I can't see why. Any advice would be greatly appreciated. Ricky -- import os, tty, sys, termios from twisted.application import service from twisted.internet import reactor, stdio, protocol, defer from twisted.python import failure, reflect, log from twisted.protocols import basic from twisted.application import internet from twisted.conch.insults import insults from twisted.conch.manhole import ColoredManhole # from twisted.conch.stdio import ConsoleManhole from twisted.application import service from twisted.conch import recvline class CLIProtocol ( recvline.HistoricRecvLine ): service = None def connectionMade ( self ): recvline.HistoricRecvLine.connectionMade ( self ) self.keyHandlers [ '\x01' ] = self.handle_HOME self.keyHandlers [ '\x03' ] = self.handle_QUIT self.keyHandlers [ '\x1a' ] = self.handle_QUIT def connectionLost ( self, reason ): log.msg ( "Connection Lost" ) def handle_QUIT ( self ): self.terminal.loseConnection() def lineReceived ( self, line ): self.terminal.write ( '+++' + line ) class CLIServerProtocol ( insults.ServerProtocol ): protocolFactory = CLIProtocol class CLIService ( service.Service ): def startService ( self ): fd = sys.__stdin__.fileno() oldSettings = termios.tcgetattr ( fd ) tty.setraw ( fd ) try: p = CLIServerProtocol() stdio.StandardIO ( p ) # reactor.run() finally: termios.tcsetattr ( fd, termios.TCSANOW, oldSettings ) os.write ( fd, "\r\x1bc\r" ) return service.Service.startService ( self ) ###################################################################### # Create the application service hierarchy. ###################################################################### application = service.Application ( 'cliapp' ) cs = CLIService() cs.setServiceParent ( application )

5 10

Re: [Twisted-Python] How to switch users in SSH session.
by Ross.McKerchar＠sophos.com 09 Oct '07

09 Oct '07

Paul_S_Johnson at mnb.uscourts.gov Paul_S_Johnson at mnb.uscourts.gov Thu Sep 27 16:00:14 EDT 200: > After much wrangling and a small miracle I have managed to write an object > that fetches the output of three commands in the order given through an > SSH connection. How come I cannot switch users? Some of the information I > need can only be fetched through a root account and security is such that > I cannot log in remotely from a root account but must switch once logged > in from an account with less than root privileges. I've been trying something similar. As you've probably found out you cant just write in the password to su - you'll get an error "Standard in must be a tty". Basically I think su needs a shell to work. Consequently, before sending your password, you'll need to request a shell. I would've been utterly clueless about this if it wasn't for the post @ http://twistedmatrix.com/pipermail/twisted-python/2007-July/015793.html, from which I managed to extract the crucial code: term = 'ansi' winsz = struct.pack('4H', 80, 100, 80, 100) winSize = struct.unpack('4H', winsz) ptyReqData = session.packRequest_pty_req(term, winSize, '') self.conn.sendRequest(self, 'pty-req', ptyReqData) self.conn.sendRequest(self, 'shell', '') Sticking this in your SSHChannel.channelOpen method will allow you to get a shell. Unfortunately you then have to interact with the session in an expect-like manner which turns the whole procedure into one big easily broken hack. Basically I just write "su -" and wait for my method dataReceived to return "password:" at which point I write in the password. Another point to mention is that the ssh session is still not root, the root functionality is being provided by the shell instance. Consequently any new channels opened up will not be root so you cant easily use this method to scp file's that are only readable for root. You can of course cat from the shell session but then you have issues with line wrapping and reliably figuring out when the file ends and your shell prompt begins. I looked into trying to do some setreuid/setuid magic in the hope that I could issue a command from my root shell prompt that would elevate the whole ssh process to root allowing all my channels root priviliges. However I soon got well out of my depth and decided it wasn't possible, at least by me :) So, as far as I can see, it's not possible, if your server doesn't allow root access to open up a channel that has root priviliges. The only way to do it is in the aformentioned messy way via a shell & su (which I gave up on when my regular expression took up two lines on my screen and still didn't reliable match half the shell prompts it needed to). I would love somebody to prove me wrong and let me know of a nice way round this. Of course - these problems are nothing to do with twisted and all to do with the way ssh works. -ross -- Ross McKerchar Systems Analyst, Sophos Tel: 01235 559933 Web: http://www.sophos.com Sophos - security and control Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom. Company Reg No 2096520. VAT Reg No GB 348 3873 20.

1 0

Re: [Twisted-Python] Returning a Deferred as a result from another Deferred
by Jean-Paul Calderone 09 Oct '07

09 Oct '07

On Fri, 5 Oct 2007 19:00:51 -0700, Ryan Fugger <rfugger(a)gmail.com> wrote: >On 10/5/07, Jonathan Lange <jml(a)mumak.net> wrote: >> On 10/6/07, Ryan Fugger <rfugger(a)gmail.com> wrote: >> > Returning a Deferred as a result from another Deferred seems to be >> > disallowed by an assertion at the start of the callback chain. Why is >> > this? Right now I have worked around this limitation by wrapping my >> > Deferred in a list and then unwrapping it in the callback. >> > >> >> No it's not. I can guarantee this. >> >> What's the content of the AssertionError that you are getting? > >No content in the error. Line 238 of twisted/internet/defer.py (first >line in Deferred.callback) is: > >assert not isinstance(result, Deferred) > >That's pretty explicit in disallowing Deferreds... I'm working with >the version 2.5 release. You've misread the code. Jonathan is correct. The rest of this thread seems to go off on some tangent about threads which may or may not be relevant to whatever problem you're trying to solve. What led you to be reading this code in the first place? Glyph did make a good point, though. You shouldn't ever have a Deferred in a thread other than the reactor thread. You shouldn't make them, you shouldn't add callbacks to them, you shouldn't call them back. It might be legitimate to get one from the reactor thread, hold it for a while, then send it back to the reactor thread, but I can't think of a case where this would really be very useful. Jean-Paul > >Ryan > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >

2 2

[Twisted-Python] Marking as review
by Jonathan Lange 09 Oct '07

09 Oct '07

Hello guys, Not a big deal, but is it still a good idea to mark tickets as "HIGHEST" priority when we put them up for review? 'kenaan' seems to be doing a good enough job of reminding people about review tickets as is, and I could use the extra mouse-clicks (I'm saving!) jml

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 07 Oct '07

07 Oct '07

Bug summary ______________________ Summary for 2007-09-30 through 2007-10-07 Bugs opened: 5 Bugs closed: 2 Total open bugs: 1038 (+3) |== Type Changes |== Priority Changes |== Component Changes |Defect: +3 |Highest: -2 |Core: +0 |High: +1 |Release Management: +1 |Normal: +3 |Trial: -1 |Low: +1 |Web: +2 |Web2: +1 New / Reopened Bugs ______________________ ===== High ===== [#2843] twisted.web2.log.FileAccessLoggingObserver should take a file like object instead of a logPath. (opened by dreid) defect web2 http://twistedmatrix.com/trac/ticket/2843 ===== Normal ===== [#2841] [PATCH] Add a redirect limit to HTTPPageGetter in twisted/web/client.py (opened by alexis) defect web http://twistedmatrix.com/trac/ticket/2841 [#2842] [PATCH] Handle bad formatted headers in lineRecieved() in twisted/web/http.py (opened by alexis) defect web http://twistedmatrix.com/trac/ticket/2842 [#2844] Missing parens in multiexception (opened by fumanchu) defect core http://twistedmatrix.com/trac/ticket/2844 ===== Low ===== [#2840] fails to build on Mac OS X with QuickTime 6 headers (opened by zooko) defect release management http://twistedmatrix.com/trac/ticket/2840 Closed Bugs ______________________ ===== Highest ===== [#2837] TreeReporter setup/teardown error handling broke (opened by pjd, closed by therve, fixed) defect trial http://twistedmatrix.com/trac/ticket/2837 [#1192] twisted.test.test_policies calls reactor.iterate() (opened by exarkun, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/1192 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2844] Missing parens in multiexception (since 2007-10-05 12:22:50). Mean open ticket age: 535 days, 4:48:39.014318. Median: 436 days, 3:18:46.639559. Standard deviation: 403 days, 8:09:45.372209. Interquartile range: 504 days, 4:36:19. Mean time between ticket creation and ticket resolution: 121 days, 20:41:54.745053. Median: 14 days, 13:40:23. Standard deviation is 224 days, 23:12:37.447753. The interquartile range is 128 days, 12:08:19. Mean time spent in review: 66 days, 19:49:44.881401. Median: 4 days, 9:08:37. Standard deviation: 147 days, 6:13:52.498708. Interquartile range: 29 days, 10:44:36. Mean number of times a ticket is reviewed: 1.94431554524. Median: 1 Standard deviation: 1.58638608769. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 12 unique ticket reporters 7 unique ticket reviewers 2 unique ticket resolvers In the last 24 weeks, 73 unique ticket reporters 16 unique ticket reviewers 14 unique ticket resolvers In the last 48 weeks, 155 unique ticket reporters 18 unique ticket reviewers 26 unique ticket resolvers

1 0

[Twisted-Python] Returning a Deferred as a result from another Deferred
by Ryan Fugger 06 Oct '07

06 Oct '07

Returning a Deferred as a result from another Deferred seems to be disallowed by an assertion at the start of the callback chain. Why is this? Right now I have worked around this limitation by wrapping my Deferred in a list and then unwrapping it in the callback. Maybe someone can suggest a better approach. My application takes a command from one connection and defers processing to a worker thread to permit potentially-blocking database access. That worker thread often will need to send data over another connection to fulfill the command, and await a reply before the reply can be sent back over the original command connection. The worker thread sends data over the second connection using callFromThread, and this send operation itself returns a Deferred which fires when the reply is received. It is this Deferred that I want to return as a result from the original Deferred created by deferring to the worker thread in the command handler, so I can wait until the reply comes from the second connection, and then reply on the original command connection. Can someone suggest a better approach? Thanks. Ryan

4 7

[Twisted-Python] Collecting all results from a DeferredList
by Stefan Arentz 04 Oct '07

04 Oct '07

I want to request many web pages through a DeferredList and then combine the results in an array. I don't understand how to keep an array as shared state between these requests and return it to the original caller when the DeferredList has completed. S.

1 1

[Twisted-Python] LoopingCall vs internet.TimerService
by Saptarshi Guha 03 Oct '07

03 Oct '07

Hello, Thanks for twisted - very useful. Now, my questions I have written a server which accepts connections from clients, takes requests and adds them to a Queue (a python object of Queue.Queue). Now i have two approaches a) At startup, my server(a subclass of pb.Root) runs a "dispatcher" function using task.LoopingCall(dispatcher).start(0.1) b)Create a separate subclass of internet.TimerService whose function is dispatcher [INFO: dispatcher has methods like obj.callRemote, addCallback and addErrback] So which is better, neater, safer? loopingcall of timerservice? also is it safe to do the calls mentioned in INFO within dispatcher? Thank you for your time and help. Saptarshi Guha Saptarshi Guha sapsi(a)pobox.com

3 3

[Twisted-Python] Discovering Twisted : reactor.stop()
by Steve MULLER 03 Oct '07

03 Oct '07

Hello, I discovered Python recently and even more recently the Twisted module. I have a problem with the Twisted reactor and I would like to understand for which reason my program freezes. Here the code which should loop forever : from twisted.internet import reactor from twisted.internet import task class reactorProb(object): def __init__(self): self.count = 5 self._callback = task.LoopingCall(self.Heartbeat) self._callback.start(1, now = False) def Heartbeat(self): print "Top" self.count -= 1 if (self.count == 0): print "Exit" self._callback.stop() del self._callback reactor.stop() while (True): rp = reactorProb() print "Enter into reactor.run()" reactor.run() print "Leave from reactor.run()" I got this: Enter into reactor.run() Top Top Top Top Top Exit Leave from reactor.run() Enter into reactor.run() Top Top Top Top Top Exit Thus I start first once the reactor, I stop it then I start again and it freezes instead of stopping it the reactor. Could somebody explain me it why? Thank you in advance. smu

4 6

[Twisted-Python] Project design questions
by Paul_S_Johnson＠mnb.uscourts.gov 02 Oct '07

02 Oct '07

Here's a description of my general game plan and my questions are below: THE MISSION Create a GUI-based, dashboard-like application that monitors the overall health of our resources. It would be a tool that helps pinpoint trouble spots when performance is less than desirable. Ideally, we would get such a good handle on it that it would warn us of impending trouble before it causes a failure. SAMPLE FUNCTIONS FOR THE APPLICATION Monitor web traffic by distilling down Apache logs Monitor server hardware through built in firmware http interface Monitor server health via command line commands: disk space processor load etc. Monitor database health via command line commands Monitor backup success through log output Monitor misc. server messages Monitor switch traffic Monitor processes ...And more. POSSIBLE CONSTRAINTS Our main servers are on a WAN whose security is the job of the greater org. We have access through SSL connections now, but we may not be able to get permissions to open any additional ports for our monitoring purposes. THE QUESTIONS Very basic question: am I better off leaving most of the heavy lifting (as in the programming logic) on the client side? This would mean more data would have to pass from server to client app (e.g. a whole web log or part thereof) which is the downside. The upside is that in a failure emergency situation if my server is still up enough to have a SSH connection my dashboard app may still be useful to me. Thoughts? If we are not constrained to only get our info via SSH what's the best kind of connection to have? My thought is to use the Prospective Broker since that seems to be the most Pythonistic option. Is it possible to encrypt the PB server connection? Are there any Twisted methods of tunneling through an existing SSH connection (just a wild thought)? Thanks, Paul

3 2

[Twisted-Python] reactor.spawnProcess() - differences between Python & Bash deaths
by Brendan Boerner 01 Oct '07

01 Oct '07

Hi, I'm seeing (or not seeing :-) ) behavior which I cannot explain. Attached are two .py scripts and a bash script - echo.bash - echo a message echo.py - echo a message py_spawn.py - launch either echo.bash or echo.py using reactor.spawnProcess(). If I have py_spawn.py launch echo.bash and then kill the '/bin/bash ./echo.bash' process using a SIGTERM py_spawn reports: error: errorMessage: A process has ended with a probable error condition: process ended by signal 15. error: exitCode: None error: signal: 15 error: status: 15 processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessTerminated'>> Which is what I expect. If instead I launch echo.py and then kill the '/opt/local/bin/python2.5 ./echo.py -m message' process py_spawn reports: error: exitCode: 0 error: signal: None error: status: 0 processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessDone'>> which is not what I expect. In particular I am 99% certain that this is relatively recent behaviour which showed up only last week - prior to that I was seeing that if I killed a spawned .py process that I'd see that it was terminated with a signal 15 ala the bash case. I've tested this on Mac and Debian boxes with identical results. Any suggestions or insights are appreciated. Regards, Brendan --- py_spawn.py --- #! /opt/local/bin/python2.5 from twisted.internet import error, reactor, defer, protocol import getopt import pickle import pprint import subprocess import signal import sys Dumper = pprint.pformat class MyProcessProtocol(protocol.ProcessProtocol): def __init__(self): self.outdata = "" self.errdata = "" pass def connectionMade(self): print "connectionMade!" def outReceived(self, data): print "outReceived! with %d bytes!" % len(data) self.outdata = self.outdata + data def errReceived(self, data): print "errReceived! with %d bytes!" % len(data) self.errdata = self.errdata + data def inConnectionLost(self): print "inConnectionLost! stdin is closed! (we probably did it)" def outConnectionLost(self): print "outConnectionLost! The child closed their stdout!" print "I saw them write:\n", self.outdata def errConnectionLost(self): print "errConnectionLost! The child closed their stderr." print "I saw them write:\n", self.errdata def processEnded(self, status): print "debug: type(status): %s" % type(status.value) print "debug: isinstance(status.value, error.ProcessDone): % s" % \ isinstance(status.value, error.ProcessDone) print "debug: isinstance(status.value, error.ProcessTerminated): %s" % \ isinstance(status.value, error.ProcessTerminated) if isinstance(status.value, error.ProcessDone): print "error: exitCode: %s" % status.value.exitCode print "error: signal: %s" % status.value.signal print "error: status: %s" % status.value.status elif isinstance(status.value, error.ProcessTerminated): print "error: errorMessage: %s" % status.getErrorMessage() print "error: exitCode: %s" % status.value.exitCode print "error: signal: %s" % status.value.signal print "error: status: %s" % status.value.status print "processEnded, status %s" % repr(status) print "quitting" reactor.stop() def usage(): print "Usage: py_spawn.py" def main(): processProtocol = MyProcessProtocol() # killing the bash process using SIGTERM results in: # error: errorMessage: A process has ended with a probable error condition: process ended by signal 15. # error: exitCode: None # error: signal: 15 # error: status: 15 # processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessTerminated'>> reactor.spawnProcess(processProtocol, 'bash', ["bash", "-c", "echo.bash"], env=None) # killing the bash process using SIGTERM results in: # error: exitCode: 0 # error: signal: None # error: status: 0 # processEnded, status <twisted.python.failure.Failure <class 'twisted.internet.error.ProcessDone'>> #reactor.spawnProcess(processProtocol, 'bash', ["bash", "-c", "./ echo.py -m message"], env=None) reactor.run() if __name__ == "__main__": main() --- echo.py --- #!/opt/local/bin/python2.5 from optparse import OptionParser, OptParseError from twisted.internet import reactor from twisted.spread import pb from zope.interface import implements import pprint import sys Dumper = pprint.pformat def run(duration, message): duration -= 1 print "duration: %d, message: %s" % (duration, message) if duration != 0: reactor.callLater(1, run, duration, message) else: reactor.stop() def main(): print "debug: args: %s" % sys.argv[1:] try: duration = 0 message = "No message" parser = OptionParser() parser.add_option("-d", "--duration=", dest="duration", metavar="INT", default=duration, help="How long to run, 0 will run forever [default: % default]") parser.add_option("-m", "--message=", dest="message", default=message, help="Message to display [default: %default]") (opts, args) = parser.parse_args() duration = int(opts.duration) message = opts.message print "debug: opts: %s, args: %s" % (opts, args) except OptParseError, e: # print help information and exit: print "exception: %s" % e sys.exit(2) reactor.callLater(0, run, duration, message) reactor.run() if __name__ == "__main__": main() --- echo.bash --- #!/bin/bash while true; do echo -n "This is a message "; date; sleep 1; done

1 0

[Twisted-Python] Aborting callback chain with an errback
by Andy Fundinger 01 Oct '07

01 Oct '07

What is the proper way to handle an uncorrectable error? I have code that looks like: def callXMLRPC(self,stringValue,intValue): return(xmlProxy.callRemote("myFunc",{...}) \ .addCallback(self.rpcSuccess).addErrback(self.rpcFailure)) and in a derived class: def specificQuery(self): return self.callXMLRPC("Ask Thing", 3).addCallback(self.handleResult) what I'd like to have happen is if an error is raised in callRemote, rpcFailure does some cleanup and no further callbacks are executed. I can't fix the failure, but I don't want a message logged into my error log everytime, it's not unhandled, just that there's nothing to be done and callbacks won't have the data to run. If I want to deal with it further I'd want to do it in the base class. How can I do that? Am I doing something nonsensical here? Andy Fundinger -- Blog: http://channel3b.wordpress.com Second Life Name: Ciemaar Flintoff

2 1

Twisted October 2007