Mailman 3 February 2009 - Twisted

[Twisted-Python] 123912652 C-A-N-A-D-l-A-N P-H-A-R-M-A-C-Y
by exarkun 20 Feb '09

20 Feb '09

About this mailing: You are receiving this e-mail because you subscribed to MSN Featured Offers. Microsoft respects your privacy. If you do not wish to receive this MSN Featured Offers e-mail, please click the "Unsubscribe" link below. This will not unsubscribe you from e-mail communications from third-party advertisers that may appear in MSN Feature Offers. This shall not constitute an offer by MSN. MSN shall not be responsible or liable for the advertisers' content nor any of the goods or service advertised. Prices and item availability subject to change without notice. ©2009 Microsoft | Unsubscribe at http://wonnucul.cn | More news at http://wonnucul.cn | Prvacy at http://wonnucul.cn Microsoft Corporation, One Microsoft Way, Redmond, WA 98052

1 0

[Twisted-Python] Twisted slow in Mac OS X fast on Windows
by Brian Granger 20 Feb '09

20 Feb '09

Hi, We use Twisted extensively in the IPython project for parallel and distributed computing. We have an extensive test suite that uses trial to test our servers and clients. Everything works great...but...the test suite takes much longer to run on OS X than windows. Here is what I mean: 1. 2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5. Test suite takes 67 seconds 2. *Same machine*, running Windows XP as VM in VMWare. Test suite takes 17 seconds. I have verified that all the tests are being run on Windows. This is very surprising to me as I would expect, if anything, the Windows VM to be a bit slower, not 3x faster. This is the same hardware, the same code, the only thing that I can think of that is different is the reactor implementation. I am using the default reactor on both platforms. Questions: * Is the Windows reactor really that fast? * Is the default (I think it is select) reactor on OS X really that slow? * Why else would it be that slow on OS X? * What can I do to speed this up? Cheers, Brian

2 2

[Twisted-Python] How to use ampoule?
by Chris 20 Feb '09

20 Feb '09

Hi everyone, I am using twisted to build a server,and the computing request maybe costs lot of cpu resources.I have checked the maillist,it seems I can use ampoule plugin to create another process ,I have checked the website,and downloaded the source code.It seems there is no any document for it.I did find a example dir,after checking the code,I was confused.There should be two files,one for client and one for server,am I right?the client will send the request to the server(could be in another machine?),and the server responses.but I can't find anything which matches what I thought.Can anyone explain a little bit to me?Or if there is some code that would be better. And the twisted server will receive lot of binary data from the client,if I use ampoule,I have to send the same data to the ampoule server again(now the twisted server acts as the ampoule client).Is the ampoule suitable for such a kind of task? Or I should just use twisted process module? I don't know my understanding is correct or not,please correct me. Best Wishes Chris

2 1

[Twisted-Python] mind, twisted.cred and HTTPAuthSessionWrapper
by Esteve Fernandez 19 Feb '09

19 Feb '09

Hi all, I'm using HTTPAuthSessionWrapper to wrap a portal, which uses our own checker against an internal service to authenticate users and I'm having trouble to pass extra information to sub resources. I've read about the mind object and I think that's where all that information should go. However, HTTPAuthSessionWrapper passes None as its mind to portal.login, so I either have to override _login() or write a method that generates "minds" which can be overriden in subclasses to return something else than None. I prefer the latter, though. Anyway, I could be mistaken and "mind" wouldn't be appropriate in this case, so I wanted to ask first here, since the mind argument seems to be rarely used. Cheers.

2 9

[Twisted-Python] Correct pattern to do SSH forwarding in a GUI
by Luper Rouch 19 Feb '09

19 Feb '09

I am using twisted in a GUI application to do port forwarding over an SSH connection. All the network connections of the application (database, etc...) are routed through this connection, to simplify server configuration and secure things. My first attempt was to run the reactor in the main thread, but the application was "freezing" when e.g. doing a SELECT with SQLAlchemy/psycopg2 (I guess because psycopg2 just waits for data and blocks the twisted reactor). So I put reactor.run() in a thread and it works, but I'm getting strange issues on some platforms (for example I can't connect at all on *some* OSX computers, getting a "reactor stopping" log from twisted without any further error), and I feel I'm doing the whole thing terribly wrong. So my question is, how to do proper SSH port forwarding with conch in an application that also uses the tunnels not the asynchronous way ? If running the reactor in a thread is OK, are there particular things to watch out for ?

3 3

[Twisted-Python] raw xml to element, char encoding/decoding error
by Gabriel Rossetti 18 Feb '09

18 Feb '09

Hello, I wrote some code to transform a raw XML string into a domish.Element, and I keep on getting char encoding/decoding errors : class __RawXmlToElement(object): def __call__(self, s): self.result = None def onStart(el): self.result = el def onEnd(): pass def onElement(el): self.result.addChild(el) parser = domish.elementStream() parser.DocumentStartEvent = onStart parser.ElementEvent = onElement parser.DocumentEndEvent = onEnd tmp = domish.Element(("", "s")) tmp.addRawXml(s) parser.parse(tmp.toXml()) return self.result.firstChildElement() rawXmlToElement = __RawXmlToElement() Here's a test raw XML string : >>> u"<t>reçu</t>" u'<t>re\xe7u</t>' >>> u"<t>reçu</t>".encode("utf-8") '<t>re\xc3\xa7u</t>' >>> "<t>reçu</t>" '<t>re\xc3\xa7u</t>' As you can see my system encodes strings in UTF-8, I tried the following but I keep on getting errors : >>> rawXmlToElement("<t>reçu</t>") raw xml adder error : 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128) >>> rawXmlToElement(u"<t>reçu</t>") parser error : 'ascii' codec can't encode character u'\xe7' in position 8: ordinal not in range(128) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 26, in __call__ AttributeError: 'NoneType' object has no attribute 'firstChildElement' >>> rawXmlToElement(unicode("<t>reçu</t>", "utf-8")) parser error : 'ascii' codec can't encode character u'\xe7' in position 8: ordinal not in range(128) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 26, in __call__ AttributeError: 'NoneType' object has no attribute 'firstChildElement' If I try it with ASCII encodable chars it works correctly : >>> rawXmlToElement("<t>toto</t>").toXml() u'<t>toto</t>' >>> rawXmlToElement(u"<t>toto</t>").toXml() u'<t>toto</t>' >>> rawXmlToElement(unicode("<t>toto</t>", " utf-8")).toXml() u'<t>toto</t>' Does anyone have an idea on what I'm doing wrong here? Thank you!

3 7

[Twisted-Python] KeyError when an errback is triggered within a conch TCP tunnel
by Luper Rouch 18 Feb '09

18 Feb '09

Sorry I have not been able to find a better title, please see the attached example which may be more descriptive. Here is what it does: * set up an SSH connection * open a TCP forwarding channel through this connection * do something through the tunnel that will trigger an errback (in the example, try to do an XMLRPC call with no one listening on the tunnel's end) This scenario gives me the following traceback (tested with Twisted 8.1.0 and 8.2.0): -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/selectreactor.py", line 156, in _doReadOrWrite self._disconnectSelectable(selectable, why, method=="doRead") File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/posixbase.py", line 193, in _disconnectSelectable selectable.connectionLost(f) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/internet/tcp.py", line 520, in connectionLost protocol.connectionLost(reason) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/forwarding.py", line 129, in connectionLost self.channel.loseConnection() File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/channel.py", line 253, in loseConnection self.conn.sendClose(self) File "/usr/lib/python2.5/site-packages/Twisted-8.2.0-py2.5-linux-i686.egg/twisted/conch/ssh/connection.py", line 491, in sendClose self.channelsToRemoteChannel[channel])) exceptions.KeyError: <twisted.conch.ssh.forwarding.SSHListenClientForwardingChannel instance at 0x972048c> -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- It works well in the other cases, i.e.: * the errback is triggered but the tunnel is not used (comment line 57, uncomment line 59) * the errback is not triggered (start the XMLRPC server by uncommenting line 100), using the tunnel or not You get the error only when the tunnel is used AND the errback is triggered. To run the example execute it with an user name that can login using ssh on your host, e.g. 'python test.py someuser' and enter its password. Cheers, Luper

1 0

[Twisted-Python] twisted.web benchmarks
by Alvin Delagon 17 Feb '09

17 Feb '09

Hello All, First of all thanks for all the pointers in the #twisted about twisted.web being able to use threadpool. Here's the benchmark between cherryPy 3.0 and twisted.web that I promised. I hope this could serve as a reference in the future. *Testing parameters:* - Office PC: Intel(R) Pentium(R) D CPU 2.80GHz with 1GB RAM - Apache bench set to 500 concurrent attempts within 30 secs - LinuxMint Elyssa/Python 2.5.2/Twisted 2.5 - Twisted using SelectReactor - Webservers just returning "Hello!" after some mathematical calculation: result = 0 for i in range(random.randint(100, 300)): angle = math.radians(random.randint(0, 45)) result += math.tanh(angle)/math.cosh(angle) * CherryPy Results:* Concurrency Level: 500 Time taken for tests: 30.1441 seconds Complete requests: *6421* Failed requests: 0 Write errors: 0 Total transferred: 834990 bytes HTML transferred: 38538 bytes Requests per second: 214.02 [#/sec] (mean) Time per request: 2336.197 [ms] (mean) Time per request: 4.672 [ms] (mean, across all concurrent requests) Transfer rate: 27.17 [Kbytes/sec] received * twisted-web Results (no threading):* Concurrency Level: 500 Time taken for tests: 30.35509 seconds Complete requests: *10051* Failed requests: 0 Write errors: 0 Total transferred: 502600 bytes HTML transferred: 60312 bytes Requests per second: 334.64 [#/sec] (mean) Time per request: 1494.155 [ms] (mean) Time per request: 2.988 [ms] (mean, across all concurrent requests) Transfer rate: 16.31 [Kbytes/sec] received *twisted-web Results (with threading, threadpool [min=100, max=400]):* Concurrency Level: 500 Time taken for tests: 30.16998 seconds Complete requests: *11068* Failed requests: 0 Write errors: 0 Total transferred: 553450 bytes HTML transferred: 66414 bytes Requests per second: 368.72 [#/sec] (mean) Time per request: 1356.026 [ms] (mean) Time per request: 2.712 [ms] (mean, across all concurrent requests) Transfer rate: 17.99 [Kbytes/sec] received *Source Listing (using Twisted): * #!/usr/bin/python from twisted.internet import reactor from twisted.web import http from twisted.python.threadpool import ThreadPool import random, math, sys class HTTPProtocol(http.Request): def process(self): def process_finish(): result = 0 for i in range(random.randint(100, 300)): angle = math.radians(random.randint(0, 45)) result += math.tanh(angle)/math.cosh(angle) self.setHeader('Content-Type', 'text/html') self.write("Hello!") self.finish() if self.channel.factory.app.threads_enabled: print("Using threads") reactor.callFromThread(process_finish) else: print("Not using threads") process_finish() class HTTPChannel(http.HTTPChannel): requestFactory = HTTPProtocol class HTTPFactory(http.HTTPFactory): protocol = HTTPChannel def __init__(self, app): self.app = app def start(self, port=8080): reactor.listenTCP(port, self) class BenchmarkApplication(): def __init__(self): self.httpfactory = None self.pool = None self.threads_enabled = False def start(self): try: minthreads = int(sys.argv[1]) maxthreads = int(sys.argv[2]) self.pool = ThreadPool(minthreads, maxthreads) self.pool.start() self.threads_enabled = True print "Threads enabled: MIN:%d MAX:%d" % (minthreads, maxthreads) except Exception, e: print("minthreads and maxthreads is not set or malformed. Threading will be disabled") self.httpfactory = HTTPFactory(self) self.httpfactory.start() return (True) def end(self): if self.pool != None: self.pool.stop() if __name__ == "__main__": gba = BenchmarkApplication() if gba.start(): reactor.run() gba.end()* Source Listing (using CherryPy): *#!/usr/bin/python import cherrypy from amazilia import * from amazilia import base from amazilia import mysql import random, math class Root: def index(self): result = 0 for i in range(random.randint(100, 300)): angle = math.radians(random.randint(0, 45)) result += math.tanh(angle)/math.cosh(angle) return "Hello!" index.exposed = True root = Root() cherrypy.quickstart(root)* *I haven't had much time to tinker with CherryPy. Threre might be some ways to increase its performance. :) --- Alvin -- http://www.alvinatorsplayground.blogspot.com/

1 0

[Twisted-Python] Test, please ignore
by Itamar Shtull-Trauring 17 Feb '09

17 Feb '09

1 0

Re: [Twisted-Python] Communication with Async Java NIO, Socketchannel
by Johann Höchtl 15 Feb '09

15 Feb '09

On Sun, 15 Feb 2009 17:20:53 +0100, Johann HÃ¶chtl <hoechtl(a)in.tum.de> wrote: > >Is it possible to communicate with a nonblocking Java SocketChannel from > >Java.nio? > >What has to be attended? > >Are you certain you have the correct mailing list? This doesn't sound like >a question which is related to Twisted. > >Jean-Paul Dear Jean Paul, Yes I'm quite sure about this mailinglist. As I read in a Python book, twisted is the framework, which I should use for asynchronous network communication. Jo

2 1

Twisted February 2009