Twisted-web December 2008

twisted-web@python.org

18 participants
14 discussions

Start a nNew thread

Test
by Philippe Lafoucrière 05 Aug '21

05 Aug '21

Hi all just a test

3 2

database best practices
by Martin Bright 21 Dec '08

21 Dec '08

Hi! What is the best way to implement synchronous DB access in a twisted application? I've read about defer.deferredGenerator, and various threads about how it's a stupid idea to even attempt this. Do I have restructure my code such that everything that depends on results from a database query is in a callback to a deferred object? -martin

10 9

SSH Server with commands filtering
by Tzury Bar Yochay 18 Dec '08

18 Dec '08

Hi, I am willing to deploy on my server an SSH daemon which I can control what commands can be executed, and by who. After playing with jailkit [link] I realised that implementing my own SSH server will give me more power and scalability (especially regarding of users, keys and accounts management). Naturally I approached twisted knowing it will solve this problem as a smart kiddo solves a simple safari puzzle. Playing with samples found on the web and at Twisted docs, I realised that twisted has everything I need for the task, but I am missing the knowledge of how to glue it all together. In fact, a command if not working on the first strike, works on the second try. at http://gist.github.com/37446 I pasted the server.py code. which is a modification of what I've found at: http://twistedmatrix.com/projects/conch/documentation/examples/sshsimpleser… for instance trying to run git over ssh yeilds error at first, and when running again seems to be working. below is the console output on the client side #first attempt tzury@regulus:~$ git clone ssh://tzury@localhost:5022/home/tzury/devel/ice/snoip/freespeech clone-repo7 Initialized empty Git repository in /home/tzury/clone-repo7/.git/ remote: Counting objects: 347, done. remote: Compressing objects: 100% (232/232), done. Receiving objects: 100% (347/347), 91.75 KiB, done. remote: Total 347 (delta 234), reused 177 (delta 114) fatal: The remote end hung up unexpectedly Resolving deltas: 100% (234/234), done. error: waitpid (async) failed fatal: error in sideband demultiplexer #second attempt tzury@regulus:~$ git clone ssh://tzury@localhost:5022/home/tzury/devel/ice/snoip/freespeech clone-repo7 Initialized empty Git repository in /home/tzury/clone-repo7/.git/ remote: Counting objects: 347, done. remote: Compressing objects: 100% (232/232), done. Receiving objects: 100% (347/347), 91.75 KiB, done. remote: Total 347 (delta 234), reused 177 (delta 114) Resolving deltas: 100% (234/234), done. tzury@regulus:~$

1 1

Athena & browser 2-connection limit
by Harald Blåtand 17 Dec '08

17 Dec '08

Hi there, Been thinking about this. It _would_ be cool if you could have more than one LivePage per browser session, like in popups, or even frames. What do you think: Is there any obvious & fundamental reason why the ReliableMessageDelivery stuff couldn't be modified, so that one instance of it could be shared between several PageWidgets? Harald

6 19

Finding memory leaks in Nevow/Twisted
by Abdul-Wahid Paterson 17 Dec '08

17 Dec '08

Hi I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies. This only happens for the pages with Athena elements which suggests that something isn't been cleaned up when AJAX methods are being called. Can anyone suggest a possible cause? Or what I really need is a suggestion in how to track down the memory leak. Is there a way of doing memory profiling to find the culprit? Any advice is welcomed. AW

4 6

How can I change the HTTP request to avoid gzip
by Radu Dragusin 16 Dec '08

16 Dec '08

I have a HTTP Proxy made with twisted.web and want to change the request that the browser sends to the Proxy such that I erase the value of the 'accept-encoding' key from 'gzip,deflate' to ' '. I use the example from the Tisted Book: By adding the overriden process method in WordCountProxyRequest I can get the request header but have found no way to set a key, value pair. I want make the server think that the browser does not support gzip because twisted seems to not support gzip as the response from www.google.com and many (but not all) sites appears still encoded. www.dpreview.com seems not to gzip the response, and so the resonse is processed correctly. What can I do to either correctly decode gzip responses or modify the 'accept-encoding' value to nothing so the server does not compress the response? Thank you! *Example 4-8. wordcountproxy.py* import sgmllib, re from twisted.web import proxy, http import sys from twisted.python import log log.startLogging(sys.stdout) WEB_PORT = 8000 PROXY_PORT = 8001 class WordParser(sgmllib.SGMLParser): def __init__(self): sgmllib.SGMLParser.__init__(self) self.chardata = [] self.inBody = False def start_body(self, attrs): self.inBody = True def end_body(self): self.inBody = False def handle_data(self, data): if self.inBody: self.chardata.append(data) def getWords(self): # extract words wordFinder = re.compile(r'\w*') words = wordFinder.findall("".join(self.chardata)) words = filter(lambda word: word.strip( ), words) print "WORDS ARE", words return words class WordCounter(object): ignoredWords = "the a of in from to this that and or but is was be can could i you they we at".split( ) def __init__(self): self.words = {} def addWords(self, words): for word in words: word = word.lower( ) if not word in self.ignoredWords: currentCount = self.words.get(word, 0) self.words[word] = currentCount + 1 class WordCountProxyClient(proxy.ProxyClient): def handleHeader(self, key, value): proxy.ProxyClient.handleHeader(self, key, value) if key.lower( ) == "content-type": if value.split(';')[0] == 'text/html': self.parser = WordParser( ) def handleResponsePart(self, data): proxy.ProxyClient.handleResponsePart(self, data) if hasattr(self, 'parser'): self.parser.feed(data) def handleResponseEnd(self): proxy.ProxyClient.handleResponseEnd(self) if hasattr(self, 'parser'): self.parser.close( ) self.father.wordCounter.addWords(self.parser.getWords( )) del(self.parser) class WordCountProxyClientFactory(proxy.ProxyClientFactory): def buildProtocol(self, addr): client = proxy.ProxyClientFactory.buildProtocol(self, addr) # upgrade proxy.proxyClient object to WordCountProxyClient client.__class__ = WordCountProxyClient return client class WordCountProxyRequest(proxy.ProxyRequest): protocols = {'http': WordCountProxyClientFactory} def __init__(self, wordCounter, *args): self.wordCounter = wordCounter proxy.ProxyRequest.__init__(self, *args) * def process(self): proxy.ProxyRequest.process(self) print "received_headers", proxy.ProxyRequest.getAllHeaders(self)* class WordCountProxy(proxy.Proxy): def __init__(self, wordCounter): self.wordCounter = wordCounter proxy.Proxy.__init__(self) def requestFactory(self, *args): return WordCountProxyRequest(self.wordCounter, *args) class WordCountProxyFactory(http.HTTPFactory): def __init__(self, wordCounter): self.wordCounter = wordCounter http.HTTPFactory.__init__(self) def buildProtocol(self, addr): protocol = WordCountProxy(self.wordCounter) return protocol # classes for web reporting interface class WebReportRequest(http.Request): def __init__(self, wordCounter, *args): self.wordCounter = wordCounter http.Request.__init__(self, *args) def process(self): self.setHeader("Content-Type", "text/html") words = self.wordCounter.words.items( ) words.sort(lambda (w1, c1), (w2, c2): cmp(c2, c1)) for word, count in words: self.write("<li>%s %s</li>" % (word, count)) self.finish( ) class WebReportChannel(http.HTTPChannel): def __init__(self, wordCounter): self.wordCounter = wordCounter http.HTTPChannel.__init__(self) def requestFactory(self, *args): return WebReportRequest(self.wordCounter, *args) class WebReportFactory(http.HTTPFactory): def __init__(self, wordCounter): self.wordCounter = wordCounter http.HTTPFactory.__init__(self) def buildProtocol(self, addr): return WebReportChannel(self.wordCounter) if __name__ == "__main__": from twisted.internet import reactor counter = WordCounter( ) prox = WordCountProxyFactory(counter) reactor.listenTCP(PROXY_PORT, prox) reactor.listenTCP(WEB_PORT, WebReportFactory(counter)) reactor.run( ) -- Radu

2 4

ASN1_TIME_to_generalizedtime error on windows box
by Sean Buchanan 16 Dec '08

16 Dec '08

installed: python 2.4.5 twisted 8.1.0 nevow 0.9.32 error: when i installed nevow and everytime i run a .tac i get.. python.exe - Entry Point Not Found The procedure entry point ASN1_TIME_to_generalizedtime could not be located in libeay32.dll version of libeay32.dll in c:\windows\system32\ 777kb 09 July 2008 8:05a i must be missing something.. where do i look.. i have googled and scanned archives for past 2 years.. thx for help.. - cheers =)

2 1

possible bug in t.web.wsgi
by Phil Christensen 12 Dec '08

12 Dec '08

Hi All, I haven't opened a ticket for this because I'm not sure if it's a bug or not. I found this in my WSGI code that I had copied from web2, and it looks like the current twisted.web.wsgi inherited the same usage. Basically, the issue is with SERVER_PORT; in t.w.wsgi, around line 171, the environ dictionary is created like this: self.environ = { 'REQUEST_METHOD': request.method, 'SCRIPT_NAME': '/' + '/'.join(request.prepath), 'PATH_INFO': pathInfo, 'QUERY_STRING': queryString, 'CONTENT_TYPE': request.getHeader('content-type') or '', 'CONTENT_LENGTH': request.getHeader('content-length') or '', 'SERVER_NAME': request.getRequestHostname(), 'SERVER_PORT': str(request.getHost().port), 'SERVER_PROTOCOL': request.clientproto} This means SERVER_PORT is always set to the 'real' port that the transport is using, not the port that was provided in the request. The port used in the request ends up appended to the end of SERVER_NAME. Several sources I've referenced indicate something like the following: SERVER_PORT Typically, Web servers listen to HTTP requests on port 80. However, a Web server can listen to any port that's not in use by another application. A CGI program can find out the port the Web server is serving HTTP requests by looking at the value of the SERVER_PORT environment variable. When displaying self-referencing hypertext links at runtime by examining the contents of SERVER_NAME, be sure to append the port number of the Web server (typically port 80) by concatenating it with the value of SERVER_PORT. Which leads me to believe that SERVER_NAME should not have the port attached, and that SERVER_PORT should be the *requested* port, not the 'real' port. The end result of this is that WSGI apps break if run under a port- forwarding scenario, where the external port is different from the internal one, and where the code is using SERVER_PORT to construct URLs. Let me know if this is the commonly accepted understanding of these variables, and I'll file a bug report. -phil

1 0

finger tutorial: problems rewriting finger08.py to use inlineCallbacks
by Paul Goins 11 Dec '08

11 Dec '08

I'm having problems with using inlineCallbacks in the finger tutorial. I've used them successfully in some other test programs, but in this case I really don't know what I'm doing wrong. Here's the FingerProtocol code from the finger tutorial: class FingerProtocol(basic.LineReceiver): def lineReceived(self, user): self.factory.getUser(user) \ .addErrback(lambda _: "Internal error in server") \ .addCallback(lambda m: (self.transport.write(m + "\r\n"), self.transport.loseConnection())) And here's my modified version: class FingerProtocol(basic.LineReceiver): @defer.inlineCallbacks def lineReceived(self, user): try: data = yield self.factory.getUser(user) except Exception: data = "Internal error in server" self.transport.write(data + "\r\n") self.transport.loseConnection() For some reason, in my version, the connection is getting terminated before any writes can take place. Even if I comment out the loseConnection line, still nothing goes across to the client and the connection still gets terminated. Could anyone tell me where I'm going wrong on this? Thanks in advance. - Paul Goins

2 1

twisted.web.wsgi usage example
by Waldemar Osuch 09 Dec '08

09 Dec '08

Hi, I'm trying to come up with an example on how to use the spanking new twisted.web.wsgi module. Here it is. Any comments? 8<-------------------------------------------------------------------------------------------- from twisted.internet import reactor from twisted.application import service, internet from twisted.web import server, static, wsgi from twisted.python import threadpool def wsgi_app(environ, start_response): status = '200 OK' response_headers = [('Content-type', 'text/plain')] start_response(status, response_headers) return ['Hello from twisted.web.wsgi'] class WSGIService(service.Service): def __init__(self, minPoolSize=5, maxPoolSize=20): self.pool = threadpool.ThreadPool(minPoolSize, maxPoolSize) def startService(self): self.pool.start() service.Service.startService(self) def stopService(self): self.pool.stop() service.Service.stopService(self) application = service.Application('wsgi') root = static.File("www/documents") site = server.Site(root) internet.TCPServer(8080, site).setServiceParent(application) serv = WSGIService() root.putChild('wsgi', wsgi.WSGIResource(reactor, serv.pool, wsgi_app)) serv.setServiceParent(application)

3 2

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

Twisted-web December 2008