Mailman 3 October 2005 - Twisted-web

web2.client keep-alive problem
by Scott Lamb Oct. 21, 2005

Oct. 21, 2005

I've run into another problem in my application that uses the dreid/ web2-client branch. This one I'm not sure how to solve, or even kludge around for the moment. My application makes a bunch of HTTP requests in serial. It needs to see the previous response to make the next request. When I handle a HTTP response, I originally wrote it like this (simplified): def _makeRequest(self, ...): req = client.Request(...) req.addCallback(self._handleResponse) def _handleResponse(self, response): stream.readStream(response.stream, self._handleResponseBody) def _handleResponseBody(self, body): self._makeRequest(...) This worked fine - proper HTTP Keep-Alives and whatnot - until I got a large response back. Then I realized that stream.readStream gives me chunks and I should instead be using this (if I want to handle the whole response at once): def _handleResponse(self, response): self.body = '' stream.readStream(response.stream, self._handleResponseChunk ).addCallback(self._handleResponseBody) def _handleResponseChunk(self, chunk): self.body += chunk def _handleResponseBody(self): self._makeRequest(...) self.body = None This works with the chunking, but I am now making an HTTP connection for each request. HTTPClient._endResponse() calls self.response.stream.finish() and then tears down the connection if no request is pending. But stream.finish() doesn't call my deferred right away; it happens later from the reactor. So though I want to make another request, I haven't had my chance yet. My connection is torn down, then I make a new one and do an expensive SSL handshake, which is all wasted round trips, bandwidth, and CPU time. I can think of two ways to solve this: (1) make stream.ProducerStream.finish() call _handleResponseBody right away. But there's a "reactor.callLater(0, self._read)" in stream._StreamReader that looks to be there for a reason. I haven't thought it through, but I suspect there are unsolvable recursion problems with this approach. (2) make HTTPClient not check if the connection should be torn down until after my deferred has had a chance to run. Would just a "reactor.callLater(0, self._endResponse2)" be enough? Are callLaters guaranteed to be run in order? -- Scott Lamb <http://www.slamb.org/>

1 1

Status of Livepage ?
by Thomas HERVE Oct. 21, 2005

Oct. 21, 2005

I was wondering what is the current status of Livepage in Nevow : for us Livepage users, is the move to 0.5 worthy ? I don't see big improvements, maybe on client side (server.handle is cool) and cleaner API, but I do see things broken. With 0.4.1 I'm stuck with Twisted 1.3, so it's problematic (Twisted.Web and Twisted.Mail updates for examples). Will Athena (official name ?) become the standard Livepage in 0.6, so it could be wise to wait for it ? To be more constructive, I've identify 2 problems in the flatten process in Livepage: 1) The __call__ function in livepage makes a non-exhaustive test (basestring, stan.Tag, types.FunctionType, types.MethodType, types.UnboundMethodType), and it's a problem for examples with lists : client.set("mynode", [tags.span["tes't"], tags.span["tes't"]]) 2) The _jsSingleQuoteQuote function in flat.flatstan seems to operate _before_ the flattening of stan content, so doesn't quote content when it added in flattener: class Dummy(object): pass def flatter(o, res): return "Hi! I've got a quote!" flat.registerFlattener(flatter, Dummy) client.set("mynode", tags.span[Dummy()]) Thanks for your answers, -- Thomas

1 0

Preventing XSS when using Nevow's vhost functionality
by David Remahl Oct. 20, 2005

Oct. 20, 2005

Say I implement web server that serves content for a number of different users, each on their own sub domain. My nevow site has a root resource that is sensitive to the host / port information in the request (virtual hosting). Since this happens is in a reverse-proxy situation, the root resource has a child, vhost, that allows the application to gain knowledge of which host the user accessed. One of the users, "evil", has a goal of stealing cookie content from people who visit goody's web shop. The attacker makes the victim go to this URL: http://goody.com/vhost/http/evil.net/pageWithAttackJS.html which gets rewritten (by the reverse proxy) to: http://internalserver:1234/vhost/http/goody.com/vhost/http/ evil.net/pageWithAttackJS.html The web server looks up the vhost child of the root resource which consumes the next two components. It sets the host to goody.com, as it should be and passes control back to the root resource. The remaining URI at this point is: http://goody.com/vhost/http/evil.net/pageWithAttackJS.html This once more invokes the vhost functionality, the following results and is accessed: http://evil.net/pageWithAttackJS.html The malicious javascript code is sent to the client which interprets it. The problem is that the client still believes it accessed a resource on goody.com, and therefore allows the JS to access cookies set by goody.com. The same design flaw can be used for phising attacks etc (faking the "sender" of some information). Can anyone suggest a good approach for preventing this kind of violation? I think limiting vhost from working unless the current request's host is (internalserver, 1234) would be safest. Other ideas? / Thanks, David Remahl

3 7

Probleming Building Nevow
by Misto . Oct. 20, 2005

Oct. 20, 2005

Hi folks! Today I did these steps: nevow>svn co http://divmod.org/svn/Divmod/trunk/Nevow/ ... nevow>python setup.py build c:\Documents and Settings\Giovanni\workspace\Nevow>python setup.py Traceback (most recent call last): File "setup.py", line 8, in ? import setupcommon File "c:\Documents and Settings\Giovanni\workspace\Nevow\setupcommon.py", line 1, in ? import nevow File "c:\Documents and Settings\Giovanni\workspace\Nevow\nevow\__init__.py", line 12, in ? from nevow import compy File "c:\Documents and Settings\Giovanni\workspace\Nevow\nevow\compy.py", line 16, in ? from compyCompat import * AttributeError: 'module' object has no attribute 'IComponentized' I'am running on windows XP What appens? Misto

2 1

twisted.web getPage-- retrieving error information
by David E. Konerding Oct. 19, 2005

Oct. 19, 2005

Hi folks, I am tryign to get useful error information when I use twisted.web.getPage. For example, this code doesn't supply a trailing slash when making a request for www.google.com. This causes getPage to make an invalid request to the web server ("GET HTTP/1.1") which causes the server to disconnect immediately (that's true for the google web server. for apache, it produces an "Invalid Request" message and error code). It doens't seem like the error object passed to the errback contains any useful information which I can bubble up to the user. For example, I'd like to know what data was received before the server closed the connection. ANy ideas? from twisted.internet import reactor from twisted.web.client import getPage def gotPage(result): print "gotPage: result=", result def gotPageErr(error): print "gotPageErr: error=", error def getAPage(url): gp = getPage(url) gp.addCallback(gotPage) gp.addErrback(gotPageErr) reactor.callWhenRunning(getAPage, "http://google.com") reactor.run()

1 0

nevow livepage - where are my parameters (request.args) gone?
by Peter Arwanitis Oct. 19, 2005

Oct. 19, 2005

Hi there, first: Thanks for nevow 0.5! JIT :) for evaluation and integration in an ongoing project of mine. Searching for AJAX features with python-support the "chatola" example is an amazing eye-opener!!! The bi-directional support is stunning! And the code is compact... Hacking through the roughs, I have * migrated my existing twisted.web.server to nevow.appserver stuff: DONE (cannot have them both, cause of interface-adaption-hell) * sucessful broadcasting from wxPython/twisted.reactor-application to connected browsers, via send() * ... Problems I cannot hack: * using nevow:render="store" and appropriate render_store() method. I'm returning plain string WITH rendered html (from existing code) --> HOW can I avoid the escaping of all '<' '&' and '>'??? No, it's not sure that ii is xhtml. * Problem using href-links form inside my lifepage with additional parameters http://localhost:8080/lifesite?when=20051010 (not AJAX -> plain request) Where does my parameters gone??? only things like ('livepage_client', '0', 'output') occuring? thanks for hints Peter (=PA=)

1 1

sessions how to
by Michele Simionato Oct. 19, 2005

Oct. 19, 2005

Can anybody point out to me some examples of session management with twisted web? For instance, how do I make a login page? Users and passwords should be stored into a PostgreSQL DB. TIA, Michele Simionato

1 0

my first post, writing a Web interface to a PostgreSQL db
by Michele Simionato Oct. 17, 2005

Oct. 17, 2005

I need to write a Web interface to a PostgreSQL database and I am considering to use Twisted for that. So, I have a few questions. First of all, I would like to know if there already something available that I should look at. Second, I have a more specific question. Suppose I have a large table with 1,000,000 records and a SELECT query that returns 1000 results and takes 10 minutes. I don't want my users to wait. I want to display the result of the query as an HTML table of 50 rows. The user should click, wait 30 seconds, see the result, then click a 'next' button to see the other 50 results, etc. In other words, I am looking for a lazy query functionality. Is that already available? If not, is it easy/difficult to implement? Thanks for any suggestion, Michele Simionato

3 7

looking for a request.getfirst
by Michele Simionato Oct. 17, 2005

Oct. 17, 2005

I am surprise I don't find a method to extract the value of a field from a request object. At the moment I am using a helper function def getfirst(name, request): return request.args.get(name, [None])[0] but is there a better solution? Michele Simionato

2 1

bind_*
by Garanin Michael Oct. 16, 2005

Oct. 16, 2005

Hello! Sorry i am nevow-newcomer. I use nevow 0.5.0 and my page has method: def bind_doSomething(self, ctx): return [('param',annotate.Integer()) ] It's work and i see labels "Do Something" and "Call" in my browser. My question: how i can change labels? Thanks!

2 1