Mailman 3 December 2005 - Twisted-web

Test
by Philippe Lafoucrière 06 Aug '21

06 Aug '21

Hi all just a test

3 2

web2 response.setCookie and request.getCookie shortcuts?
by Clark C. Evans 21 Jan '06

21 Jan '06

I'm still having fun with web2, it is such a huge improvment over twisted.web and I'm absolutely thrilled with it. However, I have a few suggestions: 1. It would be supreme if render() and locateChild() just provided me the Request object; the incantation to ctx.locate() seems a tedious an unnecessary step. 2. I would love a request.getCookie('cookie-name') and response.setCookie('cookie-name','cookie-value', path = '/') helper; that adds the given cookie to the current list. 3. Make the 'path' in the Cookie constructor and setCookie default to '/'. Almost all of the time you want your cookies to be site-wide; adding path = '/' to each request adds a point of failure that makes the system less usable. Kind Regards, Clark P.S. Following is sample cookie code that works with web2 --- from twisted.internet import reactor, defer from twisted.web2 import server, http, resource, channel, iweb from twisted.web2.http_headers import Cookie class Toplevel(resource.Resource): addSlash = True def render(self, ctx): request = ctx.locate(iweb.IRequest) found = False for cookie in request.headers.getHeader('cookie',[]): if cookie.name == 'testing': print "found", cookie.value found = True response = http.Response(stream="Hello monkey!") if not found: print "not-found" cookie = Cookie("testing","womble",path="/") response.headers.setHeader('Set-Cookie',(cookie,)) return response # Standard twisted application Boilerplate from twisted.application import service, strports application = service.Application("demoserver") site = server.Site(Toplevel()) s = strports.service('tcp:8080', channel.HTTPFactory(site)) s.setServiceParent(application)

6 12

Nevow PageCache
by Andrea Arcangeli 07 Jan '06

07 Jan '06

I wanted to push into the tracker but I'm a bit confused on how to do that with trac, so I resend with the mailing list. This patch implements html caching at the page level with a optional max cache size per page and with optional timeout. Even when the timeout is very small, this makes sure that all blocked waiting clients gets the same copy of the html allowing the server to scale. This only works for dynamic data that is the same for all users, so it's good for the homepage etc... I use it online for months and I had no problems at all. Index: Nevow/nevow/util.py =================================================================== --- Nevow/nevow/util.py (revision 3434) +++ Nevow/nevow/util.py (working copy) @@ -133,6 +133,7 @@ from twisted.python.failure import Failure from twisted.trial.unittest import deferredError from twisted.python import log + from twisted.internet import reactor try: # work with twisted before retrial Index: Nevow/nevow/rend.py =================================================================== --- Nevow/nevow/rend.py (revision 3434) +++ Nevow/nevow/rend.py (working copy) @@ -491,6 +491,56 @@ self.children[name] = child +class PageCache(object): + def __init__(self): + self.__db = {} + def cacheIDX(self, ctx): + return str(url.URL.fromContext(ctx)) + def __storeCache(self, cacheIDX, c): + self.__db[cacheIDX] = c + def __deleteCache(self, cacheIDX): + del self.__db[cacheIDX] + def __deleteCacheData(self, cacheIDX, page): + size = self.__db[cacheIDX][1] + assert len(self.__db[cacheIDX][0]) == size + page.subCacheSize(size) + self.__deleteCache(cacheIDX) + def __lookupCache(self, cacheIDX): + return self.__db.get(cacheIDX) + def getCache(self, ctx): + cacheIDX = self.cacheIDX(ctx) + c = self.__lookupCache(cacheIDX) + + if c is None: + self.__storeCache(cacheIDX, [util.Deferred()]) + return + + if isinstance(c[0], util.Deferred): + d = util.Deferred() + c.append(d) + return d + + return c[0] + def cacheRendered(self, ctx, data, page): + cacheIDX = self.cacheIDX(ctx) + defer_list = self.__lookupCache(cacheIDX) + assert(isinstance(defer_list[0], util.Deferred)) + size = len(data) + if page.canCache(size): + # overwrite the deferred with the data + timer = None + if page.lifetime > 0: + timer = util.reactor.callLater(page.lifetime, + self.__deleteCacheData, cacheIDX, page) + page.addCacheSize(size) + self.__storeCache(cacheIDX, (data, size, timer, )) + else: + self.__deleteCache(cacheIDX) + for d in defer_list: + d.callback(data) + +_CACHE = PageCache() + class Page(Fragment, ConfigurableFactory, ChildLookupMixin): """A page is the main Nevow resource and renders a document loaded via the document factory (docFactory). @@ -504,8 +554,27 @@ afterRender = None addSlash = None + cache = False + lifetime = 0 + max_cache_size = None + __cache_size = 0 + flattenFactory = lambda self, *args: flat.flattenFactory(*args) + def hasCache(self, ctx): + if not self.cache: + return + return _CACHE.getCache(ctx) + def addCacheSize(self, size): + assert self.canCache(size) + self.__cache_size += size + def subCacheSize(self, size): + self.__cache_size -= size + assert self.__cache_size >= 0 + def canCache(self, size): + return self.max_cache_size is None or \ + self.__cache_size + size <= self.max_cache_size + def renderHTTP(self, ctx): if self.beforeRender is not None: return util.maybeDeferred(self.beforeRender,ctx).addCallback( @@ -530,11 +599,20 @@ if self.afterRender is not None: return util.maybeDeferred(self.afterRender,ctx) - if self.buffered: + c = self.hasCache(ctx) + if c is not None: + assert self.afterRender is None + finishRequest() + return c + + if self.buffered or self.cache: io = StringIO() writer = io.write def finisher(result): - request.write(io.getvalue()) + c = io.getvalue() + if self.cache: + _CACHE.cacheRendered(ctx, c, self) + request.write(c) return util.maybeDeferred(finishRequest).addCallback(lambda r: result) else: writer = request.write

2 3

Re: [Twisted-Python] WSGI Thread-management strategy
by Jim Fulton 31 Dec '05

31 Dec '05

James Y Knight wrote: > (BTW, the correct mailing list for twisted webbish stuff is twisted- > web(a)twistedmatrix.com) > On Dec 15, 2005, at 4:22 PM, Jim Fulton wrote: > >> The strategy used by twisted WSGI, as I understand it, doesn't meet >> our needs. Currently, a thread is created for each request. The total >> number of threads is throttled, I gather using a general Twisted >> thread limit. WSGI applications are called as soon as input headers >> have been received completely. An application may be called before all >> body input is received. We need application calls to be delayed until >> all request input has been received, >> >> [...] >> >> I propose that the default thread-management strategy should be to delay >> calling an application until all request input has been received. If >> this isn't the default, then there should at least be an option to get >> this behavior. (Of course, the buffering strategy needs to be clever >> enough to switch to a file when the input gets over some size.) > > > Sounds sensible, and is doable external to the WSGI wrapper. Here's a > little bit I whipped up. (works on the 2.1.x branch and head). Could be > smarter, by starting out the buffer in memory and switching to a file > if necessary. Also shows off a couple of minor bugs I need to fix. :) > > def simple_wsgi_app(environ, start_response): > print "Starting wsgi app" > start_response("200 OK", [('Content-type','text/html; > charset=ISO-8859-1')]) > data = environ['wsgi.input'].read() > return ['<pre>', data, '</pre>'] > > > class Prebuffer(resource.WrapperResource): > def hook(self, ctx): > req = iweb.IRequest(ctx) > temp = tempfile.TemporaryFile() > def done(_): > temp.seek(0) > # Replace the request's stream object with the tempfile > req.stream = stream.FileStream(temp) > # Hm, this shouldn't be required: > req.stream.doStartReading = None > return stream.readStream(req.stream, temp.write).addCallback (done) > > # Oops, fix missing () in lambda in WrapperResource > def locateChild(self, ctx, segments): > x = self.hook(ctx) > if x is not None: > return x.addCallback(lambda data: (self.res, segments)) > return self.res, segments > > if __name__ == '__builtin__': > from twisted.application import service, strports > from twisted.web2 import server, channel > > res = Prebuffer(wsgi.WSGIResource(simple_wsgi_app)) > > site = server.Site(res) > application = service.Application("demo") > > s = strports.service('tcp:8080', channel.HTTPFactory(site)) > s.setServiceParent(application) > This doesn't work for me if the input gets over a 8K, It turns out the MMap wrapper is broken. When the resulting stream is read, the first line of output seems to be a mmap repr: <mmap.mmap object at 0xb68ef6c0> and subsequent calls to readline return '\n'. Disabling MMap support makes this work. Debugging this was a bit frightening. To read a line from a temporary file, we end up calling the reactor. Is all this indirection and machinery really necessary to read a temporary file? That question is mostly rhetorical. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

1 0

XMLRPC and sessions
by Rob McMillen 28 Dec '05

28 Dec '05

I am trying to build an XMLRPC server that utilizes sessions to keep track of authenticated users. However, I have not been able to find too much on session management and the only way I have figured out how to get at the site from within the XMLRPC server is by overriding the render method. Is this the proper way? Are there other ways to do it? I am currently thinking about adding a mechanism to store the uid and time in a db. Any help would be great! Rob class Handler(xmlrpc.XMLRPC): def __init__(self, cmd_schema, cmd_xml): log.msg("Using %s and %s to configure api" % (cmd_schema, cmd_xml)) self.api = API(cmd_schema, cmd_xml) log.msg("Preparing authentication mechanism") self.creds = Creds() log.msg("Preparing role mechanism") self.roles = Roles() def render(self, request): s = request.site.makeSession() print s.uid return xmlrpc.XMLRPC.render(self, request) def xmlrpc_login(self, user, password): return self.creds.authenticate(user, password).addErrback( self.handleError) if __name__ == "__main__": import sys from twisted.internet import reactor from twisted.python import log log.startLogging(sys.stdout) site = server.Site(Handler('conf/api.xsd', 'conf/api.xml')) reactor.listenSSL(8000, site, ServerContextFactory('etc/host.pem', 'etc/host.pem')) reactor.run()

1 0

Using Twisted Web for browser-based control
by Drake Smith 21 Dec '05

21 Dec '05

4 6

(no subject)
by Wilfredo Sánchez Vega 21 Dec '05

21 Dec '05

I think I'm ready to do a merge of the DAV branch. exarkun, I think I've addressed all on the list you sent me. Anyone else want to review the code first? Other objections? I'll give you some targets to criticize: - The DAV property store code depends on Bob Ippolito's xattr library, which works on Mac OS, some Linuxes, and possible FreeBSD. Non-Mac uses require a filesystem that supports it. Alternate property stores are not going to be had to plug in, and we have several ways to proceed there, but I think these are generally out of scope for work on this branch, which I'm trying to keep from growing into a large and scary thing. - I use datetime in XML processing, which is apparently a Python 2.3+ feature. It would be unfortunate to have to avoid datetime, though, since it would mean adding the relevant code to twisted somewhere. The datetime issue we can either wave off by saying you need 2.3 to use dav, or we go ahead and copy in a bunch of code. I'd prefer to do that after merging if we think it's necessary to do so. I vote for requiring 2.3, but maybe that's my inner laziness talking. - Some python files have capital letters. exarkun says that may bug some people, specifically Windows types. Is it actually a problem? I can fix the capital letters if it's an issue. -wsv

5 6

xpost: multiple job openings
by Paul G 20 Dec '05

20 Dec '05

folks, i'm crossposting this to twisted-python-l and twisted-web-l for the benefit of those who don't read both. apologies to those who will consequently have to read this twice. --- We've got multiple job openings for great engineers familiar with python and twisted, senior and junior positions available. We're building a small team of folks who like working hard, solving interesting problems, writing clean code and having fun doing it. If you got into coding because that's where the money was in the '90s, please skip this one and see my next job posting where I'm recruiting bodyguards for work in the Persian Gulf. The dev team is run by an engineer ('run' in the sense that I shield you from the crazy business folk), so it is a strict meritocracy. You will be working on rebuilding/refactoring an existing PHP (ugh) web application into python services built on top of twisted as well as implementing new features. This is a high volume system, with millions of transactions a day - if you dig concurrency, you won't be bored. Must be able to handle the excitement ;) Requirements: * We're looking for great engineers - while your ability to discuss Wittgenstein over our 5th beer is welcome, a piece of paper^W^W^Wdiploma is not required. * You should be old enough not to get us in trouble with Amnesty International for child exploitation. * Experience working on hobby, open source and commercial projects counts with me. * Python/Twisted and SQL a must. * Reading/grokking bad PHP code a big plus. Writing PHP code useful, since we'll be hacking in the twisted/python bits over time. * Javascript/DHTML/AJAX(ugh, buzzowrd) familiarity needed in at least one hire, so a plus. * Experience with C and ASM good in that it is a good predictor of your understanding of memory allocation and performance issues. * Don't mention Java, unless it has caused a medical (eg mental health) problem we should know about. <grin> * Understanding of performance and scalability principles and issues. Work environment: * Onsite in NYC/NY Metro area strongly preferred. * Don't let the above deter you - we can make telecommuting work under the right conditions. * Competitive compensation - I know what good people are worth and have managed to beat that into the management with a clue-by-four. * Flex time - relaxed cluture - getting your job done is all that matters. --- Shoot me a resume, CV, writeup of what you've done, absolutely anything. Put [twisted-resume] in the subject. Include your phone number, time zone and a good time to call. I'll respond to everyone. cheers, -p

1 0

s: * select-with-other quick patch for renderImmutable
by Paul Reznicek 19 Dec '05

19 Dec '05

Hi Matt, Attached a quick (an dirty) solution for renderImmutable in class SelectOtherChoice. For me, it's better solution than raising an error ... The SelectOtherChoice - widget is nice and very helpful, Thanks ! Paul Index: widget.py =================================================================== --- widget.py (Revision 164) +++ widget.py (Arbeitskopie) @@ -367,7 +367,9 @@ return tag def renderImmutable(self, ctx, key, args, errors): - raise NotImplemented +## raise NotImplemented + return SelectChoice(self.original, zip(self.options, self.options) + ).renderImmutable( ctx, key, args, errors) def processInput(self, ctx, key, args): value = self._valueFromRequestArgs(key, args)

1 0

Nevow VHostMonsterResource and lighttpd
by Richard Wall 19 Dec '05

19 Dec '05

I got Nevow examples.tac and a simple Mantissa app working behind a lighttpd reverse proxy, by adding a VHostMonsterResource to the root resource in each case. However, the lighttpd proxy request headers seem to contain enough information about the public request (scheme, hostname, port), that I wonder if VHostMonsterResource is always necessary. For example given lighttpd listening on localhost.localdomain:10080 and netcat listening at localhost.localdomain:8080, the headers look like this... richard@lazar:~$ nc -v -l -p 8080 listening on [any] 8080 ... connect to [127.0.0.1] from localhost.localdomain [127.0.0.1] 34734 GET / HTTP/1.0 Host: localhost.localdomain:10080 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.12) Gecko/20051010 Firefox/1.0.7 (Ubuntu package 1.0.7) Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: UTF-8,* Keep-Alive: 300 X-Forwarded-For: 127.0.0.1 X-Host: localhost.localdomain:10080 X-Forwarded-Proto: http Infact, in my case anyway, the only problem running behind a proxy is that twisted.web refers to it's listening port instead of reading it from the http headers. Looking at twisted.web.server the following change fixes the port problem for me... Index: /home/richard/lib/Twisted/trunk/twisted/web/server.py =================================================================== --- /home/richard/lib/Twisted/trunk/twisted/web/server.py (revision 15339) +++ /home/richard/lib/Twisted/trunk/twisted/web/server.py (working copy) @@ -342,11 +342,17 @@ return self.session def _prePathURL(self, prepath): - port = self.getHost().port if self.isSecure(): default = 443 else: default = 80 + + hostheader = self.getHeader('host') + if hostheader: + port = int(dict(zip(('hostname', 'port'), hostheader.split(":"))).get('port', default)) + else: + port = self.getHost().port + if port == default: hostport = '' else: ...which just checks if there is a host header and gets the port number from there if it's available. I can't think that this would break anything and I hope someone will consider applying it. The second problem, I suppose, is with https. I haven't been able test it yet because I haven't got ssl working with lighttpd, but i should think the solution would be to check for the "X-Forwarded-Proto" header and if it exists, replace the request.isSecure method as in VHostMonsterResource. Once I am able to try this, I'll submit a patch. I'm not sure though whether this header is standard in reverse proxies or whether it's specific to lighttpd. Will try and check with Apache. The only remaining need for VHostMonsterResource then, is when you want to serve a Nevow app at a none root url. I wondered if it might be possible to add another header (eg X-Root-Path) using mod_headers and use that somewhere in the nevow.url class. Again, if I get chance to test this, I'll submit a patch. -- Richard Wall PS. According to Jan Knesche, Lighttpd mod_proxy will shortly have the ability to forward requests to a Unix socket which I think will provide quite a neat way of doing name based virtual hosting of multiple twisted.web sites.

1 1

Twisted-web December 2005