Mailman 3 January 2006 - Twisted-web

[PATCH] nonbuffered cache
by Andrea Arcangeli 12 Jan '06

12 Jan '06

Hello, After klive pages become larger and the db become larger too, I realized that enabling my PageCache methods was destroing the very nice and useful nonbuffered mode that allows showing the fragment of pages that are already rendered while nevow keeps working to finishing rendering the whole page. The nonbuffered mode makes interactive behaviour completely different for klive (while it's almost invisible for other projects that renders much more quickly because of much smaller pages). Disabling the cache is a no way, a simple ab2 would DoS the server without cache, it wouldn't even survive ./ . So I modified the my cache patches to support nonbuffered mode. That was very easy. I cleaned up the code a bit too. I added some db queries to take several seconds to complete to block the page rendering a few times, and it was real fun to open 3 browsers and see the bar on the right enlarging every few seconds on all three browsers at the same time after each query returned. Clicking reload also re-display only the part of the page already rendered and then it waits nevow to complete. Performance is excellent as usual (it greatly exceeds the performance of the network, it'd require gigabit ethernet to the internet to saturate the link): andrea@opteron:~> ab2 -n2000 -c 200 http://localhost:8818/ This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 200 requests Completed 400 requests Completed 600 requests Completed 800 requests Completed 1000 requests Completed 1200 requests Completed 1400 requests Completed 1600 requests Completed 1800 requests Finished 2000 requests Server Software: TwistedWeb/SVN-Trunk Server Hostname: localhost Server Port: 8818 Document Path: / Document Length: 166396 bytes Concurrency Level: 200 Time taken for tests: 7.210944 seconds Complete requests: 2000 Failed requests: 0 Write errors: 0 Total transferred: 335883918 bytes HTML transferred: 335625492 bytes Requests per second: 277.36 [#/sec] (mean) Time per request: 721.094 [ms] (mean) Time per request: 3.605 [ms] (mean, across all concurrent requests) Transfer rate: 45487.94 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 104 547.7 0 3003 Processing: 204 563 470.2 390 1978 Waiting: 110 334 68.9 329 544 Total: 204 667 716.4 390 3625 Percentage of the requests served within a certain time (ms) 50% 390 66% 449 75% 498 80% 530 90% 1959 95% 1977 98% 3528 99% 3543 100% 3625 (longest request) Even while the benchmark runs the server is accessible immediately. This delivered 45Mbytes/sec of payload (not megabit), with 277 requests per second. Without it such an ab2 command DoS the server. You can see the effect online as usual on klive website. I recommend applying to nevow CVS so others can use it too. If you don't change your code to enable it with the following API this patch *cannot* break anything even if it would be completely broken, so there are no excuses for not applying it ;), this is in the _obviously_ safe category. Here the API to enable it in your code. NOTE: you cannot use nevow_carryover/IHand with this, it makes no sense to cache forms output anyway. Recommended use is with cached_page_class and a LIFETIME of 5 seconds (the above benchmark has a lifetime of 5 sec). class forever_cached_page_class(rend.Page): cache = True class cached_page_class(forever_cached_page_class): lifetime = LIFETIME class cached_page_class(rend.Page): cache = True lifetime = LIFETIME max_cache_size = 20*1024*1024 (LIFETIME in seconds) I consider this a must-have for most nevow sites, thanks! Index: Nevow/nevow/util.py =================================================================== --- Nevow/nevow/util.py (revision 4051) +++ Nevow/nevow/util.py (working copy) @@ -132,6 +132,7 @@ from twisted.python import failure from twisted.python.failure import Failure from twisted.python import log + from twisted.internet import reactor except ImportError: class Deferred(object): pass Index: Nevow/nevow/rend.py =================================================================== --- Nevow/nevow/rend.py (revision 4051) +++ Nevow/nevow/rend.py (working copy) @@ -491,6 +491,62 @@ self.children[name] = child +class PageCache(object): + def __init__(self): + self.__db = {} + def cacheIDX(self, ctx): + return str(url.URL.fromContext(ctx)) + def __storeCache(self, cacheIDX, c): + self.__db[cacheIDX] = c + def __deleteCache(self, cacheIDX): + del self.__db[cacheIDX] + def __deleteCacheData(self, cacheIDX, page): + size = self.__db[cacheIDX][1] + assert len(self.__db[cacheIDX][0]) == size + page.subCacheSize(size) + self.__deleteCache(cacheIDX) + def __lookupCache(self, cacheIDX): + return self.__db.get(cacheIDX) + def getCache(self, ctx, request): + cacheIDX = self.cacheIDX(ctx) + c = self.__lookupCache(cacheIDX) + + if c is None: + c = ['', (util.Deferred(), request)] + self.__storeCache(cacheIDX, c) + def writer(buf): + c[0] += buf + for d, r in c[1:]: + r.write(buf) + return None, writer + elif isinstance(c, list): + d = util.Deferred() + request.write(c[0]) + c.append((d, request)) + return d, None + + return c[0], None + def cacheRendered(self, ctx, result, page): + cacheIDX = self.cacheIDX(ctx) + defer_list = self.__lookupCache(cacheIDX) + assert isinstance(defer_list[1][0], util.Deferred) + data = defer_list[0] + size = len(data) + if page.canCache(size): + # overwrite the deferred with the data + timer = None + if page.lifetime > 0: + timer = util.reactor.callLater(page.lifetime, + self.__deleteCacheData, cacheIDX, page) + page.addCacheSize(size) + self.__storeCache(cacheIDX, (data, size, timer, )) + else: + self.__deleteCache(cacheIDX) + for d,r in defer_list[1:]: + d.callback(result) + +_CACHE = PageCache() + class Page(Fragment, ConfigurableFactory, ChildLookupMixin): """A page is the main Nevow resource and renders a document loaded via the document factory (docFactory). @@ -504,8 +560,23 @@ afterRender = None addSlash = None + cache = False + lifetime = 0 + max_cache_size = None + __cache_size = 0 + flattenFactory = lambda self, *args: flat.flattenFactory(*args) + def addCacheSize(self, size): + assert self.canCache(size) + self.__cache_size += size + def subCacheSize(self, size): + self.__cache_size -= size + assert self.__cache_size >= 0 + def canCache(self, size): + return self.max_cache_size is None or \ + self.__cache_size + size <= self.max_cache_size + def renderHTTP(self, ctx): if self.beforeRender is not None: return util.maybeDeferred(self.beforeRender,ctx).addCallback( @@ -530,7 +601,18 @@ if self.afterRender is not None: return util.maybeDeferred(self.afterRender,ctx) - if self.buffered: + if self.cache: + cache, writer = _CACHE.getCache(ctx, request) + if cache: + return cache + + assert not self.buffered + assert self.afterRender is None + + def finisher(result): + _CACHE.cacheRendered(ctx, result, self) + return result + elif self.buffered: io = StringIO() writer = io.write def finisher(result):

3 5

simple registry failure - __adapt__
by charlie detar 07 Jan '06

07 Jan '06

Hi, I noticed an email from Dickon Reed referencing this issue last June (http://twistedmatrix.com/pipermail/twisted-web/2005-June/001536.html), but there is no reply in the archives - I was wondering if someone had discovered a solution? I am attempting to write a simple rpi script which uses a registry, as described in the documentation: http://twistedmatrix.com/projects/web/documentation/howto/using-twistedweb.… My rpy script is as follows: ---- from twisted.web import resource class Counter: def __init__(self): self.value = 0 def increment(self): self.value += 1 def getValue(self): return self.value counter = registry.getComponent(Counter) if not counter: registry.setComponent(Counter, Counter()) counter = registry.getComponent(Counter) class MyResource(resource.Resource): def render_GET(self, request): counter.increment() return "you are visitor %d" % counter.getValue() resource = MyResource() ---- I get the following error: exceptions.AttributeError: class Counter has no attribute '__adapt__' /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/site-packages/twisted/web/server.py, line 152 in process 150 self.postpath = map(unquote, string.split(self.path[1:], '/')) 151 try: 152 resrc = self.site.getResourceFor(self) 153 self.render(resrc) ... I would be happy to send the full traceback if it would be useful in identifying the problem. I am using Twisted, Twisted Web, and Zope Interfaces installed from the 2005-11-06 Twisted "Sumo" build on an OS X 10.3.9 G5 running python 2.3 (the latest for OS 10.3.x). cheers, Charlie

2 2

Donovan Preston: Nevow wiki content
by Richard Wall 06 Jan '06

06 Jan '06

Donovan, I really like the document you wrote about Nevow / Formless http://www.python.org/pycon/dc2004/papers/60/ ...and wanted your permission to copy and update some of it for use on the Nevow wiki. Or perhaps you'd prefer to update it yourself. Either way I think it would be a valuable addition to the wiki. Thanks. -- Richard Wall

3 2

Nevow PageCache
by Andrea Arcangeli 06 Jan '06

06 Jan '06

I wanted to push into the tracker but I'm a bit confused on how to do that with trac, so I resend with the mailing list. This patch implements html caching at the page level with a optional max cache size per page and with optional timeout. Even when the timeout is very small, this makes sure that all blocked waiting clients gets the same copy of the html allowing the server to scale. This only works for dynamic data that is the same for all users, so it's good for the homepage etc... I use it online for months and I had no problems at all. Index: Nevow/nevow/util.py =================================================================== --- Nevow/nevow/util.py (revision 3434) +++ Nevow/nevow/util.py (working copy) @@ -133,6 +133,7 @@ from twisted.python.failure import Failure from twisted.trial.unittest import deferredError from twisted.python import log + from twisted.internet import reactor try: # work with twisted before retrial Index: Nevow/nevow/rend.py =================================================================== --- Nevow/nevow/rend.py (revision 3434) +++ Nevow/nevow/rend.py (working copy) @@ -491,6 +491,56 @@ self.children[name] = child +class PageCache(object): + def __init__(self): + self.__db = {} + def cacheIDX(self, ctx): + return str(url.URL.fromContext(ctx)) + def __storeCache(self, cacheIDX, c): + self.__db[cacheIDX] = c + def __deleteCache(self, cacheIDX): + del self.__db[cacheIDX] + def __deleteCacheData(self, cacheIDX, page): + size = self.__db[cacheIDX][1] + assert len(self.__db[cacheIDX][0]) == size + page.subCacheSize(size) + self.__deleteCache(cacheIDX) + def __lookupCache(self, cacheIDX): + return self.__db.get(cacheIDX) + def getCache(self, ctx): + cacheIDX = self.cacheIDX(ctx) + c = self.__lookupCache(cacheIDX) + + if c is None: + self.__storeCache(cacheIDX, [util.Deferred()]) + return + + if isinstance(c[0], util.Deferred): + d = util.Deferred() + c.append(d) + return d + + return c[0] + def cacheRendered(self, ctx, data, page): + cacheIDX = self.cacheIDX(ctx) + defer_list = self.__lookupCache(cacheIDX) + assert(isinstance(defer_list[0], util.Deferred)) + size = len(data) + if page.canCache(size): + # overwrite the deferred with the data + timer = None + if page.lifetime > 0: + timer = util.reactor.callLater(page.lifetime, + self.__deleteCacheData, cacheIDX, page) + page.addCacheSize(size) + self.__storeCache(cacheIDX, (data, size, timer, )) + else: + self.__deleteCache(cacheIDX) + for d in defer_list: + d.callback(data) + +_CACHE = PageCache() + class Page(Fragment, ConfigurableFactory, ChildLookupMixin): """A page is the main Nevow resource and renders a document loaded via the document factory (docFactory). @@ -504,8 +554,27 @@ afterRender = None addSlash = None + cache = False + lifetime = 0 + max_cache_size = None + __cache_size = 0 + flattenFactory = lambda self, *args: flat.flattenFactory(*args) + def hasCache(self, ctx): + if not self.cache: + return + return _CACHE.getCache(ctx) + def addCacheSize(self, size): + assert self.canCache(size) + self.__cache_size += size + def subCacheSize(self, size): + self.__cache_size -= size + assert self.__cache_size >= 0 + def canCache(self, size): + return self.max_cache_size is None or \ + self.__cache_size + size <= self.max_cache_size + def renderHTTP(self, ctx): if self.beforeRender is not None: return util.maybeDeferred(self.beforeRender,ctx).addCallback( @@ -530,11 +599,20 @@ if self.afterRender is not None: return util.maybeDeferred(self.afterRender,ctx) - if self.buffered: + c = self.hasCache(ctx) + if c is not None: + assert self.afterRender is None + finishRequest() + return c + + if self.buffered or self.cache: io = StringIO() writer = io.write def finisher(result): - request.write(io.getvalue()) + c = io.getvalue() + if self.cache: + _CACHE.cacheRendered(ctx, c, self) + request.write(c) return util.maybeDeferred(finishRequest).addCallback(lambda r: result) else: writer = request.write

2 3

Resource.child_()
by Wilfredo Sánchez Vega 06 Jan '06

06 Jan '06

I'm wondering how married we are to the child_ method registration implementation for children in Resource. it's a bit of a strange way to do this... it means that valid URI characters can't be used, for one thing. Additionally, File doesn't use this mechanism at all, and instead puts registered children into a dictionary instead. That implementation is more flexible, but not compatible with the documented API in its Resource superclass, which says that putChild ("foo", x) means child_foo() = x. Furthermore, Resource also allows for a childFactory() method to be a source for children, and that API is also ignored by File. My feeling is that Resource should be doing none of this at all, and that there should be two subclasses (ie. FunkyChildMethodDispatchingResource and FactoryResource) which do one or the other, if we need these mechanisms to be provided by web2 in a generic class at all. Going to the extreme, the notion of resources having children at all is a strange thing to build into a generic HTTP resource. Resources that contain other resources (collections) are specified in WebDAV, but (intentionally) not in HTTP. A lot of implementations map URIs with slashes in them to a hierarchical back-end (eg. a file system), but that's not inherrent in HTTP; /foo and /foo/ can be different and completely unrelated resources. That leads me to think that locateChild() itself shouldn't be an IResource thing, but perhaps that's too pedantic given common usage of URI paths. -wsv

1 0

XForms HTTP POST with twisted web ?
by Alain Fréhel 03 Jan '06

03 Jan '06

Hi, I have an application using XForms and I would like to know if it is possible to receive (and process) the XML instance sent via XForms' HTTP POST submission (that is having method="post" in the <submission> element). Until now the only way I manage to receive any data is by using method="get" or method="urlencoded-post", which is not what I want. I need the original XML instance, not a dictionary of elements/values. I'm debuting with twisted, so if I'm missing something trivial, please forgive me. I'm using twisted 1.3.0-8 (Debian Sarge). -- Alain Fréhel

2 1

Few forms on page.
by Maciej Kucharz 03 Jan '06

03 Jan '06

Hi, Is there any way to add few forms (Matt's forms) to one page ? I tried to use fragments (rend.Fragment), but i have only errors. Thanks for any advice.. -- Maciej Kucharz

1 0

Is it possible to call the client from the server with nevow/athena
by Martin Möllenbeck 01 Jan '06

01 Jan '06

Hi and happy new year, is it possible to call the client from the server with nevow/athena like nevow/livepage? I have take a look to the source, but I didn't unterstand it quite well. Any help would be great. Thanks Martin

2 2