Hello, With Cheetah I have full control on the serialization or parallelism of my queries and thanks to a real dbms I can execute very expensive queries in parallel on multiple cpus with just a small change: --- server/web.py 20 Jan 2006 03:16:08 -0000 1.113 +++ server/web.py 22 Jan 2006 23:24:05 -0000 @@ -907,6 +907,17 @@ class root_page_class(cached_basepage_cl t.title = 'KLive: Linux Kernel Live Usage Monitor' s.write(t.block_title()) + render_list = [ + self.render_nr_sessions(req), + defer.maybeDeferred(archs_class().render, req), + defer.maybeDeferred(kernels_class().render, req), + defer.maybeDeferred(fs_class().render, req), + defer.maybeDeferred(mod_class().render, req), + defer.maybeDeferred(pci_class().render, req), + defer.maybeDeferred(branch_class().render, req), + defer.maybeDeferred(vendor_class().render, req), + ] + def finish(result): t.latest_install = 'klive.sh' t.latest_tac = 'klive.tac' @@ -916,35 +927,21 @@ class root_page_class(cached_basepage_cl s.write(self.render_copyright(req)) s.finish() - def render_branch(result): - d = defer.maybeDeferred(branch_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(finish) - def render_pci(result): - d = defer.maybeDeferred(pci_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_branch) - def render_vendor(result): - d = defer.maybeDeferred(vendor_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_pci) - def render_mod(result): - d = defer.maybeDeferred(mod_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_vendor) - def render_fs(result): - d = defer.maybeDeferred(fs_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_mod) - + def render_list_callback(result): + try: + d = render_list.pop(0) + except IndexError: + finish(result) + else: + d = d.addCallback(s.write) + d = d.addCallback(render_list_callback) def back_archs(result): t.archs = result s.write(t.block_archs()) - d = defer.maybeDeferred(kernels_class().render, req) + d = render_list.pop(0) d = d.addCallback(s.write) - d = d.addCallback(render_fs) - + d = d.addCallback(render_list_callback) def back_nr_sessions(result): t.nr_sessions = result t.live = live_class().render(req) @@ -954,9 +951,11 @@ class root_page_class(cached_basepage_cl t.ip = ip_class().render(req) s.write(t.block_desc1()) - d = defer.maybeDeferred(archs_class().render, req).addCallback(back_archs) + d = render_list.pop(0) + d.addCallback(back_archs) - self.render_nr_sessions(req).addCallback(back_nr_sessions) + d = render_list.pop(0) + d.addCallback(back_nr_sessions) return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, I've an old (slow) 2-way smp server, and by sending (readonly) queries to postgresql in parallel I just reduce the time it takes to render the usual klive pages from 2.9sec to 1.49sec (2 cpus working at 100% executes the queries in parallel and so the results now arrives in _exactly_ half the time, postgresql is really scaling perfectly in this test). This new smp improvement comes after the huge boost I had after moving from Nevow to Cheetah for the rendering of the html (from >5.9sec to 2.9sec). On my 4-way dualcore opteron desktop the rendering time changes from ~1sec to ~370msec. So the overall since I started migrating to Cheetah, the time it takes to render the klive homepage went down from 5.9sec (old nevow+twisted.web code) to 1.5sec (new cheetah+twisted.web2 code). A ~300% performance improvement in rendering pages isn't bad. The code is almost unchanged, just the html rendering model has changed. This is just to share my real life experience and perhaps this could be useful to others. Now the 1.5sec are almost all spent waiting the sql queries and anyway both cpus are fully utilized (Cheetah renders the first #blocks while the other cpu calculates the results for the next block in parallel), so there's probably not much left to optimize (nothing as easy as the changes I did in the last days to move from 5.9sec to 1.5sec at least). Parallel expensive queries to the db, is something that the axiom model will never be able to achieve (as long as the api isn't changed to return deferreds to allow real dbms instead of sqllite, which means all code written today for the current axiom API will have to changed significantly). Having to schedule to another context while waiting the deferred to fire, may open a window for for race conditions too, and have fun writing unit-test verifying race conditions. While you can scale the network load by running multiple twisted servers, you can't scale the db queries coming from different twisted servers in parallel with the axiom API. This ignoring the fact nevow doesn't contemplates the possibility of sending all queries in parallel.