Twisted-web January 2006

twisted-web@python.org

39 participants
28 discussions

Start a nNew thread

Test
by Philippe Lafoucrière 05 Aug '21

05 Aug '21

Hi all just a test

3 2

guard in web2?
by Andrea Arcangeli 03 Feb '06

03 Feb '06

Hello, is there a guard functionality in web2 that could obsolete the nevow guard? Is it under a different name perhaps? I think the guard could very well belong to web2 the same way vhost does. Or should I rewrite the guard in my code? Comments welcome, thanks! PS. if my code will be broken anyway by the obsolescence of formless and web2, to be safer in terms of API stability for the long term future, I'm considering moving away from divmod products and stick with twisted only that has a bigger chance to be solid, but the guard is something I'd rather not want to duplicate in my code given it may be useful to more people like vhost.

2 1

Formless disabling widget
by Suha Onay 26 Jan '06

26 Jan '06

Hi, Is there a way to disable a formless widget(i.e. annotate.Text) ? I could not find any solution. ... annotate.Text(..., disable=True) ... annotate.Text(..., attributes={'disable': True}) Thanks -- Suha

4 5

Actual Useful Post
by glyph＠divmod.com 24 Jan '06

24 Jan '06

Going back over the recent thread that everyone else has probably already killfile'd, I have extracted several useful points, and I will present them here detached from the other thread so they don't get lost in the noise. The fact that I took exception to the presentation of these ideas doesn't mean that the problems aren't real. #1, something for the Twisted team to do The bugtracker(s) need to be kept in a more up-to-date state. If a patch has been stalled for lack of tests, it should be very clear that it's been stalled for lack of tests so that people can write some. For lack of a better convention, maintainers, please look at your bug-lists and make sure that the status is "need-eg" (in roundup) or it has the keyword "testblocked" (in trac) if they need more tests or an explanation. There is a stark difference between tickets which might not receive attention because the maintainer doesn't have any time, and tickets which definitely won't receive attention because other tickets that have tests already, or are in the maintainer's interests, are higher priority. This will also make it easier for people who don't want to submit tests to submit patches and have a mechanism for test fanatics to contribute tests to those fixes they want merged. In trac I have also tried to mark tickets with "low" or "lowest" priority if they are tickets that I would prefer someone to steal from me. I would suggest that others do the same. #2, something for the community to do Twisted, and Twisted's web-related projects should have more regular releases, so that other projects don't need to track them or deal with self-installed monkeypatches any longer than necessary. Although I've defended this practice, and I do think that it's a requirement of the practical necessity of providing the community with stable releases, it's *not* ideal; if everyone were running apt and we had a dedicated package maintainer and a mechanism for strict backwards compatibility verification, I'd want to update every copy of Twisted in the world when we fixed a bug. What we need to do a stable release of Twisted is fixes for the few release-critical bugs, and then user-testing of the package. Normally the release manager does this, but the more feedback we have, the surer we are that the package is ready to go, and the less that the release manager has to do. You can help out by looking at the 'admin/subproject.txt' in Twisted. Make a release using the technique described there, run the unit tests, use some applications, and verify that whatever code you've written using Twisted is still working. Look over the SVN log for noteworthy changes and help write a NEWS file that summarizes them comprehensibly. Anything that turns the buildbot greener also increases the chance that a release will be forthcoming. I am lead to believe that Chris Armstrong (our GLORIOUS release-manager, hallowed be his name for all time, amen) is going to be putting together a release this weekend. I hope he will post a followup that includes more detailed instructions. (Hint, hint.)

3 4

Unhashable Instance with Twisted imported into Jython
by Stephen Nightingale 23 Jan '06

23 Jan '06

Not sure if this is a Twisted issue or a Jython issue or both, so I'll post both ways. Using Jython 2.2 alpha, the latest. Trying to import Twisted into Jython on Win XP I get: TypeError: unhashable instance With the following stacktrace: Jython 2.2a1 on java1.4.2_03 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> import twisted Traceback (innermost last): File "<console>", line 1, in ? File "c:\R.and.D\Jython\Lib\site-packages\twisted\__init__.py", line 18, in ? File "c:\R.and.D\Jython\Lib\site-packages\zope\interface\__init__.py", line 54, in ? File "c:\R.and.D\Jython\Lib\site-packages\zope\interface\interface.py", line 802, in ? File "c:\R.and.D\Jython\Lib\site-packages\zope\interface\interface.py", line 418, in __in it__ File "c:\R.and.D\Jython\Lib\site-packages\zope\interface\interface.py", line 225, in __in it__ File "c:\R.and.D\Jython\Lib\weakref.py", line 147, in __init__ TypeError: unhashable instance >>> There is a prior issue that Jython doesn't recognize site-packages on Win XP, though this seems to be solved with: sitedirs = [prefix, makepath(prefix, "Lib", "site-packages")] Can anybody in this list shed light on the problem? Regards, Stephen Nightingale. --------------------------------- Yahoo! Cars NEW - sell your car and browse thousands of new and used cars online search now ---------------------------------

2 1

how to send parallel requests to the dbms while rendering html with Cheetah
by Andrea Arcangeli 22 Jan '06

22 Jan '06

Hello, With Cheetah I have full control on the serialization or parallelism of my queries and thanks to a real dbms I can execute very expensive queries in parallel on multiple cpus with just a small change: --- server/web.py 20 Jan 2006 03:16:08 -0000 1.113 +++ server/web.py 22 Jan 2006 23:24:05 -0000 @@ -907,6 +907,17 @@ class root_page_class(cached_basepage_cl t.title = 'KLive: Linux Kernel Live Usage Monitor' s.write(t.block_title()) + render_list = [ + self.render_nr_sessions(req), + defer.maybeDeferred(archs_class().render, req), + defer.maybeDeferred(kernels_class().render, req), + defer.maybeDeferred(fs_class().render, req), + defer.maybeDeferred(mod_class().render, req), + defer.maybeDeferred(pci_class().render, req), + defer.maybeDeferred(branch_class().render, req), + defer.maybeDeferred(vendor_class().render, req), + ] + def finish(result): t.latest_install = 'klive.sh' t.latest_tac = 'klive.tac' @@ -916,35 +927,21 @@ class root_page_class(cached_basepage_cl s.write(self.render_copyright(req)) s.finish() - def render_branch(result): - d = defer.maybeDeferred(branch_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(finish) - def render_pci(result): - d = defer.maybeDeferred(pci_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_branch) - def render_vendor(result): - d = defer.maybeDeferred(vendor_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_pci) - def render_mod(result): - d = defer.maybeDeferred(mod_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_vendor) - def render_fs(result): - d = defer.maybeDeferred(fs_class().render, req) - d = d.addCallback(s.write) - d = d.addCallback(render_mod) - + def render_list_callback(result): + try: + d = render_list.pop(0) + except IndexError: + finish(result) + else: + d = d.addCallback(s.write) + d = d.addCallback(render_list_callback) def back_archs(result): t.archs = result s.write(t.block_archs()) - d = defer.maybeDeferred(kernels_class().render, req) + d = render_list.pop(0) d = d.addCallback(s.write) - d = d.addCallback(render_fs) - + d = d.addCallback(render_list_callback) def back_nr_sessions(result): t.nr_sessions = result t.live = live_class().render(req) @@ -954,9 +951,11 @@ class root_page_class(cached_basepage_cl t.ip = ip_class().render(req) s.write(t.block_desc1()) - d = defer.maybeDeferred(archs_class().render, req).addCallback(back_archs) + d = render_list.pop(0) + d.addCallback(back_archs) - self.render_nr_sessions(req).addCallback(back_nr_sessions) + d = render_list.pop(0) + d.addCallback(back_nr_sessions) return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, I've an old (slow) 2-way smp server, and by sending (readonly) queries to postgresql in parallel I just reduce the time it takes to render the usual klive pages from 2.9sec to 1.49sec (2 cpus working at 100% executes the queries in parallel and so the results now arrives in _exactly_ half the time, postgresql is really scaling perfectly in this test). This new smp improvement comes after the huge boost I had after moving from Nevow to Cheetah for the rendering of the html (from >5.9sec to 2.9sec). On my 4-way dualcore opteron desktop the rendering time changes from ~1sec to ~370msec. So the overall since I started migrating to Cheetah, the time it takes to render the klive homepage went down from 5.9sec (old nevow+twisted.web code) to 1.5sec (new cheetah+twisted.web2 code). A ~300% performance improvement in rendering pages isn't bad. The code is almost unchanged, just the html rendering model has changed. This is just to share my real life experience and perhaps this could be useful to others. Now the 1.5sec are almost all spent waiting the sql queries and anyway both cpus are fully utilized (Cheetah renders the first #blocks while the other cpu calculates the results for the next block in parallel), so there's probably not much left to optimize (nothing as easy as the changes I did in the last days to move from 5.9sec to 1.5sec at least). Parallel expensive queries to the db, is something that the axiom model will never be able to achieve (as long as the api isn't changed to return deferreds to allow real dbms instead of sqllite, which means all code written today for the current axiom API will have to changed significantly). Having to schedule to another context while waiting the deferred to fire, may open a window for for race conditions too, and have fun writing unit-test verifying race conditions. While you can scale the network load by running multiple twisted servers, you can't scale the db queries coming from different twisted servers in parallel with the axiom API. This ignoring the fact nevow doesn't contemplates the possibility of sending all queries in parallel.

3 3

Different behaviour of Nevow loaders xmlstr and stan when used in macro methods.
by Richard Wall 22 Jan '06

22 Jan '06

A few hours ago I posted a unit test to #twisted.web that I thought demonstrated a problem I'm having with Nevow. It was wrong though, and this is an updated version: http://rafb.net/paste/results/sf5B5s57.html What the test is supposed to demonstrate, is a difference in the behaviour of loaders.stan and loaders.xmlstr (or xmlfile). The test contains a rend.Page class with a macro_pattern(ctx, name) method. The purpose of the macro is to lookup patterns called "name" from a nevow.loaders.* instance at self.patternFactory. If a matching pattern is found in patternFactory, it's children are inserted into the tag that called the macro otherwise the original contents are returned instead. So given a patternFactory: <div n:pattern="body"> <p>Page specific Text</p> </div> ...and docFactory:... <div n:macro="pattern body"> <p>Default text</p> </div> You'd get: <div> <p>Default text</p> </div> That works fine with both loaders.stan and loaders.xmlstr. If however the pattern its self uses the same macro, I get a recursion error in loaders.xmlstr only. eg <div n:pattern="body"> <div n:macro="pattern menu"> <ul> <li>Default menu</li> </ul> </div> <p>Page specific Text</p> </div> I can understand why this might cause recursion, and i have managed to fix it in my app by adding checks to the macro method. I just don't understand why the xml and stan loaders don't behave the same. I have tried to isolate where the recursion starts and it seems to be in loaders.xmlstr.load (~L200): doc = flatsax.parseString(self.template, self.ignoreDocType, self.ignoreComment) # recursion seems to start here. doc = flat.precompile(doc, ctx) ...I can't work out how the stan version prevents the recursion though. Has anyone come across this before? Is it a bug or just an unsupported use of macros? If anyone has any ideas, I'd be grateful. I hope to post a fuller example of what I'm trying to achieve with this later. Something like what I remember of Zope, where child pages can inherit page patterns from parent pages. -- Richard Wall

2 1

CPUShare-Twisted
by Andrea Arcangeli 22 Jan '06

22 Jan '06

Hello everyone, if you've pending patches that have not been applied or you've no time for unittests until your project works, you can consider switching to CPUShare-Twisted. Mercurial allows distributed development so we can merge our branches against each other with very little effort. If you've suggestion for betters names you're welcome. I could have done it over sourceforge but they lack mercurial AFIK and since I've the setup on cpushare.com I was quick to add Twisted to it. We can move it elsewhere later if needed (especially if it consumes too much network bandwidth ;). I have no interest to maintain twisted except for the fact I'm using it in semi-production and I plan to use it in production, and so I need it to be stable, and with all possible bugfixes and features applied without formalities. I assume other business usages of twisted may have the same needs, so perhaps we can join our efforts with this project. My choice was to either dump twisted or to maintain it locally, and so I clearly have an interest to make the CPUShare-Twisted branch public and to include other people fixes and features (I was already maintaining it over the last year except it was just a patch with fixes that I failed to push in SVN, and not a real fork). You can find the quickstart, basic objectives (including the not trying to reinvent the wheel) and mailing list details on the project page: http://www.cpushare.com/twisted I'll keep tracking SVN. I'll also keep submitting patches upstream since one can always hope for the best. Also note, over the last year I've fixed at least one bug in core twisted basic protocols that would have never been found with any unittest out there. Careful auditing and reading code and filtering of the patches, and thinking deeply about the design before writing code (to write the code in a way that won't break easily over time), is much more important than spending time on unittests. Unittests still makes perfect sense after stuff is included and works in basic testing, but they should be separated from the logic of committing valid patches to the tree. NOTE: before clicking on that page you may want to also answer these questions: 1) is python much better than ruby and all other language on earth? 2) is twisted much better than any other framework to write network apps? 3) are twisted and python fast and scalable enough for all applications? 4) is the single threaded model scalable enough for all applications in smp? 5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network? 6) would you rather prefer to go broke than to use code "not-invented-here" or not written with python and twisted? (of course the python interpreter the c compiler and the underlying operative system are magically excluded from the not-invented-here clause for whatever unknown reason) I'm welcome anybody who wants to join cpushare-twisted, but if the answer of _any_ of the above questions is "yes", I think you may be better off ignoring this email.

3 5

Re: [Twisted-web] fix web2 connection lost error handler
by glyph＠divmod.com 22 Jan '06

22 Jan '06

On Sun, 22 Jan 2006 05:27:18 +0100, Andrea Arcangeli <andrea(a)cpushare.com> wrote: >comments and patch here: > >http://www.cpushare.com/hg/Twisted/?cs=a924218612ef Please send comments and patches to the bugtracker: http://twistedmatrix.com/bugs/ It's a basic courtesy, like using the proper format of diff. You can just as easily link to external repositories or files from there as from the mailing list. I skimmed the patch since it was so short, although I do wish it had been in the tracker. - You're disabling logging which someone else may have been depending upon in the same revision as you're fixing an arguably legitimate bug. An exception was being trapped incorrectly. Maybe you want to log a message rather than an error in the event of disconnects? Maybe it should be an option? At any rate, not separating the issues makes it harder to include the patch. - The aforementioned 'legitimate bugfix' is still wrong, except now it's slow and stylistically wrong instead of fast and logically wrong. The appropriate API to use there is reason.check(error.ConnectionLost). (Actually, I personally think it should be reason.trap(error.ConnectionLost), but then you'd depend on finalizing-deferred exception reporting, and it looks like someone was trying to avoid that...?) - You didn't really explain what the problem was, besides log noise. What is the expected behavior? Why is it expected? What is the incorrect observed behavior? You're randomly changing things around to suit your personal preferences (Twisted does not include an SMS log exception reporter, so I'm sure that wasn't bothering anyone else), and while that's fine for you, to be included in a release other people will use, it should have some justification. One procedural error for every line of code that you changed - *and* you didn't write tests, not because you didn't have time yet, but because you refuse to. Do you really wonder why your patches don't get included?

2 1

web2 log patch
by Andrea Arcangeli 22 Jan '06

22 Jan '06

This fixes a typo in CVS and adds a common handler for file logging: Index: Twisted/twisted/web2/log.py =================================================================== --- Twisted/twisted/web2/log.py (revision 15644) +++ Twisted/twisted/web2/log.py (working copy) @@ -137,7 +137,7 @@ request = eventDict['request'] response = eventDict['response'] loginfo = eventDict['loginfo'] - firstLine = '%s %s HTTP/%' %( + firstLine = '%s %s HTTP/%s' %( request.method, request.uri, '.'.join([str(x) for x in request.clientproto])) @@ -166,6 +166,13 @@ tlog.removeObserver(self.emit) +class FileAccessLoggingObserver(BaseCommonAccessLoggingObserver): + def __init__(self, logpath): + self.f = file(logpath, 'a', 1) + + def logMessage(self, message): + self.f.write(message + '\n') + class DefaultCommonAccessLoggingObserver(BaseCommonAccessLoggingObserver): """Log requests to default twisted logfile.""" def logMessage(self, message):

11 22

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

Twisted-web January 2006