Mailman 3 September 2005 - Twisted-web

Test
by Philippe Lafoucrière 05 Aug '21

05 Aug '21

Hi all just a test

3 2

[PATCH] Fix web2.client auth on Python 2.3
by Scott Lamb 30 Sep '05

30 Sep '05

The base64.b64encode() method doesn't seem to exist on Python 2.3. So I've switched it back to encodestring(...).strip(). I'm curious why buildbot didn't pick this up, since there is an existing unit test. web2 isn't in the setup.py list; maybe it should be? How does buildbot work? -- Scott Lamb <http://www.slamb.org/>

2 2

overkill context switches
by Andrea Arcangeli 29 Sep '05

29 Sep '05

Hello, when a page that requires _several_ sql queries (through psycopg2) is loaded, an overkill amount of context switches happens: 0 0 0 18772 614380 227500 0 0 0 20 1037 55 0 0 99 1 0 0 0 18772 614380 227500 0 0 0 0 1035 48 0 0 100 0 1 0 0 18772 614380 227500 0 0 0 0 1032 24613 17 31 53 0 0 0 0 18764 614380 227500 0 0 0 0 1086 6766 20 6 75 0 0 0 0 18764 614380 227500 0 0 0 0 1133 64 1 0 99 0 These shall be thread switches, otherwise I doubt my slow server system would be able to schedule 24k times per second and still provide decent performance. However those useless context switches are certainly wasting quite some cpu. So I'd like to fix this. There's an huge number of futex(FUTEX_WAKE) call in the strace, python calls futex_wake even when you simply invoke it with `python` from the shell (it makes no sense to call futex syscall until the first pthread_create is called, so this is certainly suboptimal but I'm unsure if it's related to the context switches, a flood of futex calls would just waste tonds of cpu with enter/exit kernels without necessairly switch the task). The point of futex is exactly to avoid entering the kernel in the fast path so this is all no-sense code. I could reproduce the overkill context switches even with linuxthreads instead of NPTL, so this is probably not a NPTL bug. It's also probably not a psycopg2 bug because psycopg has no notion of threading. This should be a python or twisted-thread-pool bug. I'm not sure about the best way to track it down, especially because so far I can reproduce it only on the server, that is a bit slower and that I cannot use for debugging. Can people try in their systems to run `vmstat 1` and then to reload some complex page, and see if you also get a flood of context switches for certainly no good reason? I wonder if perhaps a select call on the thread is buggy, and it keeps yielding the cpu to the other tasks for some reasons. This doesn't seem a starvation issue, it seems just that some thread instead of waiting keeps looping and trying. So this is noticeable only as a slowdown, there is no effective malfunction. If you can reproduce let me know, otherwise I'll try to debug it in the next days (it's not really urgent, since as said it's only a performance issue and if the system is under load the caching patches makes it fast anyway). OTOH watch the stats, 31% of the cpu is spent in system load, those are the context switches for sure, so the page would be rendered at least twice as fast if this was fixed, only 17% of the cpu was spent in userland rendering the page.

3 5

How to render an empty stan tag in Nevow
by Pedro Sanchez 28 Sep '05

28 Sep '05

Hello, I have a situation like this: class myPage(rend.Page): def render_salute(self, context, data): rv = <something-I-don't-know> if <condition>: rv = loaders.stan(T.p['You made it']) return rv docFactory = loaders.stan( .... T.invisible(render=T.directive('salute')) .... What I want in render_salute is to return 'nothing' if <condition> is false. Basically, as if the T.invisible in the docFactory didn't exist. I've tried <something-I-don't-know> = loaders.stan(T.p['']) but it has the side effect of forcing a line break which is undesirable (in my real code the T.invisible is inside a table cell). Even returning a   string would do it! Any suggestions? Thanks, -- Pedro

2 1

Flatten problem in livepage trunk
by Thomas HERVE 27 Sep '05

27 Sep '05

I've managed to make some tests on Livepage in Nevow trunk, and I've found a problem. The following code work without problem : def handle_action(self, ctx): client = livepage.IClientHandle(ctx) client.set("mynode", tags.span["test"]) But the following don't : def handle_action(self, ctx): client = livepage.IClientHandle(ctx) client.set("mynode", [tags.span["test1"], tags.span["test2"]]) (it prints 'test1''test2' instead of 'test1test2' This works on nevow 0.4.1. Am I doing something wrong, or is there really a problem ? My knowledge of the flatten mecanism is too poor for tracking the cause of this... By the way, I'll also look at the callRemote branch that seems to offer great things. Thanks, -- Thomas

2 3

Reminder: Virtual Twisted Sprint, 8-9 October
by Mary Gardiner 27 Sep '05

27 Sep '05

Just a reminder for everyone that there will be a bunch of people online on the weekend of 8-9 October working their hearts out on the Twisted codebase. All participants welcome, people unfamiliar with the codebase will need to be a bit self-motivated though in terms of being willing to find their own problem to work on (check the bug tracker for ideas). We're particularly keen for people who would be interested in adding unti tests. Sprinting discussion will be in #twisted and #twisted.web on freenode.net. There should be people around at most times. -Mary -- <e@ircnet> on the internet the concepts of time and space lose meaning

2 2

twisted.web2.stream.IStream.reset()?
by Scott Lamb 26 Sep '05

26 Sep '05

Hi David and all, I stumbled across your twisted web2-client branch from google. I need to make a specialized HTTP load generator that uses Keep-Alives. twisted.web2.client seems the best way, so I'm trying to make this branch work for me. My load generator needs to make a bunch of sequences of authenticated POSTs, picking up a cookie along the way. I've stumbled onto a bug: the first request will return a 401Unauthorized. HTTPClientFactory.handleStatus_401 deals with this...but the postdata is gone. This code generates the new request: req = Request(request.method, request.uri, request.args, request.headers, request.stream) It looks like web2.stream.IStream is single-use - it returns a None when there's no more data. I think this - or maybe a IResettableStream subclass - needs a reset() method. I've got a patch that allowed me to proceed. I wouldn't recommend applying as-is - I only messed with the MemoryStream that I'm using, and there's no unit test. I want to send this in before I forget about it, though. Regards, Scott -- Scott Lamb <http://www.slamb.org/>

3 4

[PATCH] web2.client: Callback on "no body" requests
by Scott Lamb 25 Sep '05

25 Sep '05

Got another one. On status 204, among others, my callback is not invoked. Test (just uncommented) and fix. Regards, Scott -- Scott Lamb <http://www.slamb.org/>

2 1

[PATCH] Distinct headers per web2.client.HTTPClientFactory
by Scott Lamb 25 Sep '05

25 Sep '05

I was getting failures because my "Authorization:" and "Cookie:" headers were getting shared between all HTTPClientFactory instances. Patch, with unit test. -- Scott Lamb <http://www.slamb.org/>

2 1

web2.client: Extraneous '\n' in Authorization: header
by Scott Lamb 25 Sep '05

25 Sep '05

The base64 encoding seems to introduce a newline. There was a strip in this line, but not in the right place to take care of this. (Does strip('') do anything? Is there some other reason to strip _before_ base64 encoding?) If the answers to those questions are both "No", this patch fixes it. I noticed this because I'm sending XML. The server was complaining about my "<?xml ...?>" preamble; apparently the '<' has to be the very first character of an XML document. -- Scott Lamb <http://www.slamb.org/>

2 2