Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)
On 10:41 pm, leyssw@ihug.co.nz wrote:
Isn't the simplest option to place a decent reverse proxy between the webserver and our clients?
Possibly so! I gave this a half-hearted attempted a few weeks ago but quickly gave up. If someone who is familiar with configuring such a proxy would like to help out, perhaps we can get something useful up and running.
For example, varnish will cache and proxy simple GETs, so simple views of the front page and docs (which I imagine is a large chunk of the traffic) will be offloaded from the webserver: http://varnish.projects.linpro.no/wiki/VarnishFeatures
Also, is the current installation (Db, apache + trac) all on one machine or are the DB and the webserver split already?
They're all on one machine. Changing this is an expensive proposition with a hard to predict payoff, so it's rather low on the list of ideas to try.
These would be easy ways to throw more hardware at the problem.
What is the current web-server configuration?
Can you be more specific about which configuration you're interested in?
Can you share traffic stats for the top 20 pages, so we get an idea of where the load is?
Yep. Here are counts for the last few days. Rank, request count, URL. 1) 22058 /trac/ 2) 8742 /trac/attachment/wiki/TSF/SponsorLogos/seomoves.png?format=raw 3) 8646 /trac/attachment/wiki/TSF/SponsorLogos/zenoss.2.png?format=raw 4) 8550 /trac/attachment/wiki/TSF/SponsorLogos/Appropriate- Solutions-Inc.gif?format=raw 5) 8499 /trac/attachment/wiki/TSF/SponsorLogos/securitymetrics.png?format=raw 6) 3075 /trac/wiki/Documentation 7) 2615 /trac/timeline?changeset=on&max=100&daysback=10&format=rss 8) 2448 /trac/wiki/Downloads 9) 1788 /trac/wiki/FrequentlyAskedQuestions 10) 1356 /trac/wiki 11) 1019 /trac/wiki/TwistedProject 12) 966 /trac/wiki/TracGuide 13) 828 /trac/wiki/TwistedWeb 14) 814 /trac/robots.txt 15) 781 /trac/search 16) 710 /trac/wiki/TwistedProjects 17) 662 /trac/wiki/WikiStart?format=txt 18) 605 /trac/wiki/TwistedAdvantage 19) 595 /trac/report/9 20) 590 /trac/timeline?milestone=on&ticket=on&changeset=on&wiki=on&max=50&daysback=90&format=rss Looking at this, I'm reminded that we should try moving the sponsor logos to somewhere else Unfortunately one thing we don't know is the relative cost of serving each of these URLs. The sponsor logos, while accounting for a significant number requests, may actually be a very small contribution to overall load. Anyhow, thanks! Jean-Paul
On Nov 7, 2009, at 6:05 PM, exarkun@twistedmatrix.com wrote:
Looking at this, I'm reminded that we should try moving the sponsor logos to somewhere else
1> Move any images to CDN of any description (Cloud Files @ rackspace does this for free, if I remember correctly, included in the cost of storage; I'd be happy to give you some of mine) 2> None of your main content has expires headers on it -- without that, they're not really cacheable, so images, css, etc are getting loaded every time. S
On 7 Nov, 11:56 pm, ssteinerx@gmail.com wrote:
On Nov 7, 2009, at 6:05 PM, exarkun@twistedmatrix.com wrote:
Looking at this, I'm reminded that we should try moving the sponsor logos to somewhere else
1> Move any images to CDN of any description (Cloud Files @ rackspace does this for free, if I remember correctly, included in the cost of storage; I'd be happy to give you some of mine)
Moving them to somewhere on twistedmatrix.com where they might be cached and don't need to involve trac to be served would probably be a good enough start. That's where most of the effort is anyway (as URLs will need changing on various pages).
2> None of your main content has expires headers on it -- without that, they're not really cacheable, so images, css, etc are getting loaded every time.
Yes. That's one of trac's problems. Jean-Paul
On 7 Nov, 11:05 pm, exarkun@twistedmatrix.com wrote:
On 10:41 pm, leyssw@ihug.co.nz wrote:
Isn't the simplest option to place a decent reverse proxy between the webserver and our clients?
Possibly so! I gave this a half-hearted attempted a few weeks ago but quickly gave up. If someone who is familiar with configuring such a proxy would like to help out, perhaps we can get something useful up and running.
Actually, I just looked at Varnish again and remembered that because of the way trac sessions work, it's really hard to figure out what may and may not be correctly cached. I don't know how to resolve this without serious changes to trac itself. Jean-Paul
participants (2)
-
exarkun＠twistedmatrix.com -
ssteinerX＠gmail.com