Hey,
I operate Omegle (http://omegle.com/), which is a COMET-driven Twisted.Web app. (It doesn't use Nevow or anything; just Twisted.Web.) It gets fairly heavy traffic; right now, it peaks daily at around 6500 concurrent users. Technical details:
- Hosted on a Linode 2880 virtual private server (https://www.linode.com/); Ubuntu, 2880 MB of RAM, 4 cores available - EPollReactor
Omegle currently runs at close to 100% CPU for most of the day, and I'm a bit concerned about that. Here are some questions I have:
- How much more load might my app be able to take before it dies? Currently, during peak load, a request from localhost is completed in around 350ms; not ideal, but acceptable. I think anything under 1000ms would be all right, but I'd prefer the latency to be as low as possible, of course. Is this likely to scale linearly with traffic, or what? - I've looked at my code pretty closely, and it doesn't seem to be doing anything particularly inefficient, but are there any tricks I might be able to use to increase performance? Tweaking settings, etc.? - Ultimately, I think my app will need to be able to use multiple cores. If I have to, I'm going to either rewrite it in Erlang, or rewrite it to load balance across multiple Twisted processes. Is there anything easier I could do to allow the use of multiple cores?
Thanks a lot, Leif
On Sun, Jan 3, 2010 at 5:26 AM, Leif K-Brooks eurleif@gmail.com wrote:
Hey,
I operate Omegle (http://omegle.com/), which is a COMET-driven Twisted.Web app. (It doesn't use Nevow or anything; just Twisted.Web.) It gets fairly heavy traffic; right now, it peaks daily at around 6500 concurrent users. Technical details:
You should write a success story! I've used Omegle when the boredom hits hard.
- Hosted on a Linode 2880 virtual private server
(https://www.linode.com/); Ubuntu, 2880 MB of RAM, 4 cores available
- EPollReactor
Omegle currently runs at close to 100% CPU for most of the day, and I'm a bit concerned about that.
Try use the standard reactor instead of epoll. I found a similar problem when I switched my product to the epoll reactor, the cpu usage problem went away when I reverted back to the normal reactor.
Hey I believe twisted is developed in a way that it can only use one processor at a time. As process scheduling is part of operating system it can only schedule one twisted reactor based application on one processor. For taking advantage of multiple processor you can use fork, but then it is not twisted way of doing stuff,, i am a new bie in twisted world, so comments are appreciated. Moreover I think twisted is similar to C select for tcp server
Thanks Arun
Sent from my iPhone
On Jan 3, 2010, at 12:20 AM, Colin karnaugh@karnaugh.za.net wrote:
On Sun, Jan 3, 2010 at 5:26 AM, Leif K-Brooks eurleif@gmail.com wrote: Hey,
I operate Omegle (http://omegle.com/), which is a COMET-driven Twisted.Web app. (It doesn't use Nevow or anything; just Twisted.Web.) It gets fairly heavy traffic; right now, it peaks daily at around 6500 concurrent users. Technical details:
You should write a success story! I've used Omegle when the boredom hits hard.
- Hosted on a Linode 2880 virtual private server
(https://www.linode.com/); Ubuntu, 2880 MB of RAM, 4 cores available
- EPollReactor
Omegle currently runs at close to 100% CPU for most of the day, and I'm a bit concerned about that.
Try use the standard reactor instead of epoll. I found a similar problem when I switched my product to the epoll reactor, the cpu usage problem went away when I reverted back to the normal reactor.
Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Leif K-Brooks wrote: [...]
- EPollReactor
Omegle currently runs at close to 100% CPU for most of the day, and
100% CPU with epoll sounds like http://twistedmatrix.com/trac/ticket/2809, which was fixed in Twisted 8.0.0. Which version of Twisted are you using?
[...]
- I've looked at my code pretty closely, and it doesn't seem to be
doing anything particularly inefficient, but are there any tricks I might be able to use to increase performance? Tweaking settings, etc.?
Run it under a profiler. You can't know where the time is going if you don't measure. Guesses about this sort of thing are usually wrong, so measurement is vital.
-Andrew.
On Sun, Jan 3, 2010 at 7:32 AM, Andrew Bennetts andrew@bemusement.org wrote:
Leif K-Brooks wrote: [...]
- EPollReactor
Omegle currently runs at close to 100% CPU for most of the day, and
I really hope us people on this mailing list can help you fix this issue, instead of you re-writing your (very cool) app in Erlang or whatever. In my opinion, then we will have a success story.
Load-balancing over several twisted.web instances seems like something easy to try, have you looked harder into that yet? Maybe because you are running the entire app on just one machine you are seeing some sort of contention between processes?
More statistics/benchmarks could really help us pin-point the issue.
100% CPU with epoll sounds like http://twistedmatrix.com/trac/ticket/2809, which was fixed in Twisted 8.0.0. Which version of Twisted are you using?
It looks like he is using version "TwistedWeb/8.1.0". (from the Firefox 'server spy' plugin I have).
-Alex
[...]
- I've looked at my code pretty closely, and it doesn't seem to be
doing anything particularly inefficient, but are there any tricks I might be able to use to increase performance? Tweaking settings, etc.?
Run it under a profiler. You can't know where the time is going if you don't measure. Guesses about this sort of thing are usually wrong, so measurement is vital.
-Andrew.
Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (5)
-
Alex Clemesha -
Andrew Bennetts -
Arun Gmail -
Colin -
Leif K-Brooks