[Twisted-Python] CPU choice for big twisted server.
Hello I am running a poll reactor with Twisted 2.2 on 2.6.11 kernel. "top" currently shows 80-90% CPU utilization for this twistd process. We have not switched to epoll Twisted 2.5 yet, although we are planning to. We are going to add boxes to run these servers. We have a choice of getting either Opteron 2216 with 1MB L2 cache and faster RAM access, or Intel Woodcrest 5140 with 4MB L2 and slower FB-DIMM RAM access. The current RSS size of the twisted process is about 60MB. The main load comes from poll() and from modifying long dictionaries of the protocol class instances (searching/modifying dictionaries of client sessions). Where should we expect a better performance: on Woodcrest 5140 with bigger L2 or on Opteron 2216 with faster RAM access? We need to order the new box tomorrow afternoon, so if someone could reply today, it'd be great for us! Thanks Alec.
On 01:51 am, matusis@matusis.com wrote:
Hi, Alec! Nice to hear that you're using Twisted in such a demanding situation.
I'm only replying to this message because of your extremely short timeframe. I have _no_ reliable data that I would feel comfortable making a decision based on, but then, either of these systems will likely be adequate to your purposes. So, take all this with a grain of salt, and I hope you get something better. My new MacBook - a core 2 duo with 4MB of cache - slightly (10% time difference) outstrips my desktop work machine - an opteron64 with 1MB of cache - when running the Twisted unit tests. The Opteron is almost 8 months old now, but the MacBook is a laptop. I am being intentionally vague about the rest of the specs here to stress the anecdotal and unscientific nature of this knowledge :-). However, this result doesn't surprise me, and I guess it may generalize to the current generation of hardware, which is to say, Intel wins. RAM is important, as is raw CPU speed, but L2 cache is _the_ major determinant of Python's performance in all of the cross-machine profiling I've done. That is to say, this experiment, and this one other time I think I ran some stuff on a PowerPC or a MIPS or something (those are different, right?). Good luck on making the right choice here with such tight constraints!
Alec Matusis wrote:
Being async, a twisted server can only make use of a single processor. With dual core chips, it can only use one of the cores. No matter what hardware you throw at it, you'll eventually max out that core. If you're planning to scale out a lot more, you'll want to structure things so you can load balance several (1 process per core) twisted processes running the service. cheers, Andy.
matusis@matusis.com wrote:
Incidentally, does Twisted API have anything for load-balancing? We have implemented it ourselves.
PythonDirector is based on Twisted: http://pythondirector.sourceforge.net/ You may also use nginx: http://wiki.codemongers.com/NginxLoadBalanceExample -- Nicola Larosa - http://www.tekNico.net/ There's no gentle way to break somebody's heart To say, it's over, it's time for us to part -- Prefab Sprout, Love Will Find Someone for You, The Gunman and Other Stories, 2001
On 01:51 am, matusis@matusis.com wrote:
Hi, Alec! Nice to hear that you're using Twisted in such a demanding situation.
I'm only replying to this message because of your extremely short timeframe. I have _no_ reliable data that I would feel comfortable making a decision based on, but then, either of these systems will likely be adequate to your purposes. So, take all this with a grain of salt, and I hope you get something better. My new MacBook - a core 2 duo with 4MB of cache - slightly (10% time difference) outstrips my desktop work machine - an opteron64 with 1MB of cache - when running the Twisted unit tests. The Opteron is almost 8 months old now, but the MacBook is a laptop. I am being intentionally vague about the rest of the specs here to stress the anecdotal and unscientific nature of this knowledge :-). However, this result doesn't surprise me, and I guess it may generalize to the current generation of hardware, which is to say, Intel wins. RAM is important, as is raw CPU speed, but L2 cache is _the_ major determinant of Python's performance in all of the cross-machine profiling I've done. That is to say, this experiment, and this one other time I think I ran some stuff on a PowerPC or a MIPS or something (those are different, right?). Good luck on making the right choice here with such tight constraints!
Alec Matusis wrote:
Being async, a twisted server can only make use of a single processor. With dual core chips, it can only use one of the cores. No matter what hardware you throw at it, you'll eventually max out that core. If you're planning to scale out a lot more, you'll want to structure things so you can load balance several (1 process per core) twisted processes running the service. cheers, Andy.
matusis@matusis.com wrote:
Incidentally, does Twisted API have anything for load-balancing? We have implemented it ourselves.
PythonDirector is based on Twisted: http://pythondirector.sourceforge.net/ You may also use nginx: http://wiki.codemongers.com/NginxLoadBalanceExample -- Nicola Larosa - http://www.tekNico.net/ There's no gentle way to break somebody's heart To say, it's over, it's time for us to part -- Prefab Sprout, Love Will Find Someone for You, The Gunman and Other Stories, 2001
participants (6)
-
Alec Matusis -
Andy Gayton -
glyph＠divmod.com -
Itamar Shtull-Trauring -
matusis＠matusis.com
-
Nicola Larosa