Re: [Twisted-Python] Twisted server is 5 times SLOWER on Solaris than Linux?
On 01:59 am, jarrod@vertigrated.com wrote:
We have LOTS and LOTS of test harness code and profiling code to pinpoint where bottlenecks are. We are going to have process a couple of terabytes a day thru this system. Latency thru the system is a high priority because of what kind of system it is.
If you do manage to keep Twisted in the loop on this project, please share what strategy you used. It is unfortunately rare that Twisted is itself a really performance-critical component of any system (especially a highly parallel system), so information about optimization is hard to find.
On 1/18/07, glyph@divmod.com <glyph@divmod.com> wrote:
On 01:59 am, jarrod@vertigrated.com wrote:
We have LOTS and LOTS of test harness code and profiling code to pinpoint where bottlenecks are. We are going to have process a couple of terabytes a day thru this system. Latency thru the system is a high priority because of what kind of system it is.
If you do manage to keep Twisted in the loop on this project, please share what strategy you used. It is unfortunately rare that Twisted is itself a really performance-critical component of any system (especially a highly parallel system), so information about optimization is hard to find.
It isn't really a failing of Twisted, on non-sparc hardware it preforms awesome! The SPARC hardware is the problem, and the T1000 / T2000 we have be benchmarking just doesn't place nice with single threaded CPU bound apps. We are going to try and replace Twisted with Apache 2.2.x and create a module that does raw connection handliing, and go with the threaded mpm and see how that preforms with the same test client harness we are benchmarking with now.
It would be interesting to know where the CPU time is actually going.
From what I've seen of this thread, performance is worst on the SPARC hardware, but that doesn't necessarily mean that it's the root cause. It may be, e.g., the implementation of some service call for that hardware is especially slow. Do you have any profiling information that shows whether the CPU time is spent in Twisted code itself, or in some system call that it's making?
BTW, the fact that you get 1k+ connections/s on the x86 hardware is still slow (if you're just running benchmarking code that isn't doing any work per request). With epoll you should be able to get way more than that. Just my $.02, -- Jacob ________________________________ From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jarrod Roberson Sent: Thursday, January 18, 2007 2:11 PM To: Twisted general discussion Subject: Re: [Twisted-Python] Twisted server is 5 times SLOWER on Solaris thanLinux? On 1/18/07, glyph@divmod.com <glyph@divmod.com> wrote: On 01:59 am, jarrod@vertigrated.com wrote: >We have LOTS and LOTS of test harness code and profiling code to pinpoint >where bottlenecks are. We are going to have process a couple of terabytes a >day thru this system. Latency thru the system is a high priority because of >what kind of system it is. If you do manage to keep Twisted in the loop on this project, please share what strategy you used. It is unfortunately rare that Twisted is itself a really performance-critical component of any system (especially a highly parallel system), so information about optimization is hard to find. It isn't really a failing of Twisted, on non-sparc hardware it preforms awesome! The SPARC hardware is the problem, and the T1000 / T2000 we have be benchmarking just doesn't place nice with single threaded CPU bound apps. We are going to try and replace Twisted with Apache 2.2.x and create a module that does raw connection handliing, and go with the threaded mpm and see how that preforms with the same test client harness we are benchmarking with now.
On Fri, 2007-01-19 at 11:13 -0500, Jarrod Roberson wrote:
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote: It would be interesting to know where the CPU time is actually going.
The Twisted thread is eating up all the CPU time.
Have you tried the poll reactor?
On 1/19/07, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Fri, 2007-01-19 at 11:13 -0500, Jarrod Roberson wrote:
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote: It would be interesting to know where the CPU time is actually going.
The Twisted thread is eating up all the CPU time.
Have you tried the poll reactor?
That's what he said in the first message: "We are using Python 2.4.4 and Twisted 2.4.0 using the poll reactor in all cases." -bob
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote:
It would be interesting to know where the CPU time is actually going.
Sorry hit return by accident. Twisted is eating up all the CPU time. What is "slow" is the latency of the entire round trip thru the Twisted reactor loop. It is without a doubt Twisted that is the bottleneck.
On Fri, 19 Jan 2007 11:16:20 -0500, Jarrod Roberson <jarrod@vertigrated.com> wrote:
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote:
It would be interesting to know where the CPU time is actually going.
Sorry hit return by accident.
Twisted is eating up all the CPU time.
What is "slow" is the latency of the entire round trip thru the Twisted reactor loop. It is without a doubt Twisted that is the bottleneck.
Can you tell if most of the time is user or system? If system time is what dominates, it may be worthwhile to investigate /dev/poll. If most time is spent in userspace, though, you may be right about Twisted being the bottle neck. Jean-Paul
Yeah, it would be nice to see what the output of "vmstat 10" (or whatever the equivalent is under Solaris) was during the test. -- Jacob -----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Jean-Paul Calderone Sent: Friday, January 19, 2007 11:08 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Twisted server is 5 times SLOWER on SolaristhanLinux? On Fri, 19 Jan 2007 11:16:20 -0500, Jarrod Roberson <jarrod@vertigrated.com> wrote:
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote:
It would be interesting to know where the CPU time is actually
going.
Sorry hit return by accident.
Twisted is eating up all the CPU time.
What is "slow" is the latency of the entire round trip thru the Twisted reactor loop. It is without a doubt Twisted that is the bottleneck.
Can you tell if most of the time is user or system? If system time is what dominates, it may be worthwhile to investigate /dev/poll. If most time is spent in userspace, though, you may be right about Twisted being the bottle neck. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Saturday 20 January 2007 03:16, Jarrod Roberson wrote:
On 1/18/07, Jacob Gabrielson <jacob@cozi.com> wrote:
It would be interesting to know where the CPU time is actually going.
Sorry hit return by accident.
Twisted is eating up all the CPU time.
What is "slow" is the latency of the entire round trip thru the Twisted reactor loop. It is without a doubt Twisted that is the bottleneck.
Have you tried using cProfile to see where the time is going? I did this many years ago now with twisted when I first moved pythondirector across from asyncore to twisted, and found a couple of easily-fixed performance problems. It's quite possible you'll see a couple of easily-fixed optimisation targets if you profile the code. Anthony
On 1/18/07, glyph@divmod.com <glyph@divmod.com> wrote:
On 01:59 am, jarrod@vertigrated.com wrote:
We have LOTS and LOTS of test harness code and profiling code to pinpoint where bottlenecks are. We are going to have process a couple of terabytes a day thru this system. Latency thru the system is a high priority because of what kind of system it is.
If you do manage to keep Twisted in the loop on this project, please share what strategy you used. It is unfortunately rare that Twisted is itself a really performance-critical component of any system (especially a highly parallel system), so information about optimization is hard to find.
Here is an update. I have learned more about the Niagra chip in the T1000/T2000 servers. The T2000 we are using has a 8 core 1 Ghz processor, each core gets 4 threads, so Solaris sees a 32 processor machine. We noticed from watching top that each thread is only getting 3.1% of CPU time. Doing the math, that means our Twisted thread is running on a 250 mhz processor :-( The Niagra chip is basically like having 32 250mhz processors with a really effiecient scheduler. With that information in hand, we can now begin to understand how we need to approach deployment of our Twisted server in this environment. To get the kind of client counts ( 2000 clients ) we would have to run 20 instances of our Twisted server, with 100 clients each. We have tried this and it appears it will work, we need more time to test.
participants (7)
-
Anthony Baxter -
Bob Ippolito -
glyph＠divmod.com -
Itamar Shtull-Trauring -
Jacob Gabrielson -
Jarrod Roberson -
Jean-Paul Calderone