[Twisted-Python] What kind of throughput can we expect to achieve when using DatagramProtocol (UDP)
Hi, We have an application we intend to release where we intend to have about a million applications sending UDP packets to a Twisted Python server which needs to process incoming data (Approx 128 bytes of text) per packet, which a BACK end system is inserting this data into mySQL tables. Has anyone done some serious "stress testing" of Twisted Python simple server code to see just how much data it can digest at a time. Our server will no doubt be hosted on an OC3 - capable of 150 megabits of throughput with approx a 75 - 80% load. We need to know how many servers we need to put into some kind of a load sharing cluster to be able to handle this very high data before it chokes. Right now, in our proof of performance, we are using UDP, but planning to move to the more reliable TCPIP protocol when we get into production. These data requests will come in fast and violent spurts of data. John
One data point: I have a server that processes about 20 UDP multicast packets per second. This server does many other things, but the load from those 20 UDP pps is negligible- it uses 0% CPU with just that (Xeon L5420)
On Mon, 2009-03-30 at 15:07 -0700, John Draper wrote:
In these sort of applications, batching helps a lot: less function calls, less I/O overhead, etc.. So TCP should likely *speed* things up, if messages come grouped in spurts. Benchmarks done with inserting one MySQL row at a time will be far slower than batch inserts.
One data point: I have a server that processes about 20 UDP multicast packets per second. This server does many other things, but the load from those 20 UDP pps is negligible- it uses 0% CPU with just that (Xeon L5420)
On Mon, 2009-03-30 at 15:07 -0700, John Draper wrote:
In these sort of applications, batching helps a lot: less function calls, less I/O overhead, etc.. So TCP should likely *speed* things up, if messages come grouped in spurts. Benchmarks done with inserting one MySQL row at a time will be far slower than batch inserts.
participants (3)
-
Alec Matusis -
Itamar Shtull-Trauring -
John Draper