[Twisted-Python] Speed of twisted on Amazon EC2
Hi all I recently wrote a hello world HTTP app in Twisted, Cyclone, node, Go and Scala on the Play framework, and used a combination of ab and gattling to see how fast each was on an m1.small instance on Amazon EC2. I also tested it on bare metal (my laptop:)) but I'll get to that later. The code I used for twisted was the web server code sample from the front page of twistedmatrix.com with the numberRequests increment dropped and a simple "Hello World!" returned instead. For cyclone I used the sample code from the cyclone docs with the same string being returned. Node was the sample from their front page. Play and Go I've never worked with before so those were copy pastes from various examples on their pages but again same URI and return string as the other frameworks.
From one m1.small to another, Go, Play and node were handling about 1.5k requests per second while Twisted and Cyclone were only handling 400-500 requests per second.
On loopback from the same m1.small I suspect I was starting to load test the load testers more than the apps but locally all the frameworks were performing similarly. Load testing from another machine to my laptop I was seeing far better performance from Twisted and Cyclone, both doing closer to 2k requests per second. The other frameworks were only a little quicker but again, there's a chance the machine running the load tests was the bottle neck. These tests were very unscientifically done but I still wasn't expecting to see such a big difference between specifically node and Twisted. 500 req/sec is plenty quick for 90% of what I do and of course I know there's so much more to an application than returning hello world but I was wondering if I was missing some trick to speed Twisted up to get close to the others on EC2.
Tom, which OS are you running on your EC2 instance? Amazon Linux? Which reactor are you running your Twisted test with? Epoll? Are you running on CPython or PyPy? Can we see you Twisted hello world test app? Tobias Mit freundlichen Grüßen, Tobias Oberstein Geschäftsführer +49(0)9131 940 3574 +49(0)176 23 752 055 tobias.oberstein@tavendo.de Skype: toberstein Twitter: @oberstet Xing: Tobias Oberstein Tavendo GmbH · Am Weihersbach 13 · 91074 Herzogenaurach · Deutschland Office / Postanschrift : Waldstrasse 18 · 91054 Erlangen · Deutschland +49(+)700 GO TAVENDO · +49(0)700 46 828 36 36 · www.tavendo.de Amtsgericht Fürth · HRB 10365 · USt-Id.Nr DE246462916 · Geschäftsführer Tobias Oberstein Von: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] Im Auftrag von Tom van Neerijnen Gesendet: Sonntag, 21. April 2013 12:58 An: twisted-python@twistedmatrix.com Betreff: [Twisted-Python] Speed of twisted on Amazon EC2 Hi all I recently wrote a hello world HTTP app in Twisted, Cyclone, node, Go and Scala on the Play framework, and used a combination of ab and gattling to see how fast each was on an m1.small instance on Amazon EC2. I also tested it on bare metal (my laptop:)) but I'll get to that later. The code I used for twisted was the web server code sample from the front page of twistedmatrix.com with the numberRequests increment dropped and a simple "Hello World!" returned instead. For cyclone I used the sample code from the cyclone docs with the same string being returned. Node was the sample from their front page. Play and Go I've never worked with before so those were copy pastes from various examples on their pages but again same URI and return string as the other frameworks.
From one m1.small to another, Go, Play and node were handling about 1.5k requests per second while Twisted and Cyclone were only handling 400-500 requests per second. On loopback from the same m1.small I suspect I was starting to load test the load testers more than the apps but locally all the frameworks were performing similarly. Load testing from another machine to my laptop I was seeing far better performance from Twisted and Cyclone, both doing closer to 2k requests per second. The other frameworks were only a little quicker but again, there's a chance the machine running the load tests was the bottle neck. These tests were very unscientifically done but I still wasn't expecting to see such a big difference between specifically node and Twisted. 500 req/sec is plenty quick for 90% of what I do and of course I know there's so much more to an application than returning hello world but I was wondering if I was missing some trick to speed Twisted up to get close to the others on EC2.
On Sun, Apr 21, 2013 at 8:36 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Tom,
which OS are you running on your EC2 instance? Amazon Linux?
Which reactor are you running your Twisted test with? Epoll?
Are you running on CPython or PyPy?
Can we see you Twisted hello world test app?
Tobias
Mit freundlichen Grüßen,
Tobias Oberstein Geschäftsführer
+49(0)9131 940 3574 +49(0)176 23 752 055 tobias.oberstein@tavendo.de Skype: toberstein Twitter: @oberstet Xing: Tobias Oberstein
Tavendo GmbH · Am Weihersbach 13 · 91074 Herzogenaurach · Deutschland Office / Postanschrift : Waldstrasse 18 · 91054 Erlangen · Deutschland +49(+)700 GO TAVENDO · +49(0)700 46 828 36 36 · www.tavendo.de Amtsgericht Fürth · HRB 10365 · USt-Id.Nr DE246462916 · Geschäftsführer Tobias Oberstein
Von: twisted-python-bounces@twistedmatrix.com [mailto: twisted-python-bounces@twistedmatrix.com] Im Auftrag von Tom van Neerijnen Gesendet: Sonntag, 21. April 2013 12:58 An: twisted-python@twistedmatrix.com Betreff: [Twisted-Python] Speed of twisted on Amazon EC2
Hi all I recently wrote a hello world HTTP app in Twisted, Cyclone, node, Go and Scala on the Play framework, and used a combination of ab and gattling to see how fast each was on an m1.small instance on Amazon EC2. I also tested it on bare metal (my laptop:)) but I'll get to that later. The code I used for twisted was the web server code sample from the front page of twistedmatrix.com with the numberRequests increment dropped and a simple "Hello World!" returned instead. For cyclone I used the sample code from the cyclone docs with the same string being returned. Node was the sample from their front page. Play and Go I've never worked with before so those were copy pastes from various examples on their pages but again same URI and return string as the other frameworks.
From one m1.small to another, Go, Play and node were handling about 1.5k requests per second while Twisted and Cyclone were only handling 400-500 requests per second. On loopback from the same m1.small I suspect I was starting to load test the load testers more than the apps but locally all the frameworks were performing similarly. Load testing from another machine to my laptop I was seeing far better performance from Twisted and Cyclone, both doing closer to 2k requests per second. The other frameworks were only a little quicker but again, there's a chance the machine running the load tests was the bottle neck. These tests were very unscientifically done but I still wasn't expecting to see such a big difference between specifically node and Twisted. 500 req/sec is plenty quick for 90% of what I do and of course I know there's so much more to an application than returning hello world but I was wondering if I was missing some trick to speed Twisted up to get close to the others on EC2.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi I suggest trying pypy. python actually is quite slow with lots of function calls. I did have a benchmark for cyclone & pypy. it is almost 3 times faster. twisted can work with pypy well so you definitely should give it a try. Regards gelin yan
Thank you Gelin, as you suggested I benchmarked again on PyPy and am very impressed with it. It was significantly quicker, beating node by a fair margin on a higher concurrency. I pasted some results in another reply in this same thread. On Sun, Apr 21, 2013 at 4:26 PM, Gelin Yan <dynamicgl@gmail.com> wrote:
On Sun, Apr 21, 2013 at 8:36 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Tom,
which OS are you running on your EC2 instance? Amazon Linux?
Which reactor are you running your Twisted test with? Epoll?
Are you running on CPython or PyPy?
Can we see you Twisted hello world test app?
Tobias
Mit freundlichen Grüßen,
Tobias Oberstein Geschäftsführer
+49(0)9131 940 3574 +49(0)176 23 752 055 tobias.oberstein@tavendo.de Skype: toberstein Twitter: @oberstet Xing: Tobias Oberstein
Tavendo GmbH · Am Weihersbach 13 · 91074 Herzogenaurach · Deutschland Office / Postanschrift : Waldstrasse 18 · 91054 Erlangen · Deutschland +49(+)700 GO TAVENDO · +49(0)700 46 828 36 36 · www.tavendo.de Amtsgericht Fürth · HRB 10365 · USt-Id.Nr DE246462916 · Geschäftsführer Tobias Oberstein
Von: twisted-python-bounces@twistedmatrix.com [mailto: twisted-python-bounces@twistedmatrix.com] Im Auftrag von Tom van Neerijnen Gesendet: Sonntag, 21. April 2013 12:58 An: twisted-python@twistedmatrix.com Betreff: [Twisted-Python] Speed of twisted on Amazon EC2
Hi all I recently wrote a hello world HTTP app in Twisted, Cyclone, node, Go and Scala on the Play framework, and used a combination of ab and gattling to see how fast each was on an m1.small instance on Amazon EC2. I also tested it on bare metal (my laptop:)) but I'll get to that later. The code I used for twisted was the web server code sample from the front page of twistedmatrix.com with the numberRequests increment dropped and a simple "Hello World!" returned instead. For cyclone I used the sample code from the cyclone docs with the same string being returned. Node was the sample from their front page. Play and Go I've never worked with before so those were copy pastes from various examples on their pages but again same URI and return string as the other frameworks.
From one m1.small to another, Go, Play and node were handling about 1.5k requests per second while Twisted and Cyclone were only handling 400-500 requests per second. On loopback from the same m1.small I suspect I was starting to load test the load testers more than the apps but locally all the frameworks were performing similarly. Load testing from another machine to my laptop I was seeing far better performance from Twisted and Cyclone, both doing closer to 2k requests per second. The other frameworks were only a little quicker but again, there's a chance the machine running the load tests was the bottle neck. These tests were very unscientifically done but I still wasn't expecting to see such a big difference between specifically node and Twisted. 500 req/sec is plenty quick for 90% of what I do and of course I know there's so much more to an application than returning hello world but I was wondering if I was missing some trick to speed Twisted up to get close to the others on EC2.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi
I suggest trying pypy. python actually is quite slow with lots of function calls. I did have a benchmark for cyclone & pypy. it is almost 3 times faster. twisted can work with pypy well so you definitely should give it a try.
Regards
gelin yan
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi Tobias My apologies, I should have included that information. Before I get into my responses you've alluded to the answer already, which is to say I should have been running on PyPy in the first place. Twisted on PyPy is, at least for this really pointless benchmark, faster than node. I've re-run my tests using virtualenvs ( I was using Ubuntu packages which may have been behind the latest releases), taken more care to record the machine states, added PyPy to the mix and am responding inline with my answers. On Sun, Apr 21, 2013 at 1:36 PM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Tom,
which OS are you running on your EC2 instance? Amazon Linux?
I am running two m1.small Ubuntu 12.10 64bit in us-east-1, one for the web server and the other for the ab.
Which reactor are you running your Twisted test with? Epoll?
Default Linux reactor, so it's Epoll.
Are you running on CPython or PyPy?
I am running on CPython with twisted 13.0.0 installed into a virtualenv, but please see my results for PyPy below, they're far more impressive.
Can we see you Twisted hello world test app?
Sure: from twisted.web import server, resource from twisted.internet import reactor class HelloResource(resource.Resource): isLeaf = True def render_GET(self, request): request.setHeader("content-type", "text/plain") return "Hello world!\n" reactor.listenTCP(8080, server.Site(HelloResource())) reactor.run() And for comparison the node code: var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello world!\n'); }).listen(8080, '0.0.0.0'); As I said you've already answered my question but here are my slightly more controlled benchmarks from an ab -c 10 -n 100000 (I ran three and used the average of the last two to ensure a warm JIT). This time Twisted 13.0.0 in a CPython virtualenv yielded around 700 req/s. Twisted 13.0.0 in a PyPy virtualenv yielded around 1.6k req/s. Node 0.6.19 yielded around 1.6k req/s. And for giggles 3x the concurrency with ab -c 30 -n 100000, which shows Twisted on PyPy pulling ahead: Twisted 13.0.0 in a PyPy virtualenv yielded around 2k req/s. Node 0.6.19 yielded around 1.6k req/s.
Tobias
Mit freundlichen Grüßen,
Tobias Oberstein Geschäftsführer
+49(0)9131 940 3574 +49(0)176 23 752 055 tobias.oberstein@tavendo.de Skype: toberstein Twitter: @oberstet Xing: Tobias Oberstein
Tavendo GmbH · Am Weihersbach 13 · 91074 Herzogenaurach · Deutschland Office / Postanschrift : Waldstrasse 18 · 91054 Erlangen · Deutschland +49(+)700 GO TAVENDO · +49(0)700 46 828 36 36 · www.tavendo.de Amtsgericht Fürth · HRB 10365 · USt-Id.Nr DE246462916 · Geschäftsführer Tobias Oberstein
Von: twisted-python-bounces@twistedmatrix.com [mailto: twisted-python-bounces@twistedmatrix.com] Im Auftrag von Tom van Neerijnen Gesendet: Sonntag, 21. April 2013 12:58 An: twisted-python@twistedmatrix.com Betreff: [Twisted-Python] Speed of twisted on Amazon EC2
Hi all I recently wrote a hello world HTTP app in Twisted, Cyclone, node, Go and Scala on the Play framework, and used a combination of ab and gattling to see how fast each was on an m1.small instance on Amazon EC2. I also tested it on bare metal (my laptop:)) but I'll get to that later. The code I used for twisted was the web server code sample from the front page of twistedmatrix.com with the numberRequests increment dropped and a simple "Hello World!" returned instead. For cyclone I used the sample code from the cyclone docs with the same string being returned. Node was the sample from their front page. Play and Go I've never worked with before so those were copy pastes from various examples on their pages but again same URI and return string as the other frameworks.
From one m1.small to another, Go, Play and node were handling about 1.5k requests per second while Twisted and Cyclone were only handling 400-500 requests per second. On loopback from the same m1.small I suspect I was starting to load test the load testers more than the apps but locally all the frameworks were performing similarly. Load testing from another machine to my laptop I was seeing far better performance from Twisted and Cyclone, both doing closer to 2k requests per second. The other frameworks were only a little quicker but again, there's a chance the machine running the load tests was the bottle neck. These tests were very unscientifically done but I still wasn't expecting to see such a big difference between specifically node and Twisted. 500 req/sec is plenty quick for 90% of what I do and of course I know there's so much more to an application than returning hello world but I was wondering if I was missing some trick to speed Twisted up to get close to the others on EC2.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Tom,
which OS are you running on your EC2 instance? Amazon Linux? I am running two m1.small Ubuntu 12.10 64bit in us-east-1, one for the web server and the other for the ab. Which reactor are you running your Twisted test with? Epoll? Default Linux reactor, so it's Epoll.
ok, thats good. just wanted to make sure you are aware of this and make sure epoll is actually used.
Are you running on CPython or PyPy? I am running on CPython with twisted 13.0.0 installed into a virtualenv, but please see my results for PyPy below, they're far more impressive.
thats also good (and I was somehow expecting this .. at least thats also my experience). there is 1 thing to note when benchmarking networking stuff on PyPy vs CPython that I noticed in my testing: both the throughput (bits/s or reqs/s) as well as the average (arithmetic mean) response times are far better with PyPy vs CPy BUT (my testings - Twisted/Autobahn/WebSocket): e.g. the 95% quantile (and the worst case) response latency of PyPy is worse than CPy. this was PyPy 1.6 on both Linux (epoll) and FreeBSD (kqueue). the reason might be GC .. PyPy's GCs might be not as sophisticated as CPy's (likely) or V8's (definitely). note: this might have changed with PyPy 2 (did not test that). Armin at least was planning for a generational/incremental GC in PyPy. dunno. so: you might wanna test for latency also .. I'd be interested in results. regarding GC, V8 and Erlang (and possibly that other VMs with the J at the beginning) have an edge .. both LuaJIT and PyPy plan to improve on this though .. Tobias
Hi Tobias Some comments inline but first an update on the next load test slightly closer to my ultimate use case, which showed PyPy smoking the competition. My test is now a request to a URL scheme like this: /version/{app}/platform/{platform}. This gets translated into a get on Redis for {app}-{platform} and the value is returned to the client. All tests were on Ubuntu 12.10 on an m1.small instance on AWS with the load test machine being an m1.large. The load test ran for about 10 minutes for each framework. Using Cyclone 1.1 on Twisted 13 on PyPy 2.0beta2 I was averaging about 1.8k r/s. Using node-restify 2.4.1 and node-redis 0.8.3 node 0.10.4 I was doing about 1.2 r/s. Go and Scala on Play were comparable to node. This very simple load test has me very impressed with PyPys performance and I look forward to load testing our full app when it's ready. Thanks for the comments, they've been very helpful. On Mon, Apr 22, 2013 at 8:12 AM, Tobias Oberstein < tobias.oberstein@tavendo.de> wrote:
Tom,
which OS are you running on your EC2 instance? Amazon Linux? I am running two m1.small Ubuntu 12.10 64bit in us-east-1, one for the web server and the other for the ab. Which reactor are you running your Twisted test with? Epoll? Default Linux reactor, so it's Epoll.
ok, thats good. just wanted to make sure you are aware of this and make sure epoll is actually used.
Are you running on CPython or PyPy? I am running on CPython with twisted 13.0.0 installed into a virtualenv, but please see my results for PyPy below, they're far more impressive.
thats also good (and I was somehow expecting this .. at least thats also my experience).
there is 1 thing to note when benchmarking networking stuff on PyPy vs CPython that I noticed in my testing:
both the throughput (bits/s or reqs/s) as well as the average (arithmetic mean) response times are far better with PyPy vs CPy
BUT (my testings - Twisted/Autobahn/WebSocket): e.g. the 95% quantile (and the worst case) response latency of PyPy is worse than CPy.
this was PyPy 1.6 on both Linux (epoll) and FreeBSD (kqueue).
the reason might be GC .. PyPy's GCs might be not as sophisticated as CPy's (likely) or V8's (definitely).
note: this might have changed with PyPy 2 (did not test that). Armin at least was planning for a generational/incremental GC in PyPy. dunno.
so: you might wanna test for latency also .. I'd be interested in results.
PyPy 2.0beta2 had comparable 95 percentile latency to all the other frameworks in my limited test so maybe that's the improvements you mentioned? Or maybe my naive load tests, but I like to think it's the improvements.
regarding GC, V8 and Erlang (and possibly that other VMs with the J at the beginning) have an edge ..
both LuaJIT and PyPy plan to improve on this though ..
I don't know much about the workings of the various GCs but my feeling is I wasn't really stressing the GC with my limited test as there weren't very many data structures in play. That said over 10 minute load tests I didn't notice any blips in r/s or response times so for my limited test it was all good.
Tobias
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (3)
-
Gelin Yan -
Tobias Oberstein -
Tom van Neerijnen