[Twisted-Python] How to improve response time of twisted.web?
Hi all, I decided to benchmark my web server written in twisted against apache. I added just one resource to my twisted web server that reads a file and returns it to client. Then, I created a virtual host in Apache and put the same (html) file to the web root. I issued the following command for both servers: # http_load -parallel 100 seconds 5 url First time the url contained the path to Apache document, second time -- to twisted web. Here is the result of the benchmark. My web server is more than 3 times slower than apache, primarily due to response time as you can see below: RESULT FOR APACHE ================== 8619 fetches, 13 max parallel, 2.43918e+07 bytes, in 5.00155 seconds 2830 mean bytes/connection 1723.27 fetches/sec, 4.87684e+06 bytes/sec msecs/connect: 0.557627 mean, 2.159 max, 0.123 min msecs/first-response: 2.81528 mean, 7.367 max, 0.413 min HTTP response codes: code 200 -- 8619 RESULT FOR MY TWISTED WEB SERVER ================================= 2437 fetches, 100 max parallel, 6.86601e+06 bytes, in 5.00001 seconds 2817.4 mean bytes/connection 487.399 fetches/sec, 1.3732e+06 bytes/sec msecs/connect: 0.159532 mean, 9.453 max, 0.077 min msecs/first-response: 197.407 mean, 230.882 max, 5.978 min HTTP response codes: code 200 -- 2437 It's not that I want to outperform Apache, I'm just curious is there any way to improve the response time in twisted? Thanks in advance. -- Regards, Pavel
* Pavel <pbastov@gmail.com> [2008-03-26 19:30:41 +0600]:
I added just one resource to my twisted web server that reads a file and returns it to client.
Did you write your own resource to do this, or did you use twisted.web.static? If you wrote your own resource, it might be helpful if you posted the source code for this as well. -- mithrandi, i Ainil en-Balandor, a faer Ambar
Yes, I wrote my own resource. Looks like this: class MyRoot(Resource): def getChild(self, name, request): if "" == name: return self return Resource def render_GET(self, request): path = "/path/to/file" f = open(path) content = f.read() f.close() return content On Wed, Mar 26, 2008 at 7:47 PM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
* Pavel <pbastov@gmail.com> [2008-03-26 19:30:41 +0600]:
I added just one resource to my twisted web server that reads a file and returns it to client.
Did you write your own resource to do this, or did you use twisted.web.static? If you wrote your own resource, it might be helpful if you posted the source code for this as well. -- mithrandi, i Ainil en-Balandor, a faer Ambar
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (GNU/Linux)
iEYEARECAAYFAkfqU/EACgkQpNuXDQIV94oTCQCfd5Se70QCW+/XYvWFOY5UOGr8 A4QAmwbAVWa2HCyRumq5zJdnF78tFXvi =Gd/m -----END PGP SIGNATURE-----
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Good luck, Pavel Bastov xooChat Team Leader http://www.xoochat.com/
On Wed, Mar 26, 2008 at 10:53 PM, Pavel <pbastov@gmail.com> wrote:
Yes, I wrote my own resource. Looks like this:
class MyRoot(Resource): def getChild(self, name, request): if "" == name: return self return Resource
def render_GET(self, request): path = "/path/to/file" f = open(path) content = f.read() f.close() return content
You should probably be using twisted.web.static.File for a better comparison of static file serving performance. In fact, you could do this without writing any code at all: $ twistd web --path=<dir> Cheers, -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
Thanks for the replies. In fact, I just tried to compare it against the static files -- almost the same result -- a little bit faster, but far from Apache :( On Thu, Mar 27, 2008 at 10:26 PM, Drew Smathers <drew.smathers@gmail.com> wrote:
On Wed, Mar 26, 2008 at 10:53 PM, Pavel <pbastov@gmail.com> wrote:
Yes, I wrote my own resource. Looks like this:
class MyRoot(Resource): def getChild(self, name, request): if "" == name: return self return Resource
def render_GET(self, request): path = "/path/to/file" f = open(path) content = f.read() f.close() return content
You should probably be using twisted.web.static.File for a better comparison of static file serving performance. In fact, you could do this without writing any code at all:
$ twistd web --path=<dir>
Cheers, -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Good luck, Pavel Bastov xooChat Team Leader http://www.xoochat.com/
On 03:57 am, pbastov@gmail.com wrote:
Thanks for the replies.
In fact, I just tried to compare it against the static files -- almost the same result -- a little bit faster, but far from Apache :(
Twisted is not, and has never been, as fast as Apache in this kind of naive raw-speed comparison on static files. It's probably not as fast for most other things either. If you would like to make it faster, there's no trick or configuration option or tweak. The solution is simple: optimize the code. This may just involve using a profiler and submitting patches, this may involve using some tool like cython or rpython to put the really critical parts into an extension module. Nobody else has volunteered to do this so far because Twisted is, in fact, fast enough for most things :).
Yes, I see what you mean. I really love Twisted, and will never switch to Apache or the like, since I like its simplicity and the ability to have all my code in one place. On Fri, Mar 28, 2008 at 12:27 PM, <glyph@divmod.com> wrote:
On 03:57 am, pbastov@gmail.com wrote:
Thanks for the replies.
In fact, I just tried to compare it against the static files -- almost the same result -- a little bit faster, but far from Apache :(
Twisted is not, and has never been, as fast as Apache in this kind of naive raw-speed comparison on static files. It's probably not as fast for most other things either.
If you would like to make it faster, there's no trick or configuration option or tweak. The solution is simple: optimize the code. This may just involve using a profiler and submitting patches, this may involve using some tool like cython or rpython to put the really critical parts into an extension module. Nobody else has volunteered to do this so far because Twisted is, in fact, fast enough for most things :).
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Good luck, Pavel Bastov xooChat Team Leader http://www.xoochat.com/
On Fri, Mar 28, 2008 at 12:12 AM, Pavel <pbastov@gmail.com> wrote:
Yes, I see what you mean.
I really love Twisted, and will never switch to Apache or the like, since I like its simplicity and the ability to have all my code in one place.
Forgive me if the following is not what you are looking for but, I would recommend a setup like this: Use a fast and light server/reverse proxy like nginx ( http://wiki.codemongers.com/Main ) to 1) handle all static files (img,js,css, static html) 2) reverse proxy all your dynamic twisted.web part I use this setup, and it works great with Twisted. -Alex
On Fri, Mar 28, 2008 at 12:27 PM, <glyph@divmod.com> wrote:
On 03:57 am, pbastov@gmail.com wrote:
Thanks for the replies.
In fact, I just tried to compare it against the static files -- almost the same result -- a little bit faster, but far from Apache :(
Twisted is not, and has never been, as fast as Apache in this kind of naive raw-speed comparison on static files. It's probably not as fast for most other things either.
If you would like to make it faster, there's no trick or configuration option or tweak. The solution is simple: optimize the code. This may just involve using a profiler and submitting patches, this may involve using some tool like cython or rpython to put the really critical parts into an extension module. Nobody else has volunteered to do this so
far
because Twisted is, in fact, fast enough for most things :).
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Good luck,
Pavel Bastov xooChat Team Leader http://www.xoochat.com/
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Forgive me if the following is not what you are looking for but, I would recommend a setup like this:
Use a fast and light server/reverse proxy like nginx ( http://wiki.codemongers.com/Main ) to
1) handle all static files (img,js,css, static html) 2) reverse proxy all your dynamic twisted.web part
I use this setup, and it works great with Twisted.
-Alex
Greetings Alex, thanks for your advice. I'm aware of this configuration, and it seems to be a pretty reasonable one.
Quoting Pavel <pbastov@gmail.com>:
Thanks for the replies.
In fact, I just tried to compare it against the static files -- almost the same result -- a little bit faster, but far from Apache :(
There are 2 things I'll try if speed is a problem: * test with sendfile support. Look at http://twistedmatrix.com/trac/ticket/585, there is the branch sendfile-585-2, which should work with Linux and FreeBSD. If someone could provide some benchmark numbers, if it would be great. * if you have a multicore server, you'll probably want to make use of it. There's nothing in Twisted currently to easily do that, but I have tested some months ago the possibility to add prefork facility to Twisted. See http://twistedmatrix.com/trac/browser/sandbox/therve/prefork_ex1.py for an example. It's highly experimental, and there's currently no ticket talking about that, but some feedback would be welcome. -- Thomas
participants (6)
-
alex clemesha -
Drew Smathers -
glyph＠divmod.com -
Pavel -
Thomas Hervé -
Tristan Seligmann