[Twisted-Python] twisted 30% slower than asyncore?
I kinda suspected that twisted might be slower than asyncore for the straight go-hell-for-leather usage in pythondirector, but I'd not got around to actually testing it. It seems like it's nearly 30% slower than the equivalent asyncore code! This seems a bit higher than I'd expect -- but given all the extra overhead of object creation I guess it's not all that surprising. Current CVS python, Twisted 1.0.4. Has anyone else done similar benchmarking and found different results? I'm not planning on switching the default back to asyncore - the asyncore code is just too brittle for my tastes. (FWIW, the two equivalent modules are http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/pythondirector/pythondirector... and http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/pythondirector/pythondirector... ) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Wed, 30 Apr 2003 18:01:05 +1000 Anthony Baxter <anthony@interlink.com.au> wrote:
I kinda suspected that twisted might be slower than asyncore for the straight go-hell-for-leather usage in pythondirector, but I'd not got around to actually testing it. It seems like it's nearly 30% slower than the equivalent asyncore code! This seems a bit higher than I'd expect -- but given all the extra overhead of object creation I guess it's not all that surprising. Current CVS python, Twisted 1.0.4.
We'll see what we can do to improve this (if we can, and when we have time :). -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting
Itamar Shtull-Trauring wrote We'll see what we can do to improve this (if we can, and when we have time :).
If it helps, I have slabs of hotshot data on this. The top 10 (from 20,000 small HTTP fetches: ncalls tottime percall cumtime percall filename:lineno(function) 20082 50.981 0.003 172.623 0.009 default.py:418(doSelect) 60024 26.018 0.000 26.018 0.000 tcp.py:116(writeSomeData) 40022 22.463 0.001 24.738 0.001 failure.py:91(__init__) 488147 21.022 0.000 24.747 0.000 <string>:1(fileno) 20010 4.160 0.000 4.932 0.000 tcp.py:209(createInternetSocket) 60024 3.486 0.000 6.361 0.000 tcp.py:97(doRead) 60030 2.795 0.000 2.795 0.000 default.py:489(addReader) 40020 2.789 0.000 3.782 0.000 tcp.py:90(__init__) 40020 2.746 0.000 47.347 0.001 tcp.py:232(doConnect) 20010 2.573 0.000 4.484 0.000 abstract.py:87(doWrite) That 'failure.py' is looking like some tasty low-hanging fruit to me, but doSelect should be amenable to some speedups as well... I should probably add that even though Twisted is slower than asyncore, pythondirector using Twisted is still 10-15% faster than the written-in-C "Pound" load balancer, and about 5% slower than the "balance" load balancer (also in C). Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Thu, May 01, 2003 at 03:18:58PM +1000, Anthony Baxter wrote:
Itamar Shtull-Trauring wrote We'll see what we can do to improve this (if we can, and when we have time :).
If it helps, I have slabs of hotshot data on this. The top 10 (from 20,000 small HTTP fetches:
ncalls tottime percall cumtime percall filename:lineno(function) 20082 50.981 0.003 172.623 0.009 default.py:418(doSelect) 60024 26.018 0.000 26.018 0.000 tcp.py:116(writeSomeData) 40022 22.463 0.001 24.738 0.001 failure.py:91(__init__) 488147 21.022 0.000 24.747 0.000 <string>:1(fileno) 20010 4.160 0.000 4.932 0.000 tcp.py:209(createInternetSocket) 60024 3.486 0.000 6.361 0.000 tcp.py:97(doRead) 60030 2.795 0.000 2.795 0.000 default.py:489(addReader) 40020 2.789 0.000 3.782 0.000 tcp.py:90(__init__) 40020 2.746 0.000 47.347 0.001 tcp.py:232(doConnect) 20010 2.573 0.000 4.484 0.000 abstract.py:87(doWrite)
That 'failure.py' is looking like some tasty low-hanging fruit to me, but doSelect should be amenable to some speedups as well...
Boy would it be great if Failures could be sped up some more, but it doesn't seem likely without some mind-bending rework of the whole design. I think most, if not all, of the worthwhile local optimizations have been made on it. What I learned from Twisted.names is to not write code that uses Failures anywhere except where they are absolutely necessary. Python exceptions might be cheap enough to use as a kind of fancy flow control, but failures definitely are not.
I should probably add that even though Twisted is slower than asyncore, pythondirector using Twisted is still 10-15% faster than the written-in-C "Pound" load balancer, and about 5% slower than the "balance" load balancer (also in C).
Cool beans. Jp -- "There is no reason for any individual to have a computer in their home." -- Ken Olson, President of DEC, World Future Society Convention, 1977 -- up 42 days, 1:04, 5 users, load average: 0.00, 0.02, 0.02
On Thu, 1 May 2003 01:29:53 -0400 Jp Calderone <exarkun@intarweb.us> wrote:
What I learned from Twisted.names is to not write code that uses Failures anywhere except where they are absolutely necessary. Python exceptions might be cheap enough to use as a kind of fancy flow control, but failures definitely are not.
My soon to be checked in code changes Failures to be vastly less expensive if you explicitly pass in an Exception (no pretty tracebacks stored, but in this case you don't want them anyway.) -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting
On Thu, 01 May 2003 15:18:58 +1000 Anthony Baxter <anthony@interlink.com.au> wrote:
That 'failure.py' is looking like some tasty low-hanging fruit to me,
Indeed. Thanks for the info! -- Itamar Shtull-Trauring http://itamarst.org/ http://www.zoteca.com -- Python & Twisted consulting
participants (3)
-
Anthony Baxter -
Itamar Shtull-Trauring -
Jp Calderone