[Twisted-Python] Twisted Performance
Hi, I am new to twisted and have been having trouble finding out information about twisted's performance. I have a fairly simple setup where I need to open a bunch of TCP connections that last for varying amounts of time but dont do much. I have tried using threads(got GILed to death) and Processes(even worse). Now I am looking at either making a system to start the connection and send info to have the remote point "phone home" when its done, then closing the connection or using something like Twisted. my socket conversation: my app -> send a message that triggers an action on the other end other end -> recv's message does action(can take any amount of time) other end - > sends results back to my app Can twisted handle up to several hundred connections like this? Is there a better approach? Is there anything I should avoid? Thanks, Dan
On Oct 13, 2009, at 10:44 PM, Daniel Griffin wrote:
Hi,
I am new to twisted and have been having trouble finding out information about twisted's performance. I have a fairly simple setup where I need to open a bunch of TCP connections that last for varying amounts of time but dont do much. I have tried using threads (got GILed to death) and Processes(even worse). Now I am looking at either making a system to start the connection and send info to have the remote point "phone home" when its done, then closing the connection or using something like Twisted.
my socket conversation: my app -> send a message that triggers an action on the other end other end -> recv's message does action(can take any amount of time) other end - > sends results back to my app
Twisted Documentation: Writing Clients I would suggest deferring worrying at this point. (;-b). Twisted can almost certainly handle it. Do the simplest thing possible, see how it performs, then worry as necessary. S
On Wed, Oct 14, 2009 at 4:44 AM, Daniel Griffin <dgriff1@gmail.com> wrote:
Hi,
I am new to twisted and have been having trouble finding out information about twisted's performance. I have a fairly simple setup where I need to open a bunch of TCP connections that last for varying amounts of time but dont do much. I have tried using threads(got GILed to death) and Processes(even worse). Now I am looking at either making a system to start the connection and send info to have the remote point "phone home" when its done, then closing the connection or using something like Twisted.
my socket conversation: my app -> send a message that triggers an action on the other end other end -> recv's message does action(can take any amount of time) other end - > sends results back to my app
Sure, that's a blueprint for pretty much every Twisted app. You just need to make the thing that takes a long time not block the reactor. How you do that mostly depends on what the "long thing" is. Usually this means using an existing library, sometimes it means writing your own, and for some unfortunate cases it means deferring to a thread. (Note that deferring to a thread to fake non-blocking IO is not anywhere near as bad as doing actual _work_ in threads ;-)) Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327 And you should probably read the finger tutorial and the deferreds tutorial :-)
Can twisted handle up to several hundred connections like this? Is there a better approach? Is there anything I should avoid?
Depends how much work you're doing in the Twisted process or on the same box, of course. You're probably looking at several thousand rather than several hundred, depending on how much work needs to be done in each one.
Thanks,
Dan
On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327
Very nice, thanks for sharing! S
Sounds good, I am going to try to get this up and running over the next few days. I'll get back with some results. Dan On Wed, Oct 14, 2009 at 7:14 AM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327
Very nice, thanks for sharing!
S
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Steve Steiner (listsin) wrote:
On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327
Very nice, thanks for sharing!
Thank you. I really meant to make a public announcement about that on this list, but I was trying to get to Deferreds first :) Anyway...I'm writing an Introduction to Twisted and asynchronous programming. Surprise :) dave
Cool, please let me know off-list if you'd like a quick review before "going live." S On Oct 14, 2009, at 9:30 PM, Dave Peticolas wrote:
Steve Steiner (listsin) wrote:
On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327
Very nice, thanks for sharing!
Thank you. I really meant to make a public announcement about that on this list, but I was trying to get to Deferreds first :)
Anyway...I'm writing an Introduction to Twisted and asynchronous programming. Surprise :)
dave
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi everyone, I ported my thread based socket code to twisted and have been happy with my results. To re-hash, my code creates a connection then waits for the other end to do something and respond, each connection takes a different amount of time. I went from 3-5 connections completed per second to about 15 per second. Most importantly connections that take a long time have almost no impact on other running connections. The thread stuff would peg the CPU and now I never really venture over 20%. Also testing this code is far easier than testing stuff that directly calls sockets. Basically its a huge win on every level. Thanks for all the help everyone. Dan On Wed, Oct 14, 2009 at 8:47 PM, Steve Steiner (listsin) < listsin@integrateddevcorp.com> wrote:
Cool, please let me know off-list if you'd like a quick review before "going live."
S
On Oct 14, 2009, at 9:30 PM, Dave Peticolas wrote:
Steve Steiner (listsin) wrote:
On Oct 14, 2009, at 4:55 AM, Laurens Van Houtven wrote:
Personally I'm a big fan of this series of blog posts: http://krondo.com/blog/?page_id=1327
Very nice, thanks for sharing!
Thank you. I really meant to make a public announcement about that on this list, but I was trying to get to Deferreds first :)
Anyway...I'm writing an Introduction to Twisted and asynchronous programming. Surprise :)
dave
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 24, 2009, at 3:47 PM, Daniel Griffin wrote:
I ported my thread based socket code to twisted and have been happy with my results. (...) Basically its a huge win on every level.
Thanks for all the help everyone.
Glad to hear it! Do you want to write up something more specific for the SuccessStories page?
Sure, I am not really finished with it yet. When I do get done I will do a write up. Dan On Sun, Oct 25, 2009 at 12:15 AM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Oct 24, 2009, at 3:47 PM, Daniel Griffin wrote:
I ported my thread based socket code to twisted and have been happy with my results. (...) Basically its a huge win on every level.
Thanks for all the help everyone.
Glad to hear it! Do you want to write up something more specific for the SuccessStories page?
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Daniel Griffin -
Dave Peticolas -
Glyph Lefkowitz -
Laurens Van Houtven -
Steve Steiner (listsin)