Simple thread pools
steve at holdenweb.com
Mon Nov 8 13:52:30 CET 2004
Josiah Carlson wrote:
> Jacob Friis <lists at debpro.webcom.dk> wrote:
>>I have built a script inspired by a post on Speno's Pythonic Avocado:
>>I'm setting NUM_FEEDERS to 1000.
>>Is that crazy?
> Not crazy, but foolish. Thread scheduling in Python reduces performance
> beyond a few dozen threads. If you are doing system calls (socket.recv,
> file.read, etc.), your performance will be poor.
Is this speculative, or do you have some hard evidence to support it? I
recently rewrote a billing program that delivers statements by email.
The number of threads it uses is a parameter to the program, and we are
currently running at 200 with every evidence of satisfaction - this
month's live run sent something over 10,000 emails an hour.
>>Are there a better solution?
> Fewer threads. Try running at 10-30. If you are finding that you
> aren't able to handle the load with those threads, then your
> processor/disk/etc isn't fast enough to handle the load.
I'm tempted to say "rubbish", but that would be rude, so instead I'll
just ask for some evidence :-). Don't forget that in network-based tasks
the time spent waiting for connection turnarounds can dominate the
elapsed time for execution - did you perhaps overlook that?
Holden Web LLC +1 800 494 3119
More information about the Python-list