What's the cost of using hundreds of threads?
aahz at pythoncraft.com
Tue Mar 1 23:21:46 CET 2005
In article <d0295h$hfc$1 at julia.coi.pw.edu.pl>,
<P.Rozycki at elka.pw.edu.pl> wrote:
>I have written some code, which creates many threads for each connection
>('main connection'). The purpose of this code is to balance the load
>between several connections ('pipes'). The number of spawned threads
>depends on how many pipes I create (= 2*n+2, where n is the number of
>For good results I'll presumably share main connection's load between 10
>pipes - therefore 22 threads will be spawned. Now if about 50
>connections are forwarded the number of threads rises to thousand of
>threads (or several thousands if even more connections are established).
I'm a bit confused by your math. Fifty connections should be 102
threads, which is quite reasonable.
>My questions are:
>- What is the cost (in memory / CPU usage) of creating such amounts of
>- Is there any 'upper boundary' that limits the number of threads? (is
>it python / OS related)
>- Is that the sign of 'clumsy programming' - i.e. Is creating so many
>threads a bad habit? (I must say that it simplified the solution of my
>problem very much).
>Limiting the number of threads is possible, but would affect the
>independence of data flows. (ok I admit - creating tricky algorithm
>could perhaps gurantee concurrency without spawning so many threads -
>but it's the simplest solution to this problem :) ).
My experience with lots of threads dates back to Python 1.5.2, but I
rarely saw much improvement with more than a hundred threads, even for
heavily I/O-bound applications on a multi-CPU system. However, if your
focus is algorithmic complexity, you should be able to handle a couple of
thousand threads easily enough.
Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/
"The joy of coding Python should be in seeing short, concise, readable
classes that express a lot of action in a small amount of clear code --
not in reams of trivial code that bores the reader to death." --GvR
More information about the Python-list