What's the cost of using hundreds of threads?

[wes weston]

Przemysław Różycki wrote:
> Hello,
> 
> I have written some code, which creates many threads for each connection 
> ('main connection'). The purpose of this code is to balance the load 
> between several connections ('pipes'). The number of spawned threads 
> depends on how many pipes I create (= 2*n+2, where n is the number of 
> pipes).
> 
> For good results I'll presumably share main connection's load between 10 
> pipes - therefore 22 threads will be spawned. Now if about 50 
> connections are forwarded the number of threads rises to thousand of 
> threads (or several thousands if even more connections are established).
> 
> My questions are:
> - What is the cost (in memory / CPU usage) of creating such amounts of 
> threads?
> - Is there any 'upper boundary' that limits the number of threads? (is 
> it python / OS related)
> - Is that the sign of 'clumsy programming' - i.e. Is creating so many 
> threads a bad habit? (I must say that it simplified the solution of my 
> problem very much).
> 
> Limiting the number of threads is possible, but would affect the 
> independence of data flows. (ok I admit - creating tricky algorithm 
> could perhaps gurantee concurrency without spawning so many threads - 
> but it's the simplest solution to this problem :) ).

PR,
    I notice there's a resource module with a
getrusage(who) that looks like it would support
a test to get what you need.
wes