[python-win32] 1MB Thread Stack Size

[Tim Roberts]

On Thu, 02 Jun 2005 12:14:32 -0700, "Hughes, Chad O" 
<chad.hughes at pnl.gov> wrote:

>Actually, the bottleneck is not the CPU.  Using Perfmon, I have verified
>that I can currently have 1000 threads running with under 25% CPU usage.
>The threads are fairly light weight. However, the stack size for each
>thread is by default 1MB, so the memory is the bottleneck.  I want to
>have about 1000 threads, however, this requires that every machine I run
>my app on must have at least 1GB of memory.  Not all my machines have
>this much memory.
>

Well, let me try to restate my point.  Having 1,000 lightweight threads 
sitting around waiting for things is not a big drain on resources, but 
it also doesn't serve any purpose.  If you have a pool of 1,000 threads 
sitting around, I assume you eventually plan to have them do some real 
work.  If you try to assign real tasks to 1,000 threads, the context 
switching overhead would kill the machine.  You would get much better 
performance by having 100 threads, each one reading tasks from a queue.  
That way, a single thread can handle a couple of tasks without relying 
on the operating system to switch the context.

And if you DON'T really plan on having 1,000 threads working on tasks at 
the same time, then you don't really need 1,000 threads.

I'm sure you can solve the stack space problem.  I'm just suggesting 
that time spent on such a solution might be wasted, when you could 
actually solve your problem in a better way.

-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.