[python-win32] 1MB Thread Stack Size

[Hughes, Chad O]

Actually, the bottleneck is not the CPU.  Using Perfmon, I have verified
that I can currently have 1000 threads running with under 25% CPU usage.
The threads are fairly light weight. However, the stack size for each
thread is by default 1MB, so the memory is the bottleneck.  I want to
have about 1000 threads, however, this requires that every machine I run
my app on must have at least 1GB of memory.  Not all my machines have
this much memory.

I have read that you can change the default by recompiling Python and
have at least 16 thousand threads.  In fact, this is what I read at
http://www.gossamer-threads.com/lists/python/python/354638:

  The only way to change the stack size is to define THREAD_STACK_SIZE
in
  the thread_pthread.h header. There is an ifdef for the Mac that sets
  this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
  and recompiled Python. I can now create 16k threads although I'm not
  sure yet whether our application will run; some tuning is probably in
  order. I'm going to get the 2.2 source code and see what is different
  between the two and maybe get an idea of what a reasonable stack size
is. 

However, this pertained to Linux and I am running my app Windows and I
was looking for a solution that does not require a Python recompile.



-----Original Message-----
From: python-win32-bounces at python.org
[mailto:python-win32-bounces at python.org] On Behalf Of Tim Roberts
Sent: Thursday, June 02, 2005 11:32 AM
To: python-win32 at python.org
Subject: [python-win32] 1MB Thread Stack Size


On Wed, 01 Jun 2005 17:25:50 -0700, "Hughes, Chad O" 
<chad.hughes at pnl.gov> wrote:

>I have a program that needs to create a great deal of threads. 
>Unfortunately, I cannot seem to find a way to lower the 1MB default 
>stack size per thread.  The threading module does not seem to support 
>setting the stack size explicitly.  I have 1GB of memory on my system 
>so I can only create about 1000 threads before I receive the following 
>error trace:
>  
>

It is almost impossible to imagine a situation in which 1,000 threads 
actually provides any productivity.  Any time the number of threads in a

process is more than an order of magnitude greater than the number of 
CPUs, the operating system administration overhead begins to overwhelm 
the productive work that can be done.

Surely you could solve your problem with 100 threads and some 
appropriate queueing?

-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.

_______________________________________________
Python-win32 mailing list
Python-win32 at python.org
http://mail.python.org/mailman/listinfo/python-win32