[python-win32] 1MB Thread Stack Size
Hughes, Chad O
chad.hughes at pnl.gov
Thu Jun 2 21:14:32 CEST 2005
Actually, the bottleneck is not the CPU. Using Perfmon, I have verified
that I can currently have 1000 threads running with under 25% CPU usage.
The threads are fairly light weight. However, the stack size for each
thread is by default 1MB, so the memory is the bottleneck. I want to
have about 1000 threads, however, this requires that every machine I run
my app on must have at least 1GB of memory. Not all my machines have
this much memory.
I have read that you can change the default by recompiling Python and
have at least 16 thousand threads. In fact, this is what I read at
The only way to change the stack size is to define THREAD_STACK_SIZE
the thread_pthread.h header. There is an ifdef for the Mac that sets
this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
and recompiled Python. I can now create 16k threads although I'm not
sure yet whether our application will run; some tuning is probably in
order. I'm going to get the 2.2 source code and see what is different
between the two and maybe get an idea of what a reasonable stack size
However, this pertained to Linux and I am running my app Windows and I
was looking for a solution that does not require a Python recompile.
From: python-win32-bounces at python.org
[mailto:python-win32-bounces at python.org] On Behalf Of Tim Roberts
Sent: Thursday, June 02, 2005 11:32 AM
To: python-win32 at python.org
Subject: [python-win32] 1MB Thread Stack Size
On Wed, 01 Jun 2005 17:25:50 -0700, "Hughes, Chad O"
<chad.hughes at pnl.gov> wrote:
>I have a program that needs to create a great deal of threads.
>Unfortunately, I cannot seem to find a way to lower the 1MB default
>stack size per thread. The threading module does not seem to support
>setting the stack size explicitly. I have 1GB of memory on my system
>so I can only create about 1000 threads before I receive the following
It is almost impossible to imagine a situation in which 1,000 threads
actually provides any productivity. Any time the number of threads in a
process is more than an order of magnitude greater than the number of
CPUs, the operating system administration overhead begins to overwhelm
the productive work that can be done.
Surely you could solve your problem with 100 threads and some
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.
Python-win32 mailing list
Python-win32 at python.org
More information about the Python-win32