Maximum number of threads

[Kirby Angell]

Jeremy Jones wrote:
> 
> around 400 for FC2.  I may be wrong, but I think Python relies on the 
> underlying OS threading libraries for everything threading related.  
> Which is probably why the recent change in either 2.2 or 2.3 for a dummy 
> threading module that simulates threading on systems that do not support 
> threading.  I know my babbling isn't getting you any closer to a 
> solution.....
> 

No, your babbling has gotten me much closer.  I'm not sure what changed 
between Python 2.2 and Python 2.3, but the problem is the stack size 
allocated to each thread.  Apparently with glibc on RH9 something along 
the order of 8 MB of virtual memory is allocated for the stack of each 
thread.  256 threads takes us to the virtual memory limit of the process 
itself. Glibc on FC2 must be using a different default stack size, or 
the virtual memory limit is higher; I've only recently started running 
FC2 on our test servers so I'm not sure which it is.

The only way to change the stack size is to define THREAD_STACK_SIZE in 
the thread_pthread.h header.  There is an ifdef for the Mac that sets 
this to 64k.  Just as a test I forced THREAD_STACK_SIZE to that value 
and recompiled Python.  I can now create 16k threads although I'm not 
sure yet whether our application will run; some tuning is probably in 
order.  I'm going to get the 2.2 source code and see what is different 
between the two and maybe get an idea of what a reasonable stack size is.

There is a very nice write-up on the basic issue here:

http://www.kegel.com/stackcheck/

Thanks for your help.