Maximum number of threads
kangell at alertra.com
Wed Sep 29 19:24:20 CEST 2004
Jeremy Jones wrote:
> around 400 for FC2. I may be wrong, but I think Python relies on the
> underlying OS threading libraries for everything threading related.
> Which is probably why the recent change in either 2.2 or 2.3 for a dummy
> threading module that simulates threading on systems that do not support
> threading. I know my babbling isn't getting you any closer to a
No, your babbling has gotten me much closer. I'm not sure what changed
between Python 2.2 and Python 2.3, but the problem is the stack size
allocated to each thread. Apparently with glibc on RH9 something along
the order of 8 MB of virtual memory is allocated for the stack of each
thread. 256 threads takes us to the virtual memory limit of the process
itself. Glibc on FC2 must be using a different default stack size, or
the virtual memory limit is higher; I've only recently started running
FC2 on our test servers so I'm not sure which it is.
The only way to change the stack size is to define THREAD_STACK_SIZE in
the thread_pthread.h header. There is an ifdef for the Mac that sets
this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value
and recompiled Python. I can now create 16k threads although I'm not
sure yet whether our application will run; some tuning is probably in
order. I'm going to get the 2.2 source code and see what is different
between the two and maybe get an idea of what a reasonable stack size is.
There is a very nice write-up on the basic issue here:
Thanks for your help.
More information about the Python-list