Recommended number of threads? (in CPython)

[Dave Angel]

Paul Rubin wrote:
> Neil Hodgson <nyamatongwe+thunder at gmail.com> writes:
>   
>>    If you are running on a 32-bit environment, it is common to run out
>> of address space with many threads. Each thread allocates a stack and
>> this allocation may be as large as 10 Megabytes on Linux. 
>>     
>
> I'm sure it's smaller than that under most circumstances.  I run
> python programs with hundreds of threads all the time, and they don't
> use gigabytes of memory.
>
>   
As Neil pointed out further on, in the same message you quoted, address 
space is not the same as allocated memory.  It's easy to run out of 
allocatable address space long before you run out of virtual memory, or 
swap space.

Any time a buffer is needed that will need to be contiguous (such as a 
return stack), the address space for the max possible size must be 
reserved, but the actual virtual memory allocations (which is what you 
see when you're using the system utilities to display memory usage) are 
done incrementally, as needed.

It's been several years, but I believe the two terms on Windows are 
"reserve" and "commit."  Reserve is done in multiples of 64k, and commit 
in multiples of 4k.

DaveA