Recommended number of threads? (in CPython)
davea at ieee.org
Fri Oct 30 00:26:21 CET 2009
Paul Rubin wrote:
> Neil Hodgson <nyamatongwe+thunder at gmail.com> writes:
>> If you are running on a 32-bit environment, it is common to run out
>> of address space with many threads. Each thread allocates a stack and
>> this allocation may be as large as 10 Megabytes on Linux.
> I'm sure it's smaller than that under most circumstances. I run
> python programs with hundreds of threads all the time, and they don't
> use gigabytes of memory.
As Neil pointed out further on, in the same message you quoted, address
space is not the same as allocated memory. It's easy to run out of
allocatable address space long before you run out of virtual memory, or
Any time a buffer is needed that will need to be contiguous (such as a
return stack), the address space for the max possible size must be
reserved, but the actual virtual memory allocations (which is what you
see when you're using the system utilities to display memory usage) are
done incrementally, as needed.
It's been several years, but I believe the two terms on Windows are
"reserve" and "commit." Reserve is done in multiples of 64k, and commit
in multiples of 4k.
More information about the Python-list