questions regarding stack size use for multi-threaded python programs

[Gabriel Genellina]

En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon <eyal.gordon at gmail.com>
escribió:

> background:
> we are using python 2.4.3 on CentOS 5.3 with many threads - and our  
> shell's
> default stack size limit is set to 10240KB (i.e. ~10MB).
>
> we noticed that python's Threading module appears to create threads with
> this value as their stack size (we ran a sample program that creates 10
> threads and measured its virtual memory size, then reduced the stack size
> limit of the shell to 5120KB - and saw that the program's virtual memory
> size was reduced by ~50MBs).
>
> the problem:
> our program uses numerous threads, and thus the virtual memory size gets  
> to
> be very large. we would like to reduce the size of the stack to reduce  
> this
> size. we were looking for information about recommendation for the stack
> size to use, but found none.

You can set the thread stack size (for threads that are going to be
created, not existing threads) using threading.stack_size(SIZE)
http://docs.python.org/library/threading.html#threading.stack_size

> questions:
> 1. is there some rule-of-thumb for the recommended stack size for python
> programs of various sorts?

No idea. I've been always happy with the default settings.

> 2. is there a way for us, at runtime (from inside the code or outside the
> process), to find how much of a thread's stack we are using (in KB or  
> some
> other size units)?

see top(1)

> 3. when we define local objects - do the objects themselves get  
> allocated on
> the stack - or are they allocated on the heap and only references to them
> are kept on the stack?

They're allocated on the heap, and most references to objects are on the
heap too. The Python stack of execution frames is allocated on the heap
too. Basically, the OS stack is used for local variables in C code only.

> 4. would the size of the stacks (which are probably not really allocated  
> by
> the linux virtual memory sub-system, unless used) have a noticeable
> performance effect on a python program? same question regarding the use  
> of a
> large number of threads?

I think it doesn't matter, unless you create so many threads as to exhaust
the available addressing space (in 32bits, 4GB address space and 10MB per
thread means 400 threads maximum).

-- 
Gabriel Genellina