[Python-Dev] Python Thread Stack Size

[Geoff Gerrietts]

Quoting Casey Duncan (casey@zope.com):
> Hi All:
> 
> For those of you who don't know me, I am Casey Duncan, a recent addition to 
> the engineering team at Zope corp. 
> 
> Jim Fulton and I were talking about a problem that manifests itself on BSD in 
> threaded Python applications that use a lot of stack space (notably the CMF 
> application in Zope).
> 
> It seems that FreeBSD has a  default thread stack size of 64K, whereas most 
> unices have somewhere more like 1Mb. A patch exists to have Python bump up 
> the stack size, but this involves patching the Python source.
> 
> Jim suggested that Python should set the thread stack size itself to something 
> like 256 * the pointer size, which would work out to 1Mb on 32 bit platforms 
> and 2Mb on 64 bit platforms. This would hopefully lead to more consistent 
> behavior across platforms.
> 
> Alternately (or additionally) it could be a configure option like 
> --thread-stack=1024 or somesuch.
> 
> A third alternative (which is less desireable IMO) would be to change the 
> Python BSD port so that it includes a patch to do this. Obviously that 
> doesn't help ppl building directly from source very much tho.
> 
> Thoughts?

Having spent a lot of time looking at this problem (it was a red
herring, but thoroughly investigated) on Linux, I'd offer a slight
warning for that platform:

Linux uses pthreads just like the other platforms, has a default
thread stack size of 2MB, and under the RedHat builds of glibc I've
played with, that stack size is not mutable due to compile-time
settings on glibc. When you try to call pthread_attr_setstacksize for
anything but 2 * 1024 * 1024, it returns EINVAL. This may be an
aberration of the RedHat setup -- I haven't tested other Linux
distributions, and I never got to the point of recompiling glibc to
see if I could make "dynamic stacks" work.

Because of the limitation -- and because of the large default stack
size on Linux -- I'd suggest that if you're going to build stack size
readjustment into Python, you go with something like:
  
  int rc = 0, sz;
  pthread_attr_t pta;

  rc = phtread_attr_init(&pta);
  
  rc = pthread_attr_getstacksize(&pta, &sz);

  if (sz < YOUR_STACK_SIZE_HERE) {
    /* now try setting the stacksize */
  }

When I was thinking about this problem -- when I still thought it
might be a problem -- I gave some serious thought to making the stack
size a parameter on the thread module, so that threaded applications
could specify the amount of stack space they expected to require.

There are probably a billion reasons that's not a good idea, but it
seemed to me like it offered the flexibility at the place where you
were likely to need it, rather than at the place where it was most
convenient to provide it.

Thanks,
--G.

-- 
Geoff Gerrietts             "I am always doing that which I can not do, 
<geoff at gerrietts net>     in order that I may learn how to do it." 
http://www.gerrietts.net                    --Pablo Picasso