[Python-Dev] patch #1454481 vs buildbot

[Tim Peters]

[Andrew MacIntyre]
> In reviewing the buildbot logs after committing this patch, I see 2
> issues arising that I need advice about...
>
> 1.  The Solaris build failure in thread.c has me mystified as I can't
> find any "_sysconf" symbol - is this in a system header?

The patch's

#if THREAD_STACK_MIN < PTHREAD_STACK_MIN

assumes that the expansion of PTHREAD_STACK_MIN acts like a
compile-time constant expression, but there's no such guarantee.

    http://cvs.opensolaris.org/source/xref/on/usr/src/head/limits.h

shows that, on one version of Solaris, it's actually defined via

#define	PTHREAD_STACK_MIN ((size_t)_sysconf(_SC_THREAD_STACK_MIN))

That has a runtime value, but not a useful compile-time value.  The
only useful thing you can do with it in an #if expression is
defined(PTHREAD_STACK_MIN).

> 2.  I don't know what to make of the failure of test_threading on Linux,
> as test_thread succeeds as far as I could see.  These tests succeed on my
> FreeBSD box and also appear to be succeeding on the Windows buildbots.

Not all pthreads-using builds fail, and not all failing pthreads-using
builds fail in the same way.  Welcome to pthreads on Linux ;-)

BTW, this sucks:

test_thread
/home/buildbot/Buildbot/trunk.baxter-ubuntu/build/Lib/test/test_thread.py:138:
RuntimeWarning: thread stack size of 0x1000 bytes not supported
  thread.stack_size(tss)

That's from a successful run.  RuntimeWarning really doesn't make
sense for a failing operation.  This should raise an exception
(xyzError, not xyzWarning), or a failing stack_size() should return an
error value after ensuring the original stack size is still in effect.

> Unfortunately I don't have access to either a Solaris box or a Linux box
> so could use some hints about resolving these.

As above, they don't always fail in the same way across boxes.  The
most popular failure mode appears to be:

ERROR: test_various_ops_large_stack (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/buildslave/python/trunk.norwitz-amd64/build/Lib/test/test_threading.py",
line 101, in test_various_ops_large_stack
    self.test_various_ops()
  File "/home/buildslave/buildslave/python/trunk.norwitz-amd64/build/Lib/test/test_threading.py",
line 77, in test_various_ops
    t.start()
  File "/home/buildslave/buildslave/python/trunk.norwitz-amd64/build/Lib/threading.py",
line 434, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread

While I don't know, best guess is that the system "ulimit -s" is set
to 8MB, so it's not actually possible to get a 16MB stack (as
test_various_ops_large_stack() asks for), and this error goes
undetected at the test's

         threading.stack_size(0x1000000)

call "for some reason".  Or maybe it is, but the RuntimeWarning got
lost -- if this were an exception instead, it would be easier to
reason about, and test_various_ops_large_stack() could disable itself
gracefully (by catching the exception and giving up) if the platform
didn't allow a 16MB stack ...

Ah, _pythread_pthread_set_stacksize doesn't do anything to verify that
the requested stack size is actually usable (just ensures that it's
less than THREAD_STACK_MIN).  pthread_attr_setstacksize() isn't
attempted until PyThread_start_new_thread() in thread_pthread.h:

#if defined(THREAD_STACK_SIZE)
	tss = (_pythread_stacksize != 0) ? _pythread_stacksize
					 : THREAD_STACK_SIZE;
	if (tss != 0) {
		if (pthread_attr_setstacksize(&attrs, tss) != 0) {
			pthread_attr_destroy(&attrs);
			return -1;
		}
	}
#endif

If PyThread_start_new_thread() fails in any way
(like,pthread_attr_setstacksize() failing), ""can't start new thread"
is the error we see.

The difference between test_thread and test_threading here is that
only test_threading asks for a 16MB stack; test_thread doesn't ask for
a stack larger than 4MB.

Until all this gets resolved, I strongly suggest reverting this patch
(if you don't, someone else will ...) and hammering out the problems
on a new branch instead.  See python-dev email from yesterday for how
to force a buildbot slave to build a branch.