[Python-Dev] Buildbots and regrtest timeout

[Victor Stinner]

Hi,

Stephan Krah asked me to change how the default timeout is defined for regrtest 
(issue #12250):

"The implicit timeout in regrtest.py makes it harder to write automated
test scripts for 3rd party modules. First, you have to remember to
set --timeout=0 for long running tests. Then, you have to remember
not to use the --timeout option when compiling --without-threads.

I'd much prefer that there's no timeout unless explicitly specified.
For the buildbots, I think this could be done in the Makefile."

First I replaced the hardcoded constant in regrtest.py by a command line 
argument (--timeout=3600) in the TESTOPTS variable of the Makefile, so if you 
call regrtest directly (without make), there is no more default timeout.

But today I saw a a buildbot timeout without any traceback: a possible hang in 
test_io on "x86 FreeBSD 7.2 3.x" buildbot, "command timed out: 3900 seconds 
without output". I realized that some buildbots (all buildbots?) override the 
TESTOPTS variable ("make buildbottest TESTOPTS= TESTPYTHONOPTS=" for "x86 
FreeBSD 7.2 3.x").

I moved the timeout option from TESTOPTS to a new variable: TESTTIMEOUT, with 
a default value of 1 hour (3600 seconds). The timeout is now only used for 
"make buildbottest". Use TESTTIMEOUT=0 to disable completly regrtest timeout. 
The timeout argument if ignored (with a warning) if 
faulthandler.dump_tracebacks_later() function is missing (e.g. if Python is 
compiled without threads).

Please, don't disable the TESTTIMEOUT option for your buildbot. You may want 
to use a shorter or longer timeout, it just have to be smaller than the 
buildbot timeout (3900 seconds by default, 1 hour 6 minutes, just a little bit 
more than regrtest timeout: 1 hour). If you want a timeout longer than 1 hour, 
you can to change TESTTIMEOUT and the buildbot timeout (I don't know how to 
configure the buildbot timeout).

I didn't touch test, testall, testuniversal, quicktest make rules, which don't 
use the regrtest timeout anymore. I would prefer to use the same timeout 
options for all test rules in TESTOPTS, but it doesn't work because some 
buildbots do override TESTOPTS variable.

Victor