[New-bugs-announce] [issue12400] regrtest: always run tests in verbose mode, but hide the output on success

[STINNER Victor]

New submission from STINNER Victor <victor.stinner at haypocalc.com>:

Sporadic issues are difficult to analyze on the buildbots because sometimes a test fails, but when it runs again in verbose mode... it doesn't fail anymore. Typical example:

===============================
[ 99/356/1] test_smtplib
Re-running test test_smtplib in verbose mode
(...)
testTimeoutValue (test.test_smtplib.GeneralTests) ... ok
testBasic (test.test_smtplib.DebuggingServerTests) ... test test_smtplib failed -- multiple errors occurred; run in verbose mode for details
ok
testHELP (test.test_smtplib.DebuggingServerTests) ... ok
testNOOP (test.test_smtplib.DebuggingServerTests) ... ok
testNotImplemented (test.test_smtplib.DebuggingServerTests) ... ok
(...)
----------------------------------------------------------------------
Ran 32 tests in 23.664s

OK
===============================

I propose to always run the tests in verbose mode, write the output into a buffer, but only display the output on failure (or ignore the output on success).

Attached patch simplifies regrtest.py usage: replace 4 verbose levels (no verbose option, -v, -w, -W) by only 2 (no verbose option, -v). Always display the full output in case of an error (as we used -w before): no more "multiple error occured" without output trap, no more "oh, rerunning the test doesn't fail" trap. The -v option works as before (directly write the output in verbose mode).

regrtest is much more verbose than before in case of a failure, but I prefer too much information than no information when I try to debug something. If it is too much verbose by default, I can try to use 3 verbose levels (e.g. don't display the full output if only one test failed).

If the patch is accepted, the devguide should also be patched.

----------
components: Tests
files: regrtest_verbose.patch
keywords: patch
messages: 138981
nosy: ezio.melotti, georg.brandl, haypo, pitrou, r.david.murray
priority: normal
severity: normal
status: open
title: regrtest: always run tests in verbose mode, but hide the output on success
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file22446/regrtest_verbose.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12400>
_______________________________________