[Python-Dev] Community buildbots (was Re: User's complaints)

[Tim Peters]

[Neal Norwitz]
> ...
> That leaves 1 unexplained failure on a Windows bot.

It wasn't my Windows bot, but I believe test_profile has failed
(rarely) on several of the bots, and in the same (or very similar)
way.  Note that the failure went away on the Windows bot in question
the next time the tests were run on it.  That's typical of
test_profile failures.

Unfortunately, because test_profile works by comparing stdout against
a canned expected-output file under Lib/test/output/, when we re-run
the test in verbose mode at the end, that comparison isn't done, so
there's isn't a simple way to know whether the test passed or failed
during its second run.  I _bet_ it actually passed, but don't know.

This problem is obscured because when you run regrtest.py "by hand"
with -v, you get this message at the end:

"""
CAUTION:  stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.
"""

which is intended to alert you to the possible problem.

But when we re-run failing tests in verbose mode "by magic" (via
passing -w), that warning isn't produced.

BTW, the best solution to this is to convert all the tests away from
regrtest's original "compare stdout against a canned output/TEST_NAME
file" scheme.  That won't fix test_profile, but would make it less of
a puzzle to figure out what's wrong with it.