[Python-Dev] test failures on Debian unstable

[Tim Peters]

[martin@v.loewis.de]
> ...
> So should it be? If not: Why is it expected that test_bz2 fails on
> Linux?

I can only repeat that the expected-skip mechanism has nothing to do with
tests *failing*.  {pass, fail, skip} are the possible outcomes of running a
test, they're mutually exclusive, and the expected-skips list has only to do
with the skip outcome.  If a test passes or fails, expected-skips doesn't
enter into it.

The appearance of a test name T in the expected-skips list doesn't imply
that T *must* be skipped, either.  If T isn't skipped, fine, the regrtest
summary doesn't mention T at all if it passed; it does mention T if it
failed.

> Whether the test passes or fails simply has nothing to what system you
> run it on.

Nor does passing or failing have anything to do with expected-skips.

> To solve the real problem on Windows, it seems that a list
> "tim_has_seen_this_test" would be sufficient.

I'm not sure exactly what that means, so can't guess whether it would be
sufficient.  I do know that keeping skip lists by hand was not sufficient,
and that the current mechanism is sufficient for Windows.

>>> It won't work if you don't have the libraries.

>> If so, put it in the expected-skip list for Linux.

> Sure, but that holds for nearly every test requiring extension
> modules: if some library isn't there, the module doesn't work.
> ...
> If the test is added to the skip list, the a potential problem will be
> hidden: it may be that the test ought to pass on a certain
> installation, but doesn't because of a real bug in Python. So adding
> the test into the skipped list on grounds of the library potentially
> unavailable hides real problems.

expected-skips doesn't hide test failures.  If, wrt a specific test T, you
want to say that the failure to import a thing is a failure *of* T, then the
ImportError should be caught and transformed into an exception other than
{ImportError, TestSkipped}, so that regrtest treats T as a failure instead
of a skip.  For example, raising TestFailed in that case would be thoroughly
appropriate.  And TestSkipped should never be raised for a test failure.

If Unix weenies want a much fancier system, or want to exempt Unix entirely
from this mechanism, that's fine by me, but I'm not going to do the work.
But from what I've seen, there are still differing notions of "the default
configuration" on various non-Windows platforms, and expected-skips does
point to problems on them in real life.

The {pass, fail, skip} partition of test outcomes has been there forever,
and before the expected-skip mechanism was added too, users (including
Python developers!) had no clue about whether the tests that skipped on
their platforms were OK or were really errors.  From what I've seen,
introducing the expected-skips mechanism did more good than harm everywhere,
although it did most good on Windows, Windows relatives, and Macs.

If you exempt Linux from the mechanism, then everyone (except, presumably,
you) who runs tests on Linux is going to see that test_normalization is
skipped, and they're going to ask whether that's expected, or whether it's
an error.  It doesn't matter that the regrtest output will no longer say
that the skip isn't expected on Linux, they'll ask about *every* test that
got skipped.  For goodness sake, they even used to ask whether it was
expected than winsound got skipped on Linux -- and reasonably so, since
"winsound" doesn't mean anything to most users.  Neither does
"normalization", for that matter.  All tests are baffling to someone
installing Python for the first time, and I wager most tests remain obscure
to everyone except their author.

The best that can be done with test_normalization under this framework is to
be realistic:  virtually nobody has the file it needs, so it's going to get
skipped virtually everywhere.  A clearer case of an expected skip is hard to
imagine, so it should be added to the expected-skip list everywhere.  The
test should also be changed to avoid conflating the skip outcome with
various possible file failure outcomes.  That is, instead of

try:
    data = open("NormalizationTest.txt", "r").readlines()
except IOError:
    raise TestSkipped("NormalizationTest.txt not found, ..." ...)

it should do something like

if not os.path.exists("NormalizationTest.txt"):
    raise TestSkipped("NormalizationTest.txt not found, ..." ...)

data = open("NormalizationTest.txt").readlines()

A permission problem, or problem with reading the file, are errors here, and
should be allowed to propagate back to regrtest so that they can be reported
as test failure (!= test skipped).