[Python-Dev] test failures on Debian unstable

[Martin v. Loewis]

Tim Peters <tim.one@comcast.net> writes:

> > Whether the test passes or fails simply has nothing to what system you
> > run it on.
> 
> Nor does passing or failing have anything to do with expected-skips.

Right. I meant to make a different point, though:

For many of the tests that are somtimes skipped, knowing the system
does not tell you whether the test will should rightfully be skipped,
on that system.

> I'm not sure exactly what that means, so can't guess whether it would be
> sufficient.  I do know that keeping skip lists by hand was not sufficient,
> and that the current mechanism is sufficient for Windows.

Apparently, the only useful information drawn from the skip lists is
"there is a new test that is skipped, should we update the windows
build process?".

In all other installations, the only action people take is to silence
the message, which wouldn't be necessary if the message wasn't
produced in the first place.

> expected-skips doesn't hide test failures.  If, wrt a specific test
> T, you want to say that the failure to import a thing is a failure
> *of* T, then the ImportError should be caught and transformed into
> an exception other than {ImportError, TestSkipped}, so that regrtest
> treats T as a failure instead of a skip.

That is my point: I *cannot know* in advance whether a failure to
import T is a failure of T, in the general case. You seem to think
that one can tell by just looking at sys.platform whether import ought
to succeed: a failed import is either expect (pass), or unexpected
(fail, i.e. must investigate).

This is not the case, not even on Windows - except for your build
environment, and the Pythonlabs Windows distribution. For example, you
expect to execute test_bz2. However, on somebody else's Windows
distribution, test_bz2 might be skipped because the libraries where
not all in place when Batch Build was invoked.

So should we add test_bz2 to expected skips on Windows, as somebody
might not have bz2 libraries in his build environment?

> If Unix weenies want a much fancier system, or want to exempt Unix entirely
> from this mechanism, that's fine by me, but I'm not going to do the work.
> But from what I've seen, there are still differing notions of "the default
> configuration" on various non-Windows platforms, and expected-skips does
> point to problems on them in real life.

There are certainly modules which will never work on certain systems;
in those cases, the mechanism works as designed. In general, you need
to know much more than just the system name to determine whether
skipping a test is expected.

> The {pass, fail, skip} partition of test outcomes has been there forever,
> and before the expected-skip mechanism was added too, users (including
> Python developers!) had no clue about whether the tests that skipped on
> their platforms were OK or were really errors.  

They still don't have a clue. When they find a non-empty output, they
submit a patch listing all the tests that were skipped, and claim that
this list is correct for their platform.

> From what I've seen, introducing the expected-skips mechanism did
> more good than harm everywhere, although it did most good on
> Windows, Windows relatives, and Macs.

I disagree. It generates a constant flow of useless patches.

> If you exempt Linux from the mechanism, then everyone (except, presumably,
> you) who runs tests on Linux is going to see that test_normalization is
> skipped, and they're going to ask whether that's expected, or whether it's
> an error.  

In this specific case, the "skipped" message gives a clear indication
of the problem.

> It doesn't matter that the regrtest output will no longer say
> that the skip isn't expected on Linux, they'll ask about *every* test that
> got skipped.  For goodness sake, they even used to ask whether it was
> expected than winsound got skipped on Linux -- and reasonably so, since
> "winsound" doesn't mean anything to most users.  

If that is the problem to solve, we can always perform

print "Those skips are all expected."

at the end of the run - no need to maintain explicit lists.

Regards,
Martin