[Python-Dev] test failures on Debian unstable

[Tim Peters]

[martin@v.loewis.de, on test_normalization]
> What -u would you propose?

Someone who knows what the test does should name the resource.

>> test_bsddb3 is already in already a regrtest -u thingie, so
>> should be in the expected-skip list on all boxes.

> I don't like the expected-skip mechanism at all.

I love it:  it solves real problems on Windows.  The latest example is that
I never would have noticed that test_bsddb3 even existed if the mechanism
hadn't told me it was an unexpected skip on Windows.  Ditto
test_normalization.  Likewise Neil's report of expected skips on Debian
pointed out oddities to him.  On platforms where dozens of tests are
skipped, just listing the tests that were skipped isn't informative
enough -- the list is so big that you simply don't notice if a new test
shows up there.  Before this mechanism, entire new packages didn't make it
into the Windows distro because *nobody* noticed that their test suite was
getting skipped.  That hasn't happened again.

> Why does being Linux (or HP-UX) has anything to do with whether we
> can run the test_normalization test; that test is completely
> platform-independent.

The expected-skip list doesn't really have to do with whether we *can* run a
test, it has to do with whether it's a red flag when we find we can't run a
test.

> Why is it expected that test_bz2 works on Linux?

I don't know that it is.

> It won't work if you don't have the libraries.

If so, put it in the expected-skip list for Linux.  regrtest doesn't whine
about expected-skip entries that *do* run to completion, it only points out
tests that don't run to completion (due to ImportError or TestSkipped) that
also aren't in the platform's expected-skip list.

> Instead, I would expect that test_bz2 works if the bz2 module is
> present, and fails to work if it was not built.

Then it's an expected skip on platforms where that's a choice, in the sense
the phrase is intended.  "expected skip" may be a poor name for it, although
it makes literal sense on Windows.

> Likewise for test_normalization: If a certain file is present, it
> should succeed, otherwise, it should fail.

Ditto, but even more so in this case, because that "certain file" isn't
present on any system unless the user specifically downloads it and places
it in their current working directory before running test_normalization.
That makes it so unlikely this test will run for users that it should
probably be put into the expected-skip list on all boxes via the
_ExpectedSkips constructor (maybe a good idea for all -u tests).