Reporting unexpected import failures as test failures in regrtest.py
Some background for those not watching python-checkins: I neglected to do "svn add" for the new functools Python module when converting functional->functools. The buildbots stayed green because the ImportError triggered by the line "import functools" in test_functools was treated as a TestSkipped by regrtest.py. Georg noticed the file missing from the checkin message, but this is the second time I (and the buildbots) have missed a regression due to this behaviour. (As I recall, last time I checked in some broken code because I didn't notice the additional entry appearing in the list of unexpected skips in my local testing) Tim Peters wrote:
[Nick Coghlan]
... (we should probably do something about that misleading ImportError -> TestSkipped -> green buildbot behaviour. . . )
I looked at that briefly a few weeks back and gave up. Seemed the sanest thing was to entirely stop treating ImportError as "test skipped", and rewrite tests that legimately _may_ get skipped to catch expected ImportErrors and change them to TestSkipped themselves.
A bit of framework might help; e.g., a test that expects to get skipped due to failing imports on some platforms could define a module-level list bound to a conventional name containing the names of the modules whose import failure should be treated as TestSkipped, and then regrtest.py could be taught to check import errors against the test module's list (if any).
In the case du jour, test_functools.py presumably wouldn't define that list, so that any ImportError it raised would be correctly treated as test failure.
What if we appended unexpected skips to the list of bad tests so that they get rerun in verbose mode and the return value becomes non-zero? print count(len(surprise), "skip"), \ "unexpected on", plat + ":" printlist(surprise) # Add the next line after the previous two in regrtest.py bad.extend(surprise) (This happens after the count of failed tests has been printed, so we don't affect that output) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org
[Nick Coghlan]
What if we appended unexpected skips to the list of bad tests so that they get rerun in verbose mode and the return value becomes non-zero?
print count(len(surprise), "skip"), \ "unexpected on", plat + ":" printlist(surprise) # Add the next line after the previous two in regrtest.py bad.extend(surprise)
(This happens after the count of failed tests has been printed, so we don't affect that output)
While "expected skips" is reliable on Windows, it's just a guess everywhere else. That's because the Windows build is unique in supplying and compiling all the external packages Python ships with on Windows. On non-Windows boxes, which skips "are expected" also depends on which external packages happen to be installed on the box, and how Python was configured, and regrtest.py has no knowledge about those. For example, on the "sparc solaris10 gcc trunk" buildbot, 6 tests show up as "unexpected skips", and there's just no way for a non-platform-expert to guess which of those are and aren't pointing at problems. It's easy enough to guess that test_zipfile is skipped because the box doesn't have an external zlib installed, but why is test_ioctl skipped? I have no idea (and neither does regrtest.py), but it's presumably obvious to a Sun expert. In any case, if "unexpected skips" got appended to `bad`, the non-zero exit code would cause that buildbot slave to fail every time. It's not unique, either; e.g., the "x86 gentoo trunk" slave always skips test_tcl, presumably because Tcl isn't installed on that box. In short, there's simply no way to _guess_ which import errors are and aren't "expected". Improving that requires adding more knowledge to the testing machinery. The most important thing is to add knowledge about which import errors are _never_ expected (perhaps by adding knowledge about which imports _may_ legitimately fail), since we can know that for sure about x-platform modules, and that's exactly the problem that keeps burning us. Confusing ImportError with TestSkipped is an admirably lazy idea that unfortunately didn't turn out to work very well.
participants (2)
-
Nick Coghlan -
Tim Peters