[Python-Dev] Reporting unexpected import failures as test failures in regrtest.py

[Nick Coghlan]

Some background for those not watching python-checkins:

I neglected to do "svn add" for the new functools Python module when 
converting functional->functools. The buildbots stayed green because the 
ImportError triggered by the line "import functools" in test_functools was 
treated as a TestSkipped by regrtest.py.

Georg noticed the file missing from the checkin message, but this is the 
second time I (and the buildbots) have missed a regression due to this 
behaviour. (As I recall, last time I checked in some broken code because I 
didn't notice the additional entry appearing in the list of unexpected skips 
in my local testing)

Tim Peters wrote:
> [Nick Coghlan]
>> ... (we should probably do something about that misleading ImportError ->
>> TestSkipped -> green buildbot behaviour. . . )
> 
> I looked at that briefly a few weeks back and gave up.  Seemed the
> sanest thing was to entirely stop treating ImportError as "test
> skipped", and rewrite tests that legimately _may_ get skipped to catch
> expected ImportErrors and change them to TestSkipped themselves.
> 
> A bit of framework might help; e.g., a test that expects to get
> skipped due to failing imports on some platforms could define a
> module-level list bound to a conventional name containing the names of
> the modules whose import failure should be treated as TestSkipped, and
> then regrtest.py could be taught to check import errors against the
> test module's list (if any).
> 
> In the case du jour, test_functools.py presumably wouldn't define that
> list, so that any ImportError it raised would be correctly treated as
> test failure.

What if we appended unexpected skips to the list of bad tests so that they get 
rerun in verbose mode and the return value becomes non-zero?

     print count(len(surprise), "skip"), \
                       "unexpected on", plat + ":"
     printlist(surprise)
     # Add the next line after the previous two in regrtest.py
     bad.extend(surprise)

(This happens after the count of failed tests has been printed, so we don't 
affect that output)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org