[Python-Dev] Reporting unexpected import failures as test failures in regrtest.py
ncoghlan at gmail.com
Wed May 31 12:33:10 CEST 2006
Some background for those not watching python-checkins:
I neglected to do "svn add" for the new functools Python module when
converting functional->functools. The buildbots stayed green because the
ImportError triggered by the line "import functools" in test_functools was
treated as a TestSkipped by regrtest.py.
Georg noticed the file missing from the checkin message, but this is the
second time I (and the buildbots) have missed a regression due to this
behaviour. (As I recall, last time I checked in some broken code because I
didn't notice the additional entry appearing in the list of unexpected skips
in my local testing)
Tim Peters wrote:
> [Nick Coghlan]
>> ... (we should probably do something about that misleading ImportError ->
>> TestSkipped -> green buildbot behaviour. . . )
> I looked at that briefly a few weeks back and gave up. Seemed the
> sanest thing was to entirely stop treating ImportError as "test
> skipped", and rewrite tests that legimately _may_ get skipped to catch
> expected ImportErrors and change them to TestSkipped themselves.
> A bit of framework might help; e.g., a test that expects to get
> skipped due to failing imports on some platforms could define a
> module-level list bound to a conventional name containing the names of
> the modules whose import failure should be treated as TestSkipped, and
> then regrtest.py could be taught to check import errors against the
> test module's list (if any).
> In the case du jour, test_functools.py presumably wouldn't define that
> list, so that any ImportError it raised would be correctly treated as
> test failure.
What if we appended unexpected skips to the list of bad tests so that they get
rerun in verbose mode and the return value becomes non-zero?
print count(len(surprise), "skip"), \
"unexpected on", plat + ":"
# Add the next line after the previous two in regrtest.py
(This happens after the count of failed tests has been printed, so we don't
affect that output)
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
More information about the Python-Dev