[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Yan Choi]

Kit Yan Choi <kit at kychoi.org> added the comment:

I think Python does differentiate "test error" and "test failure" such that a test outcome state can be one of these: success, failure, error, skipped. One could refine these to six: expected success, unexpected success, expected failure, unexpected failure, error, skipped.


For example, in the documentation for failureException:

    * failureException: determines which exception will be raised when
        the instance's assertion methods fail; test methods raising this
        exception will be deemed to have 'failed' rather than 'errored'.


Another evidence: unittest.runner.TextTestResult, there are methods called "addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and "addUnexpectedSuccess".


For example, this test outcome is marked as "FAILED":

def test(self):
    x = 1
    y = 2
    self.assertEqual(x + y, 4)


======================================================================
FAIL: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_main.py", line 9, in test
    self.assertEqual(x + y, 4)
AssertionError: 3 != 4


But the test outcome for this test is "ERROR":

    def test(self):
        x = 1
        y = 2 + z  # NameError                                                  
        self.assertEqual(x + y, 4)


======================================================================
ERROR: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_main.py", line 8, in test
    y = 2 + z  # NameError
NameError: global name 'z' is not defined


The issue here being "expectedFailure" converting "error" to "success", which is not expected, and is causing decorated tests to become unmaintained. While the rest of unittest differentiates "error" and "failure", expectedFailure does not. This is either a bug in the behaviour of expectedFailure, or a bug in the documentation for not being clear on the fact that unexpected error will be considered as expected failure (which I think is wrong).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38296>
_______________________________________