[Python-Dev] Disable tests in unittest (issue3202)

[Jonathan Lange]

On Thu, Jun 26, 2008 at 7:13 AM, Justin Mazzola Paluska <jmp at mit.edu> wrote:
> Hi,
>
...
> The idea behind the patch is that it's sometimes useful to disable
> tests without removing them from the TestCase.  For example, a
> co-worker and I have a module with a few tests that will fail for the
> forseeable future because we haven't had a chance to implement the
> features the tests are testing.  The tracebacks for these "not
> implemented yet" tests obscure real failures in other tests.
...
> The patch in issue3202 implements the decorator @disabled and a few
> modifications to the classes in the unittest module.
>
> What does the Python hivemind think?  I'm also open to enhancing it if
> the list has any ideas.

I think it's really worth looking at the approach that bzrlib's tests
take here (see bzrlib.tests.ExtendedTestResult and the
out-of-date-but-helpful http://bazaar-vcs.org/BzrExtendTestSuite)

Instead of disabling the tests, their practice is to run the tests but
have them raise KnownFailure. When they write a test that they know
will fail, they have it raise this exception. The extended TestResult
object catches this exception in addFailure and reports it
appropriately.[1]

So, when there's a test that is expected to fail, they do something like this:

    def test_foo(self):
          x = get_some_value()
          self.expectFailure('get_some_value is bogus', self.assertEqual, x, 42)

Using KnownFailure is better than disabling because you'll be able to
tell if the test suddenly and mysteriously passes.

I can see that disabling still has some value, but I'd rather have
KnownFailure first.

jml

[1] They use a similar strategy for skipping tests based on platform.
(You can't test Bazaar's symlink support on Windows, for example).