Hi, I just reported issue3202 in the bugracker (http://bugs.python.org/issue3202) with a patch that implements a way to disable unittest.TestCases using a decorator. Benjamin Peterson suggested that I bring up the issue on python-dev. The idea behind the patch is that it’s sometimes useful to disable tests without removing them from the TestCase. For example, a co-worker and I have a module with a few tests that will fail for the forseeable future because we haven’t had a chance to implement the features the tests are testing. The tracebacks for these “not implemented yet” tests obscure real failures in other tests. Normally, we’d just rename the failing methods from something starting with “test” to something like “_test”, so unittest ignores them. However, doing so removes all traces of the tests when you re-run the test suite, so the disabled tests are easy to forget. Instead, issue3202 allows you to write something like: from unittest import main, TestCase, disabled class SampleTest(TestCase): def test_addition(self): self.assertEqual(2, 1+1) def test_broken(self): # this is a real failure in our code self.assertEqual(5, 2+2) @disabled def test_not_implemented(self): # Test of a feature not implemented yet. doit() if __name__ == '__main__': main() which has one test disabled. Running the test suite outputs %python sample_tests2.py .FD ====================================================================== FAIL: test_broken (__main__.SampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "sample_tests2.py", line 12, in test_broken self.assertEqual(5, 2+2) AssertionError: 5 != 4 ---------------------------------------------------------------------- Ran 3 tests in 0.001s FAILED (failures=1) showing only the real failure and a simple “D” for the disabled test. The disabled test is still there, but doesn’t get in your way. JUnit4 has a similar decorator for its tests, @Ignore. The patch in issue3202 implements the decorator @disabled and a few modifications to the classes in the unittest module. What does the Python hivemind think? I’m also open to enhancing it if the list has any ideas. —Justin
On Thu, Jun 26, 2008 at 7:13 AM, Justin Mazzola Paluska
Hi,
...
The idea behind the patch is that it's sometimes useful to disable tests without removing them from the TestCase. For example, a co-worker and I have a module with a few tests that will fail for the forseeable future because we haven't had a chance to implement the features the tests are testing. The tracebacks for these "not implemented yet" tests obscure real failures in other tests. ... The patch in issue3202 implements the decorator @disabled and a few modifications to the classes in the unittest module.
What does the Python hivemind think? I'm also open to enhancing it if the list has any ideas.
I think it's really worth looking at the approach that bzrlib's tests take here (see bzrlib.tests.ExtendedTestResult and the out-of-date-but-helpful http://bazaar-vcs.org/BzrExtendTestSuite) Instead of disabling the tests, their practice is to run the tests but have them raise KnownFailure. When they write a test that they know will fail, they have it raise this exception. The extended TestResult object catches this exception in addFailure and reports it appropriately.[1] So, when there's a test that is expected to fail, they do something like this: def test_foo(self): x = get_some_value() self.expectFailure('get_some_value is bogus', self.assertEqual, x, 42) Using KnownFailure is better than disabling because you'll be able to tell if the test suddenly and mysteriously passes. I can see that disabling still has some value, but I'd rather have KnownFailure first. jml [1] They use a similar strategy for skipping tests based on platform. (You can't test Bazaar's symlink support on Windows, for example).
On Thu, Jun 26, 2008 at 10:51:49AM +1000, Jonathan Lange wrote:
On Thu, Jun 26, 2008 at 7:13 AM, Justin Mazzola Paluska
wrote: I think it's really worth looking at the approach that bzrlib's tests take here (see bzrlib.tests.ExtendedTestResult and the out-of-date-but-helpful http://bazaar-vcs.org/BzrExtendTestSuite) Instead of disabling the tests, their practice is to run the tests but have them raise KnownFailure. When they write a test that they know will fail, they have it raise this exception. The extended TestResult object catches this exception in addFailure and reports it appropriately.[1]
I wasn’t aware of bzrlib’s extentions to unittest — many of them,
including KnownFailure, TestSkipped,
bzrlib.test.ExtendedTestResult.expectFailure, and the sundry list of
bzrlib.test.TestCase.assert* look useful enough to belong in the
standard library.
I can see that disabling still has some value, but I'd rather have KnownFailure first.
I think disabling and the bzr extensions have different use cases. KnownFailure lets someone with an intimate knowledge of the the failing test case write down that they expect the test to fail. The disabled decorator lets someone less familiar with the test cases to easily disable tests that are getting in their way, without needing to understand the body of the test function (or worse, mis-understanding it and breaking it in a hard-to-understand way). I agree that in the ideal world, we’d like to have both. In fact, if we had the bzr extensions, it’d be easy to write the disabled decorator in terms of their KnownFailure: def disabled(func): @wraps(func): def wrapper(*args, **kwargs): raise KnownFailure("%s disabled" % func.__name__) return wrapper However, unless Python’s unittest is going to incorporate the useful features of bzrlib’s TestCase object[*], it probably makes sense to implement the disabled feature separately. —Justin [*] Perhaps in the long term, this is the right thing to do as unittest seems to have fallen behind in useful feature count as its “conceptual” parent JUnit.
On Fri, Jun 27, 2008 at 4:57 AM, Justin Mazzola Paluska
I wasn't aware of bzrlib's extentions to unittest — many of them, including KnownFailure, TestSkipped, bzrlib.test.ExtendedTestResult.expectFailure, and the sundry list of bzrlib.test.TestCase.assert* look useful enough to belong in the standard library.
I agree 100%. Twisted came up with some of the same things independently, and for a lot of things, we've stolen bzrlib's ideas. I think I've filed a couple of tickets with patches along these lines. I'll try to get my act in gear and finish the rest of the patches to unittest and get them landed. jml
2008/6/25 Justin Mazzola Paluska
The idea behind the patch is that it's sometimes useful to disable tests without removing them from the TestCase. For example, a co-worker and I have a module with a few tests that will fail for the forseeable future because we haven't had a chance to implement the features the tests are testing. The tracebacks for these "not implemented yet" tests obscure real failures in other tests.
What about TestSkipped? I thought that raising test_support.TestSkipped should behave like this: you're saying that you're actually NOT executing the tests, but you say that they are there. But a few days ago I needed to comment out a test that previously was raising this, because it made a buildbot to go red. How this should behave? Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/
On Wed, Jun 25, 2008 at 7:52 PM, Facundo Batista
What about TestSkipped? I thought that raising test_support.TestSkipped should behave like this: you're saying that you're actually NOT executing the tests, but you say that they are there.
I think he's talking about outside core Python development (putting it into unittest). -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1."
participants (4)
-
Benjamin Peterson
-
Facundo Batista
-
Jonathan Lange
-
Justin Mazzola Paluska