Avoiding cascading test failures
When I was fixing tests failing in the py3k branch, I found the number duplicate failures annoying. Often, a single bug, in an important method or function, caused a large number of testcase to fail. So, I thought of a simple mechanism for avoiding such cascading failures. My solution is to add a notion of dependency to testcases. A typical usage would look like this: @depends('test_getvalue') def test_writelines(self): ... memio.writelines([buf] * 100) self.assertEqual(memio.getvalue(), buf * 100) ... Here, running the test is pointless if test_getvalue fails. So by making test_writelines depends on the success of test_getvalue, we can ensure that the report won't be polluted with unnecessary failures. Also, I believe this feature will lead to more orthogonal tests, since it encourages the user to write smaller test with less dependencies. I wrote an example implementation (included below) as a proof of concept. If the idea get enough support, I will implement it and add it to the unittest module. -- Alexandre class CycleError(Exception): pass class TestCase: def __init__(self): self.graph = {} tests = [x for x in dir(self) if x.startswith('test')] for testname in tests: test = getattr(self, testname) if hasattr(test, 'deps'): self.graph[testname] = test.deps else: self.graph[testname] = set() def run(self): graph = self.graph toskip = set() msgs = [] while graph: # find tests without any pending dependencies source = [test for test, deps in graph.items() if not deps] if not source: raise CycleError for testname in source: if testname in toskip: msgs.append("%s... skipped" % testname) resolvedeps(graph, testname) del graph[testname] continue test = getattr(self, testname) try: test() except AssertionError: toskip.update(getrevdeps(graph, testname)) msgs.append("%s... failed" % testname) except: toskip.update(getrevdeps(graph, testname)) msgs.append("%s... error" % testname) else: msgs.append("%s... ok" % testname) finally: resolvedeps(graph, testname) del graph[testname] for msg in sorted(msgs): print(msg) def getrevdeps(graph, testname): """Return the reverse depencencies of a test""" rdeps = set() for x in graph: if testname in graph[x]: rdeps.add(x) if rdeps: # propagate depencencies recursively for x in rdeps.copy(): rdeps.update(getrevdeps(graph, x)) return rdeps def resolvedeps(graph, testname): for test in graph: if testname in graph[test]: graph[test].remove(testname) def depends(*args): def decorator(test): if hasattr(test, 'deps'): test.deps.update(args) else: test.deps = set(args) return test return decorator class MyTest(TestCase): @depends('test_foo') def test_nah(self): pass @depends('test_bar', 'test_baz') def test_foo(self): pass @depends('test_tin') def test_bar(self): self.fail() def test_baz(self): self.error() def test_tin(self): pass def error(self): raise ValueError def fail(self): raise AssertionError if __name__ == '__main__': t = MyTest() t.run()
On Wed, Aug 22, 2007 at 07:44:02PM -0400, Alexandre Vassalotti wrote:
I like this idea. Be sure to have an option to ignore dependancies and run all tests. Also when skipping tests because a depedancy failed have unittest print out an indication that a test was skipped due to a dependancy rather than silently running fewer tests. Otherwise it could be deceptive and appear that only one test was affected. Greg
On 8/25/07, Gregory P. Smith <greg@krypto.org> wrote:
I like this idea.
Yay! Now, I ain't the only one. ;)
Be sure to have an option to ignore dependancies and run all tests.
Yes, I planned to add a such option.
However, that was never planned. I added the ignore_dependencies option. Also, I fixed the sub-optimal dependency resolution algorithm that was in my original example implementation. -- Alexandre --- dep.py.old 2007-08-25 19:54:27.000000000 -0400 +++ dep.py 2007-08-25 20:02:55.000000000 -0400 @@ -2,8 +2,9 @@ class CycleError(Exception): pass +class TestGraph: -class TestCase: + ignore_dependencies = False def __init__(self): self.graph = {} @@ -19,16 +20,16 @@ graph = self.graph toskip = set() msgs = [] - while graph: + if self.ignore_dependencies: + for test in graph: + graph[test].clear() # find tests without any pending dependencies - source = [test for test, deps in graph.items() if not deps] - if not source: - raise CycleError - for testname in source: + queue = [test for test, deps in graph.items() if not deps] + while queue: + testname = queue.pop() if testname in toskip: msgs.append("%s... skipped" % testname) - resolvedeps(graph, testname) - del graph[testname] + queue.extend(resolve(graph, testname)) continue test = getattr(self, testname) try: @@ -42,8 +43,9 @@ else: msgs.append("%s... ok" % testname) finally: - resolvedeps(graph, testname) - del graph[testname] + queue.extend(resolve(graph, testname)) + if graph: + raise CycleError for msg in sorted(msgs): print(msg) @@ -60,10 +62,15 @@ rdeps.update(getrevdeps(graph, x)) return rdeps - def resolvedeps(graph, testname): +def resolve(graph, testname): + toqueue = [] for test in graph: if testname in graph[test]: graph[test].remove(testname) + if not graph[test]: + toqueue.append(test) + del graph[testname] + return toqueue def depends(*args): def decorator(test): @@ -75,7 +82,9 @@ return decorator -class MyTest(TestCase): +class MyTest(TestGraph): + + ignore_dependencies = True @depends('test_foo') def test_nah(self):
On 8/22/07, Alexandre Vassalotti <alexandre@peadrop.com> wrote:
This definitely seems like a neat idea. Some thoughts: * How do you deal with dependencies that cross test modules? Say test A depends on test B, how do we know whether it's worthwhile to run A if B hasn't been run yet? It looks like you run the test anyway (I haven't studied the code closely), but that doesn't seem ideal. * This might be implemented in the wrong place. For example, the [x for x in dir(self) if x.startswith('test')] you do is most certainly better-placed in a custom TestLoader implementation. That might also make it possible to order tests based on their dependency graph, which could be a step toward addressing the above point. But despite that, I think it's a cool idea and worth pursuing. Could you set up a branch (probably of py3k) so we can see how this plays out in the large? Collin Winter
On 8/28/07, Collin Winter <collinw@gmail.com> wrote:
I am not sure what you mean by "test modules". Do you mean module in the Python sense, or like a test-case class?
That certainly is a good suggestion. I am not sure yet how I will implement my idea in the unittest module. However, I pretty sure that it will be quite different from my prototype.
Sure. I need to finish merging pickle and cPickle for Py3k before tackling this project, though. -- Alexandre
On Wed, Aug 22, 2007 at 07:44:02PM -0400, Alexandre Vassalotti wrote:
I like this idea. Be sure to have an option to ignore dependancies and run all tests. Also when skipping tests because a depedancy failed have unittest print out an indication that a test was skipped due to a dependancy rather than silently running fewer tests. Otherwise it could be deceptive and appear that only one test was affected. Greg
On 8/25/07, Gregory P. Smith <greg@krypto.org> wrote:
I like this idea.
Yay! Now, I ain't the only one. ;)
Be sure to have an option to ignore dependancies and run all tests.
Yes, I planned to add a such option.
However, that was never planned. I added the ignore_dependencies option. Also, I fixed the sub-optimal dependency resolution algorithm that was in my original example implementation. -- Alexandre --- dep.py.old 2007-08-25 19:54:27.000000000 -0400 +++ dep.py 2007-08-25 20:02:55.000000000 -0400 @@ -2,8 +2,9 @@ class CycleError(Exception): pass +class TestGraph: -class TestCase: + ignore_dependencies = False def __init__(self): self.graph = {} @@ -19,16 +20,16 @@ graph = self.graph toskip = set() msgs = [] - while graph: + if self.ignore_dependencies: + for test in graph: + graph[test].clear() # find tests without any pending dependencies - source = [test for test, deps in graph.items() if not deps] - if not source: - raise CycleError - for testname in source: + queue = [test for test, deps in graph.items() if not deps] + while queue: + testname = queue.pop() if testname in toskip: msgs.append("%s... skipped" % testname) - resolvedeps(graph, testname) - del graph[testname] + queue.extend(resolve(graph, testname)) continue test = getattr(self, testname) try: @@ -42,8 +43,9 @@ else: msgs.append("%s... ok" % testname) finally: - resolvedeps(graph, testname) - del graph[testname] + queue.extend(resolve(graph, testname)) + if graph: + raise CycleError for msg in sorted(msgs): print(msg) @@ -60,10 +62,15 @@ rdeps.update(getrevdeps(graph, x)) return rdeps - def resolvedeps(graph, testname): +def resolve(graph, testname): + toqueue = [] for test in graph: if testname in graph[test]: graph[test].remove(testname) + if not graph[test]: + toqueue.append(test) + del graph[testname] + return toqueue def depends(*args): def decorator(test): @@ -75,7 +82,9 @@ return decorator -class MyTest(TestCase): +class MyTest(TestGraph): + + ignore_dependencies = True @depends('test_foo') def test_nah(self):
On 8/22/07, Alexandre Vassalotti <alexandre@peadrop.com> wrote:
This definitely seems like a neat idea. Some thoughts: * How do you deal with dependencies that cross test modules? Say test A depends on test B, how do we know whether it's worthwhile to run A if B hasn't been run yet? It looks like you run the test anyway (I haven't studied the code closely), but that doesn't seem ideal. * This might be implemented in the wrong place. For example, the [x for x in dir(self) if x.startswith('test')] you do is most certainly better-placed in a custom TestLoader implementation. That might also make it possible to order tests based on their dependency graph, which could be a step toward addressing the above point. But despite that, I think it's a cool idea and worth pursuing. Could you set up a branch (probably of py3k) so we can see how this plays out in the large? Collin Winter
On 8/28/07, Collin Winter <collinw@gmail.com> wrote:
I am not sure what you mean by "test modules". Do you mean module in the Python sense, or like a test-case class?
That certainly is a good suggestion. I am not sure yet how I will implement my idea in the unittest module. However, I pretty sure that it will be quite different from my prototype.
Sure. I need to finish merging pickle and cPickle for Py3k before tackling this project, though. -- Alexandre
participants (3)
-
Alexandre Vassalotti -
Collin Winter -
Gregory P. Smith