Re: [Python-Dev] Avoiding cascading test failures

On 8/28/07, Collin Winter wrote:

On 8/22/07, Alexandre Vassalotti wrote:

...

When I was fixing tests failing in the py3k branch, I found the number duplicate failures annoying. Often, a single bug, in an important method or function, caused a large number of testcase to fail. So, I thought of a simple mechanism for avoiding such cascading failures.

My solution is to add a notion of dependency to testcases. A typical usage would look like this:

@depends('test_getvalue') def test_writelines(self): ... memio.writelines([buf] * 100) self.assertEqual(memio.getvalue(), buf * 100) ...

This definitely seems like a neat idea. Some thoughts:

* How do you deal with dependencies that cross test modules? Say test A depends on test B, how do we know whether it's worthwhile to run A if B hasn't been run yet? It looks like you run the test anyway (I haven't studied the code closely), but that doesn't seem ideal.

I am not sure what you mean by "test modules". Do you mean module in the Python sense, or like a test-case class?

* This might be implemented in the wrong place. For example, the [x for x in dir(self) if x.startswith('test')] you do is most certainly better-placed in a custom TestLoader implementation.

That certainly is a good suggestion. I am not sure yet how I will implement my idea in the unittest module. However, I pretty sure that it will be quite different from my prototype.

But despite that, I think it's a cool idea and worth pursuing. Could you set up a branch (probably of py3k) so we can see how this plays out in the large?

Sure. I need to finish merging pickle and cPickle for Py3k before tackling this project, though. -- Alexandre