Case study: debugging failed assertRaises bug

[Ben Finney]

Steven D'Aprano <steve+comp.lang.python at pearwood.info> writes:

> I've just spent two hours banging my head against what I *thought* 
> (wrongly!) was a spooky action-at-a-distance bug in unittest, so I 
> thought I'd share it with anyone reading.

Much appreciated. I am experiencing a tear-my-hair-out test failure
which sounds exactly the same as the behaviour you describe, so you've
given me motivation to try again at figuring it out.

> (1) assertRaises REALLY needs a better error message. If not a custom
> message, at least it should show the result it got instead of an
> exception.

+1

Is this one of the many improvements in Python 3.2's ‘unittest’ that
Michael Foord presided over? Or are we still stuck with the terrible
behaviour of ‘assertRaises’?

> (4) Unit tests should test one thing, not four. I thought my loop over 
> four different "bad input" arguments was one conceptual test, and that 
> ended up biting me. If the failing test was "testBadArgTypeStr" I would 
> have realised what was going on much faster. 

For test cases where you want to run the same test against several sets
of data, and have each combination of test-plus-data run as an
independent test identified in the report, I highly recommend
‘testscenarios’ <URL:http://pypi.python.org/pypi/testscenarios>.

> (5) There's only so many tiny little tests you can write before your
> head explodes, so I'm going to keep putting "for arg in spam" loops in
> my tests. But now I know to unroll those suckers into individual tests
> on the first sign of trouble.

Use ‘testscenarios’ to do that without repeated code.

-- 
 \            “But it is permissible to make a judgment after you have |
  `\    examined the evidence. In some circles it is even encouraged.” |
_o__)                    —Carl Sagan, _The Burden of Skepticism_, 1987 |
Ben Finney