Unit testing data; What to test

[Steve Purcell]

John J. Lee wrote:
> [snip] But if the software is going to change
> a lot, isn't it a good idea to separate the tests from their input and
> expected result data?

Do whatever is clearest, but try to arrange the tests such that their data
doesn't change often. High-maintenance test cases are either bad test cases
or a sign that the tested code is being changed for no good reason.

> In fact, do unit tests often end up having to be rewritten as code is
> refactored?  Presumably yes, given the (low) level that unit tests are
> written at.

Refactoring should normally not break tests -- it means "now the code is
working and passing the tests, let's remove cruft".

XP zealots might (don't quote me) go as far as saying that any change that
breaks the tests is a feature change, not a refactoring.

Refactoring of tests is a separate activity.

> The second (third) one is vague: What / How should one test?  Discuss.

Test *everything*, and do it thoroughly!

If you want answers implying less work, there are probably some guidelines
somewhere...

Personally, I like the XP testing approach:

  1. Identify a requirement that some code has to do X
  2. Write a test where we try to do X, and check that it worked
  3. Keep adding or changing code until the test passes, and no other test
     breaks.

Finding a bug is equivalent to identifying a requirement that some code should
not do Y -- write a test, and code until it passes.

A complete set of tests thus written forms a catalogue of what the code is
guaranteed to do correctly; use of the tested code in any other manner is
not guaranteed to work.

Some people make the mistake of wasting a lot of testing effort on checking
that their code does something "sensible" if misused. There's no time-efficient
way to catch mistakes made by people who didn't read the manual.

Think of the tests as a combination of 'programmers manual' and 'example code'.

> I'm sure there must be good and bad ways to test

True enough. The bad way is to *not* test.  :-)

Really, the point is that time spent wondering or reading about how best to
test would be better spent simply writing as many tests as possible.

-Steve

-- 
Steve Purcell, Pythangelist
Get testing at http://pyunit.sourceforge.net/
Get servlets at http://pyserv.sourceforge.net/
"Even snakes are afraid of snakes." -- Steven Wright