[Tutor] Unit testing in Python (3.3.0) for beginners

[Steven D'Aprano]

On Sun, Dec 08, 2013 at 11:22:37AM +0100, Rafael Knuth wrote:
> Hey there,
> 
> I struggle to understand what unit testing specifically means in
> practice and how to actually write unit tests for my code (my gut is
> telling me that it's a fairly important concept to understand).

In practice, unit testing is a way to check that your code does what you 
expect it should do. For every function and method (or at least, as many 
of them as you can), you should test that it does what you expect.

"What you expect" means the following:

- when given good input, does the function return the correct result?

- when given bad input, does the function fail the way it is supposed to 
fail? (e.g. raise an exception, return an error code).

Also, the unit test module is good for writing "regression tests". Every 
time you discover a bug in your code, before you fix the bug, you should 
write a test that demonstrates the existence of that bug. Then, if later 
changes to your program bring the bug back (this is called a 
regression), the test will fail and you will discover the regression 
straight away.

A simple example: suppose you have a method, Dog.bark(n) which is 
supposed to return "bark!" repeated some number of times, and if n is 
zero or negative it should return the empty string. Normally I put my 
unit tests in a separate file. So here's some unit tests for the 
Dog.bark method:


import unittest
from myprogram import Dog

class DogTest(unittest.TestCase):
    def test_bark_with_positive_argument(self):
        dog = Dog()
        self.assertEqual(dog.bark(1), "bark!")
        self.assertEqual(dog.bark(2), "bark! bark!")
        self.assertEqual(dog.bark(5), "bark! bark! bark! bark! bark!")

    def test_bark_with_zero_argument(self):
        dog = Dog()
        self.assertEqual(dog.bark(0), "")

    def test_bark_with_negative_argument(self):
        dog = Dog()
        for i in (-1, -2, -3):
            self.assertEqual(dog.bark(i), "")

    def test_walk(self):
        # test the walk method

    # and so on

if __name__ == '__main__':
    # Only run this part when running this file as a script.
    unittest.main()


(Alas, good tests often end up with tediously long names. Fortunately 
you don't have to read the test names very often.)

The tests all pass! Looks good. But here we see the problem with 
testing: tests can only show the existence of bugs, they cannot prove 
that there are no bugs at all. Oh well. At some point, some poor unlucky 
fellow discovers a bug in the Dog class, and reports it to you:

    Problem: Dog.bark() returns wrong result

    dog = Dog()
    dog.bark(-17)

    Expected result: ""
    Actual result: "meow!"


A fact of life -- you can never (well, hardly ever) test *every* 
possible result from your functions. Occasionally there will be 
surprises like this. Nevermind, this is the job of the program: fix the 
bugs as they are discovered.

So we add an extra test to the unit tests. This one is going to test for 
regressions, so we don't just modify the existing negative argument 
test, but we make sure this is a specific, individual test.

class DogTest(unittest.TestCase):
    [...other methods remain the same]

    def test_bark_with_negative_argument(self):
        dog = Dog()
        samples = [1, 2, 3, 4, 5] + random.sample(range(6, 10000), 20)
        for i in samples:
            self.assertEqual(dog.bark(-i), "")

    def test_regression_bug_id_92(self):
        dog = Dog()
        self.assertEqual(dog.bark(-17), "")


Notice that I've made the negative_argument case a lot more vigorous at 
testing the method. It's a matter of personal judgement how many cases 
you should check, but given that we've missed one bug, that's a good 
sign that we didn't check enough. So now instead of testing just a small 
handful, I test the first five negative values plus another 20 randomly 
choosen ones.

I also add a specific regression test to ensure that this bug can never 
happen again. In this example I've put the Bug Report ID in the test 
name, but that's not compulsary. The important thing is that you have a 
test for the bug.

If I run the DogTest now, test_regression_bug_id_92 fails, because I 
haven't fixed the bug. This proves that the test works as expected.

Now I fix the bug, re-run the DogTest, and hopefully everything passes. 
If so, I can be reasonably sure that there are no obvious bugs in the 
parts of the code I've actually tested.


[...]
> Also, I went through the "Beginning Test-Driven Development in Python"
> http://net.tutsplus.com/tutorials/python-tutorials/test-driven-development-in-python/
> but I have to admit I still don't fully understand how unit tests work
> in practice and how to write my own unit tests.

How unit tests work -- the unittest module is a big, complicated package 
that defines a whole lot of classes and methods. The idea is that the 
module defines a class that understands how to perform testing. It knows 
how to run "assertSomething" methods, or if you prefer, 
"failIfNotSomething" methods. It knows how to identify the test classes, 
how to identify the test methods inside those classes, how to run the 
tests, collect the results, display them to you, and report the final 
result of whether they all passed or some failed. There is a *lot* of 
smarts built into the unittest module.

To use unittest, of course you have to import it. Then you run the unit 
test main function:

unittest.main()


What this does is:

- identify the module you are running in;

- search that module for classes that inherit from TestCase (and 
  possibly a few others, but I always use TestCase);

- start collecting test results;

- for each test class, look for methods that start with "test";

- run those tests, and check whether they pass or fail;

- draw a pretty status diagram, showing a dot . for each passing 
  tests, F for failing tests, and E for errors;

- if there are any failing tests or errors, print up a report 
  showing them.


That's a lot of work, but it's all done for you by the unittest package. 
You just have to write the tests and call unittest.main().

You can read the source code to unittest:

http://hg.python.org/cpython/file/ad2cd599f1cf/Lib/unittest

but I warn you that it is a big, advanced package, and not especially 
the easiest to read. A lot of it is copied from a similar Java testing 
framework. Don't try to understand the whole thing at once, take it in 
little pieces.


> So, what I am
> specifically searching for is a very simple code sample which I can
> take apart and iterate through each component, and I was wondering if
> you are aware of resources that might be helpful?

ActiveState Python recipes. If you don't mind me linking to my own work:

http://code.activestate.com/recipes/users/4172944/

Raymond Hettinger's recipes are always worth learning from, although 
they are often quite advanced:

http://code.activestate.com/recipes/users/178123/

If you can afford it, I recommend you buy the cookbook:

http://shop.oreilly.com/product/9780596001674.do

although I'm not sure if that has been updated to Python 3 yet.



> My understanding of unit testing is that I have to embed my code into
> a test and then I have to define conditions under which my code is
> supposed to fail and pass. Is that assumption correct?

That's pretty much it. 



-- 
Steven