[Python-Dev] unittest Suggestions

[Sebastian Rittau]

[I just saw the other post about unit testing, while I was writing this.
A strange conincidence.]

Hello,

since this is the first time I post to this list, I'd like to introduce
myself briefly. As you can see my name is Sebastian. I am working as a
software developer both as a professional and as a hobbyist. During the
years I've used many programming languages, but in the last few years
I've fallen in love with Python. (Blame Martin v. Löwis for that - I
started using Python after participating in a seminar held by him at
university.)

I'd like to propose a few changes to the unittest module. These changes
are supposed to ease usage, enable future changes and enhancements, and
bring unittest more in line with other modern xUnit variants (such as
JUnit 4 and NUnit). I have implemented most of those changes in a custom
library that hacks around the current unittest behaviour. I've published
the relevant package at [1]. All proposed changes are (mostly) backward
compatible.

Change unittest into a package
------------------------------

Currently unittest is a module. Changing it into a package would enable
us to separate concerns and move code out into separate modules. For
example the unittest.asserts proposal below requires this change. If a
mocking framework would be included in the Python standard library at
some point, it could go into unittest.mock etc.

@test, @before, and @after decorators
-------------------------------------

Currently unittest uses "magic" names starting with "test_" for detecting
the test methods in a test case class. Following the "explicit is better
than implicit" guideline, I propose to add a @test decorator to
explicitly mark test methods:

  class MyTest(TestCase):

      @test
      def attributes(self):
          """Do stuff."""

I've been bitten in the past by naming methods test_xyz that were not
supposed to be called by the test runner. For now those methods still
need to be recognized, since they are widely used. But enabling (and
recommending) an alternative is a first step.

The decorator syntax makes is possible to introduce alternative
decorators as well, like @disabled_test. In the example package I
included a @performance_test decorator that takes a time argument where
the test fails if it takes more than a given amount of time.

Similarily I recommend to introduce @before and @after decorators to
supplement the setUp and tearDown methods. The decorators were named
after the decorators in JUnit 4, although I wonder whether @setup and
@teardown would be a better name. @before and @after methods are called
in any order, but @before methods in a super class are guaranteed to be
called before @before methods in a derived class, while it's vice versa
with @after methods. Introducing these has the added benefit of being
able to get rid of the mixedCase setUp and tearDown methods in my own
code.

unittest.asserts
----------------

Currently all assertions reside as methods on the TestCase class. This
means that all testing functions, including utility functions, need to be
on a TestCase-derived class. This seems wrong to me, especially since it
combines different concerns on a class: Testing the functionality of a
SUT, running a test suite, and providing test functions. This proposal
will not try to separate the first two concerns, but only the third one.
In the long term I think it's worthwhile to make tests work on any
class, not only on classes derived from TestCase, but as I said, this is
not part of this proposal.

JUnit 4 moved the assertions to a static class in a separate module. In
Python I'd propose to use functions in a separate module. A typical test
suite would then look like this:

  from unittest import TestCase
  from unittest.asserts import *

  class FooTest(TestCase):

      """Tests for the Foo class."""

      @before
      def setup(self):
          self._sut = create_an_object_to_test()

      @test
      def construct__some_attribute(self):
          assert_equals("foobar", self._sut.some_attribute)

      ...

Again, as a side effect this removes inconsistent naming (mixedCase and
three different version of each test) from my tests. There is one
regression, though. Currently it's possible to change the assertion
exception (which defaults to AssertionError) by setting an attribute of
the TestCase instance. I am not sure how relevant that is, though. I've
never used this feature myself, not even when testing the unittest
module and its extensions, and this approach wouldn't work anyway, if
you use classes with utility assertions etc.

Anyway, I am happy about any comments.

 - Sebastian

[1] http://www.rittau.biz/~srittau/unittest, though I will move it to
    http://www.rittau.org/python/unittest, once I'm home.