[Python-Dev] unittest2 plus class and module level fixtures in unittest

[Michael Foord]

Hello all,

unittest has seen quite a few new features since Python 2.6. For those 
of you who might have missed the announcements in other places, 
unittest2 is a backport of the new features in unittest to work with 
Python 2.4-2.6. It is already being used for the development of distutils2:

http://pypi.python.org/pypi/unittest2/ 
<http://pypi.python.org/pypi/unittest2/>

In other news, class and module fixtures (setUpClass / tearDownClass / 
setUpModule / tearDownModule) are now implemented in unittest (in trunk 
- not yet merged to py3k). These features are tested but I'm sure there 
are some lurking bugs or oddities, so please try them out. I have not 
yet added documentation for them; I'll pull it out from this email as a 
starting point.

I'd rather this thread didn't become *another* debate on the merit of 
these features, but perhaps that is too much to hope for.

Below are some notes on how class and module fixtures work.

Class and module level fixtures are implemented in unittest.TestSuite. 
When the test suite encounters a test from a new class then 
tearDownClass from the previous class (if there is one) is called, 
followed by setUpClass from the new class.

Similarly if a test is from a different module from the previous test 
then tearDownModule from the previous module is run, followed by 
setUpModule from the new module.

After all the tests have run the final tearDownClass and tearDownModule 
are run.

Note that shared fixtures do not play well with [potential] features 
like test parallelization and they break test isolation. They should be 
used with care.


setUpClass and tearDownClass
--------------------------------------

These must be implemented as class methods.

     import unittest

     class Test(unittest.TestCase):
         @classmethod
         def setUpClass(cls):
             cls._connection = createExpensiveConnectionObject()

         @classmethod
         def tearDownClass(cls):
             cls._connection.destroy()

If you want the setUpClass and tearDownClass on base classes called then 
you must call up to them yourself. The implementations in 
unittest2.TestCase are empty.

If an exception is raised during a setUpClass then the tests in the 
class are not run and the tearDownClass is not run. Skipped classes will 
not have setUpClass or tearDownClass run.

A setUpClass that raises a unittest2.SkipTest exception will currently 
be reported as an error rather than a skip (although the effect is the 
same). This will be fixed at some point in the future.


setUpModule and tearDownModule
--------------------------------------------

These should be implemented as functions.

     def setUpModule():
         createConnection()

     def tearDownModule():
         closeConnection()

If an exception is raised in a setUpModule then none of the tests in the 
module will be run and the tearDownModule will not be run.

A setUpModule that raises a unittest.SkipTest exception will currently 
be reported as an error rather than a skip (although the effect is the 
same). This will be fixed at some point in the future.


The Gory Details
----------------------

The default ordering of tests created by the unittest test loaders is to 
group all tests from the same modules and classes together. This will 
lead to setUpClass / setUpModule (etc) being called exactly once per 
class and module. If you randomize the order, so that tests from 
different modules and classes are adjacent to each other, then these 
shared fixture functions may be called multiple times in a single test run.

This particular issue was a point of some disussion. Shared fixtures are 
not intended to work with suites with non-standard (incompatible) 
ordering (a BaseTestSuite still exists for frameworks that don't want to 
support shared fixtures), so this is already a pathological situation. 
There are a couple of things unittest could do in this situation:

     1) Work out the best order to run the setUp and tearDowns, based on 
the position of the first and last test from each class / module in the 
suite.
     2) Run setUp and tearDowns appropriately if consecutive tests are 
from different classes and modules.

If your setUps do unrelated things then strategy 1 will be the best one, 
but (so long as your setups are repeatable) 2 will be fine just slower. 
If your setups do related things, for example pushing large test 
datasets into the same database, then having multiple setups active 
before the corresponding teardowns have been run will actually do the 
wrong thing and your tests will be broken. In addition strategy 1 relies 
on unrolling the test suite and introspecting all tests prior to the 
run, not all custom test suites implementations support this.

Given that strategy 2 was substantially easier to implement, and less 
likely to cause broken test runs, that was what I went with. It is of 
course possible to add alternative strategies, plus the metadata to 
support them, if there proves to be a substantial need for this. One of 
the major use cases is for test order randomization. It would be easy to 
add a 'randomize' method to the suite that randomizes the order of test 
modules in a suite, the order of test classes within modules and the 
order of tests within a class. This would allow for tests to be run in a 
random order whilst remaining compatible with shared fixtures.

If there are any exceptions raised during one of the shared fixture 
functions the test is reported as an error. Because there is no 
corresponding test instance an _ErrorHolder object (that has the same 
interface as a TestCase) is created to represent the error. If you are 
just using the standard unittest test runner then this detail doesn't 
matter, but if you are a framework author it may be relevant.



I have a few minor features I'd like to add to unittest before 2.7 beta 
- standard out capturing, fail fast and ctrl-c handling (first ctrl-c 
will wait for current test to exit, second will force an exit - results 
will still be reported). After that the *really* big problem for 
unittest is the extensibility story.

I'll be working on extensibility points for unittest that allow people 
to implement extensions without having to override functionality. This 
will take a fair bit of discussion and experimentation, so I will 
probably trial this in unittest2 first. It would be nice to have 
something in place for the release of 3.2, but that may be ambitious.

All the best,

Michael Foord

-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100308/0e816607/attachment.html>