[Python-Dev] setUpClass and setUpModule in unittest

[Michael Foord]

On 13/02/2010 04:01, Glyph Lefkowitz wrote:
[snipping some good points...]
>> Regarding the objection that setUp/tearDown for classes would run into
>> issues with subclassing, I propose to let the standard semantics of
>> subclasses do their job. Thus a subclass that overrides setUpClass or
>> tearDownClass is responsible for calling the base class's setUpClass
>> and tearDownClass (and the TestCase base class should provide empty
>> versions of both). The testrunner should only call setUpClass and
>> tearDownClass for classes that have at least one test that is
>> selected.
>>
>> Yes, this would mean that if a base class has a test method and a
>> setUpClass (and tearDownClass) method and a subclass also has a test
>> method and overrides setUpClass (and/or tearDown), the base class's
>> setUpClass and tearDown may be called twice. What's the big deal? If
>> setUpClass and tearDownClass are written properly they should support
>> this.
>>      
> Just to be clear: by "written properly" you mean, written as classmethods, storing their data only on 'cls', right?
>
>    

Heh, yes (answered several times in this thread already I think...).

>> If this behavior is undesired in a particular case, maybe what
>> was really meant were module-level setUp and tearDown, or the class
>> structure should be rearranged.
>>      
> There's also a bit of an open question here for me: if subclassing is allowed, and module-level setup and teardown are allowed, then what if I define a test class with test methods in module 'a', as well as module setup and teardown, then subclass it in 'b' which *doesn't* have setup and teardown... is the subclass in 'b' always assumed to depend on the module-level setup in 'a'?  Is there a way that it could be made not to if it weren't necessary? What if it stubs out all of its test methods?  In the case of classes you've got the 'cls' variable to describe the dependency and the shared state, but in the case of modules, inheritance doesn't create an additional module object to hold on to.
>
>    
This is also an interesting point. The 'naive' implementation, which I 
think I prefer, only runs the setUpModule of modules actually containing 
tests. Similarly setUpClass is only called on classes with actual tests, 
although they may call up to base class implementations.

This has a couple of consequences, particularly for setUpModule. It 
makes the basic rule:

* only use setUpModule for modules actually containing tests
* don't mix concrete TestCases (with tests) in the same module (using 
setUpModule) as base classes for other tests

The use case (that I can think of) that isn't supported (and of course 
all this needs to be documented):

* Having setUpModule / tearDownModule in modules that define base 
classes and expecting them to be called around all tests that inherit 
from those base classes

Having this in place makes the implementation simpler. If we explicitly 
document that this rule may change and so users shouldn't rely on 
setUpModule not being called for modules containing base classes, then 
we are free to rethink it later. Not having this restriction at all is 
possible, it just requires more introspection at TestSuite creation / 
ordering time.

Note that setUpClass on a base class maybe called several times if 
several base classes inherit and all call up to the base class 
implementation. As it will be a class method the cls argument will be 
different for each call.

Another question. If we are implementing TestCase.setUpClass as an 
additional test then should it be reported *even* if it is only the 
default (empty) implementation that is used?

The reason to have setUpClass implemented as a test is so that you can 
report the failure *before* you run all the tests. Lots of unit test 
users want a consistent number of tests every run - so we shouldn't 
insert an extra test only on fail. The other alternative is to report a 
setUpClass failure as part of the first test that depends on it - this 
makes the implementation more, complex (having a pseudo-test represent 
setUpClass / tearDownClass is convenient for ordering and isolating the 
'magic' in one place - the rest of the test running infrastructure 
doesn't need to know about setUpClass or Module).

If we do add a default setUpClass test for all TestCases it means extra 
noise for test runs that don't use the new feature. Saying no it 
shouldn't be shown means that we have to introspect test classes to see 
if they inherit setUpClass from TestCase or from some intermediate base 
class. Not hard just an extra complexity. One solution would be for 
TestCase *not* to have default implementations, but it is probably nicer 
for them to appear in the API.

I guess my preferred approach is to have a default implementation, but 
not to create pseudo-tests for them if they aren't used.


All the best,

Michael

> testresources very neatly sidesteps this problem by just providing an API to say "this test case depends on that test resource", without relying on the grouping of tests within classes, modules, or packages.  Of course you can just define a class-level or module-level resource and then have all your tests depend on it, which gives you the behavior of setUpClass and setUpModule in a more general way.
>
> -glyph
>
>    


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.