[Python-ideas] Test Class setup and teardown in unittest

[Mark Roddy]

On Thu, Jan 21, 2010 at 2:47 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
>> On 21/01/2010 01:37, Robert Collins wrote:
>> > On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
>> >
>> >> Earlier this week on the Testing In Python list, there was a
>> >> discussion on how to execute a setup and/or teardown for a single test
>> >> class instead of for each test fixture on the class (see the 'setUp
>> >> and tearDown behavior' thread).  I have had to deal with situation
>> >> myself before, and I am obviously not the only one (since I did not
>> >> initiate the thread).  As such I'd like to propose adding a class
>> >> level setup and tear down method the unittest TestCase class.
>> >>
>> > I think that this is the wrong approach to the problem:
>> >   - class scope for such fixtures drives large test classes, reduces
>> > flexability
>> >
>>
>> Agreed.
>>
>> >   - doing it the rough way you suggest interacts in non obvious ways with
>> > test order randomisation
>> >
>>
>> Not currently in unittest, so not really an issue.
>
> Done by nearly every extended UI out there. I would hope that messing
> them up would count as an issue.
Agreed, messing/breaking with major unittest extensions should be
taken into serious consideration.

>
>> >   - it also interacts with concurrent testing by having shared objects
>> > (classes) hold state in a way that is not introspectable by the test
>> > running framework.
>> >
>>
>> Again, unittest doesn't do concurrent testing out of the box.
>
> testtools includes a concurrent TestResult which buffers activities from
> different executing test cases and outputs them to a single regular
> TestResult. Using that run() on TestSuite can be redefined to start some
> threads, giving each one a helper TestResult forwarding to the caller
> supplied TestResult, and finally treat self.__iter__ as a queue to
> dispatch to each worker thread.
Again, this should be taken into account to be easy to accommodate.
For in process parallelization, I think the best approach would be to
make it easy to extend to add the locking that would be necessary
(thread extensions must already be doing a decent amount of locking to
accomplish this).  Multi-process/machine distribute testing will take
some more analysis to determine how to accomidate.

>
>> How are shared fixtures like this not introspectable?
>
> How can you tell if there is shared state when setUpClass is being used?
>
>> As I see it.
>>
>> Advantages of setupClass and teardownClass:
>>
>> * Simple to implement
>> * Simple to explain
>> * A commonly requested feature
>> * Provided by other test frameworks in and outside of Python
>
> Mmm. It is commonly requested, but I'm not sure I entirely agree about
> the simple claims.
Since the discussion thus far has focused almost entirely on the
merits of the feature (and not on how to implement) I don't think it's
fair to make an assumption either way on how easy or not it would be
to implement.  I already have most of the implementation mapped out in
my head (of which I'm sure there will be issues to be resolved) and
could have a first try patch before the end of the weekend, but I
thought there should be more input before I go running off and doing
so.

>
>> Disadvantages:
>>
>> * Can encourage a poor style of testing, including monolithic test classes
>
> * Can limit concurrency
>
>> > I'd much much much rather see e.g. testresources integrated into the
>> >
>>
>> How is testresources superior?
>
> It separates concerns, which allows introspection for test runners. This
> is significantly more flexible, particularly when combined with test
> parameterisation.
This is a definite upside (especially for for complicated resources)
but it also increases the complexity reducing the ease of
understanding the test case by introducing a host of new semamtics
people will not (at first) be familiar with.  By the fact that people
have frequently requested a class setup/teardown illustrates that this
has a semantic they can easily understand (since it's understood
without even existing at the moment).

For complex resources that require complicated tear down and/or the
ability to invalidate to signal a need to recreate the resource (I
believe this is refered to being 'dirty' in the framework) the loss of
ease of understanding clearly out ways the additional layer, but for
simple resources (of which I think the attached is a great example) I
don't see this trade off being paid off.

So I don't really see this as having to be an either or situation.  I
don't think people should have to understand a whole additional API in
order to achieve this behavior, and I also wouldn't advocate people
being force to fit a round peg in a square whole for extremely large
resources.

The way I see it we don't have two competing implementations, but two
possible new features: class oriented setup/teardown functionality (as
already found in JUnit and Nunit), and a resource abstraction layer
(possible testresources itself, possibly something inspired by the
JUnit Rules previously mentioned, or a combination of the two).



>
>> Can you demonstrate this - in particular
>> what is the simplest example? The simplest example of setupClass would be:
>>
>> class SomeTest(unittest.TestCase):
>>      def setupClass(self):
>>          ... # setup shared fixture
>
>                     ^ this is buggy :)
> Its buggy because if it really does write to self, you now have what
> twisted did where all the test cases will be sharing one test object
> instance - which very frequently confuses and makes isolation between
> unrelated test state fragile.
>
> If its not writing to self, then you don't know what class object to
> write to (because you might be in a subclass) - that really should be a
> class method.
>
> And the fact that you have this confusion, as demonstrated here, is
> precisely why I don't agree about easy to implement and easy to
> understand.
>
>> > core allowing fixtures to be shared across test instances in a way that
>> > doesn't prohibit their use with concurrent testing, doesn't make it
>> > awkward to do it across multiple classes.
>>
>> Can you demonstrate how testresources solves these problems.
>
> Yes. resources are declared in a list, which allows the test framework,
> when parallelising, to group together tests that use the same resources
> when partitioning. Compare this with setUpClass where all tests in an
> inheritance hierarchy have to be grouped together (after excluding your
> base TestCase).
>
> You ask in another mail for a SSCE: here is one attached to this mail
> ('foo.py'). Run with 'python -m unittest foo'.
>
>> > I'm happy to make any
>> > [reasonable] changes (including license) to testresources to make it
>> > includable in the stdlib if that's of interest.
>> >
>>
>> That's not how contributing code to Python works. You need to provide a
>> signed contributor agreement to the PSF and then you specifically
>> license to the PSF any code you are contributing using one of a few
>> specific licenses. For new modules in the standard library you also need
>> to be willing to maintain the code.
>
> There is other code of mine in the standard library unittest module,
> done without a contributor agreement: so this is at best inconsistent. I
> mention license changes because that would be required to include it in
> the standard library (its currently a copyleft license) - and I'm not
> the sole author. Regardless, whatever process and actions are needed, I
> will do them, or arrange with other people where needed.
>
> -Rob
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>