On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
On 21/01/2010 01:37, Robert Collins wrote:
On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
Earlier this week on the Testing In Python list, there was a discussion on how to execute a setup and/or teardown for a single test class instead of for each test fixture on the class (see the 'setUp and tearDown behavior' thread). I have had to deal with situation myself before, and I am obviously not the only one (since I did not initiate the thread). As such I'd like to propose adding a class level setup and tear down method the unittest TestCase class.
I think that this is the wrong approach to the problem: - class scope for such fixtures drives large test classes, reduces flexability
Agreed.
- doing it the rough way you suggest interacts in non obvious ways with test order randomisation
Not currently in unittest, so not really an issue.
Done by nearly every extended UI out there. I would hope that messing them up would count as an issue.
- it also interacts with concurrent testing by having shared objects (classes) hold state in a way that is not introspectable by the test running framework.
Again, unittest doesn't do concurrent testing out of the box.
testtools includes a concurrent TestResult which buffers activities from different executing test cases and outputs them to a single regular TestResult. Using that run() on TestSuite can be redefined to start some threads, giving each one a helper TestResult forwarding to the caller supplied TestResult, and finally treat self.__iter__ as a queue to dispatch to each worker thread.
How are shared fixtures like this not introspectable?
How can you tell if there is shared state when setUpClass is being used?
As I see it.
Advantages of setupClass and teardownClass:
* Simple to implement * Simple to explain * A commonly requested feature * Provided by other test frameworks in and outside of Python
Mmm. It is commonly requested, but I'm not sure I entirely agree about the simple claims.
Disadvantages:
* Can encourage a poor style of testing, including monolithic test classes
* Can limit concurrency
I'd much much much rather see e.g. testresources integrated into the
How is testresources superior?
It separates concerns, which allows introspection for test runners. This is significantly more flexible, particularly when combined with test parameterisation.
Can you demonstrate this - in particular what is the simplest example? The simplest example of setupClass would be:
class SomeTest(unittest.TestCase): def setupClass(self): ... # setup shared fixture
^ this is buggy :) Its buggy because if it really does write to self, you now have what twisted did where all the test cases will be sharing one test object instance - which very frequently confuses and makes isolation between unrelated test state fragile. If its not writing to self, then you don't know what class object to write to (because you might be in a subclass) - that really should be a class method. And the fact that you have this confusion, as demonstrated here, is precisely why I don't agree about easy to implement and easy to understand.
core allowing fixtures to be shared across test instances in a way that doesn't prohibit their use with concurrent testing, doesn't make it awkward to do it across multiple classes.
Can you demonstrate how testresources solves these problems.
Yes. resources are declared in a list, which allows the test framework, when parallelising, to group together tests that use the same resources when partitioning. Compare this with setUpClass where all tests in an inheritance hierarchy have to be grouped together (after excluding your base TestCase). You ask in another mail for a SSCE: here is one attached to this mail ('foo.py'). Run with 'python -m unittest foo'.
I'm happy to make any [reasonable] changes (including license) to testresources to make it includable in the stdlib if that's of interest.
That's not how contributing code to Python works. You need to provide a signed contributor agreement to the PSF and then you specifically license to the PSF any code you are contributing using one of a few specific licenses. For new modules in the standard library you also need to be willing to maintain the code.
There is other code of mine in the standard library unittest module, done without a contributor agreement: so this is at best inconsistent. I mention license changes because that would be required to include it in the standard library (its currently a copyleft license) - and I'm not the sole author. Regardless, whatever process and actions are needed, I will do them, or arrange with other people where needed. -Rob