On Thu, Jan 21, 2010 at 2:47 PM, Robert Collins <robertc@robertcollins.net> wrote:
On Thu, 2010-01-21 at 11:42 +0000, Michael Foord wrote:
On 21/01/2010 01:37, Robert Collins wrote:
On Wed, 2010-01-20 at 19:38 -0500, Mark Roddy wrote:
Earlier this week on the Testing In Python list, there was a discussion on how to execute a setup and/or teardown for a single test class instead of for each test fixture on the class (see the 'setUp and tearDown behavior' thread). I have had to deal with situation myself before, and I am obviously not the only one (since I did not initiate the thread). As such I'd like to propose adding a class level setup and tear down method the unittest TestCase class.
I think that this is the wrong approach to the problem: - class scope for such fixtures drives large test classes, reduces flexability
Agreed.
- doing it the rough way you suggest interacts in non obvious ways with test order randomisation
Not currently in unittest, so not really an issue.
Done by nearly every extended UI out there. I would hope that messing them up would count as an issue. Agreed, messing/breaking with major unittest extensions should be taken into serious consideration.
- it also interacts with concurrent testing by having shared objects (classes) hold state in a way that is not introspectable by the test running framework.
Again, unittest doesn't do concurrent testing out of the box.
testtools includes a concurrent TestResult which buffers activities from different executing test cases and outputs them to a single regular TestResult. Using that run() on TestSuite can be redefined to start some threads, giving each one a helper TestResult forwarding to the caller supplied TestResult, and finally treat self.__iter__ as a queue to dispatch to each worker thread.
Again, this should be taken into account to be easy to accommodate. For in process parallelization, I think the best approach would be to make it easy to extend to add the locking that would be necessary (thread extensions must already be doing a decent amount of locking to accomplish this). Multi-process/machine distribute testing will take some more analysis to determine how to accomidate.
How are shared fixtures like this not introspectable?
How can you tell if there is shared state when setUpClass is being used?
As I see it.
Advantages of setupClass and teardownClass:
* Simple to implement * Simple to explain * A commonly requested feature * Provided by other test frameworks in and outside of Python
Mmm. It is commonly requested, but I'm not sure I entirely agree about the simple claims.
Since the discussion thus far has focused almost entirely on the merits of the feature (and not on how to implement) I don't think it's fair to make an assumption either way on how easy or not it would be to implement. I already have most of the implementation mapped out in my head (of which I'm sure there will be issues to be resolved) and could have a first try patch before the end of the weekend, but I thought there should be more input before I go running off and doing so.
Disadvantages:
* Can encourage a poor style of testing, including monolithic test classes
* Can limit concurrency
I'd much much much rather see e.g. testresources integrated into the
How is testresources superior?
It separates concerns, which allows introspection for test runners. This is significantly more flexible, particularly when combined with test parameterisation.
This is a definite upside (especially for for complicated resources) but it also increases the complexity reducing the ease of understanding the test case by introducing a host of new semamtics people will not (at first) be familiar with. By the fact that people have frequently requested a class setup/teardown illustrates that this has a semantic they can easily understand (since it's understood without even existing at the moment). For complex resources that require complicated tear down and/or the ability to invalidate to signal a need to recreate the resource (I believe this is refered to being 'dirty' in the framework) the loss of ease of understanding clearly out ways the additional layer, but for simple resources (of which I think the attached is a great example) I don't see this trade off being paid off. So I don't really see this as having to be an either or situation. I don't think people should have to understand a whole additional API in order to achieve this behavior, and I also wouldn't advocate people being force to fit a round peg in a square whole for extremely large resources. The way I see it we don't have two competing implementations, but two possible new features: class oriented setup/teardown functionality (as already found in JUnit and Nunit), and a resource abstraction layer (possible testresources itself, possibly something inspired by the JUnit Rules previously mentioned, or a combination of the two).
Can you demonstrate this - in particular what is the simplest example? The simplest example of setupClass would be:
class SomeTest(unittest.TestCase): def setupClass(self): ... # setup shared fixture
^ this is buggy :) Its buggy because if it really does write to self, you now have what twisted did where all the test cases will be sharing one test object instance - which very frequently confuses and makes isolation between unrelated test state fragile.
If its not writing to self, then you don't know what class object to write to (because you might be in a subclass) - that really should be a class method.
And the fact that you have this confusion, as demonstrated here, is precisely why I don't agree about easy to implement and easy to understand.
core allowing fixtures to be shared across test instances in a way that doesn't prohibit their use with concurrent testing, doesn't make it awkward to do it across multiple classes.
Can you demonstrate how testresources solves these problems.
Yes. resources are declared in a list, which allows the test framework, when parallelising, to group together tests that use the same resources when partitioning. Compare this with setUpClass where all tests in an inheritance hierarchy have to be grouped together (after excluding your base TestCase).
You ask in another mail for a SSCE: here is one attached to this mail ('foo.py'). Run with 'python -m unittest foo'.
I'm happy to make any [reasonable] changes (including license) to testresources to make it includable in the stdlib if that's of interest.
That's not how contributing code to Python works. You need to provide a signed contributor agreement to the PSF and then you specifically license to the PSF any code you are contributing using one of a few specific licenses. For new modules in the standard library you also need to be willing to maintain the code.
There is other code of mine in the standard library unittest module, done without a contributor agreement: so this is at best inconsistent. I mention license changes because that would be required to include it in the standard library (its currently a copyleft license) - and I'm not the sole author. Regardless, whatever process and actions are needed, I will do them, or arrange with other people where needed.
-Rob
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas