On Wed, 2010-01-20 at 22:33 -0500, Mark Roddy wrote:
I'd take issue with the argument that this makes the intent clearer. In your motivating example, what you mean is "the test needs a connection" not "I want to do a set up that spans the scope of this class". A better approach would be to _declare_ the dependency in the test and let something else figure out how to best provide the dependency. I think having the class setup is clearer than the 'if not self.CaseIsSetup' idiom, but I wouldn't claim that it's definitely the clearest way to achieve such functionality.
I like the class setup semantics as the closely follow the existing fixture setup semantics. Having to declare and configure dependencies (while possibly being clearer as far as expressing intent) introduces a completely different set of semantics that have to be grasped in order to be used. I'm going to with hold forming an opinion as to whether or not this presents any meaningful barrier to entry until after I've had a chance to review Rob's resources library.
A data point here is that the setUp/tearDown/cleanUp semantics are not trivial themselves: here is a self check (check the source after coming up with answers :P) - does tearDown run if setUp raises? - do cleanUps? - do cleanUps run before or after tearDown? Also would you add classCleanUps or something, if you add a class setUp? clean ups are _much_ nicer than tearDown, so I'd really want to see that.
It turns out that if you are building extensions to testing frameworks (like any big app does), it helps a lot to have an object per runnable test. In particular, it makes distributing tests across multiple processors & multiple computers much, much easier.
True, and clearly the case for highly focused unit tests. However, for integration tests (or whatever we could call tests that are explicitly designed to work with an expensive resource) it can be cost prohibitive to have this type of isolation (or flat out impossible in the case that I gave).
Its been trimmed, but here was your example: "For example, I have several hundred integration tests that hit a live database. If I create the connection object for each fixture, most of the tests fail due to the maximum connection limit being reached. As a work around, I create a connection object once for each test case." Database connections can be dirtied - they need to be in a clean state with no outstanding requests, and outside a transaction, to be valid for the start of a test. I do appreciate that if you have a buggy TCP stack, or are not closing your connections, you can certainly hit connection limits. I'd be inclined to have a pool of connections, rather than one connection per class. That would under ideal circumstances give you one connection for an entire test run (without concurrency), but also mean that a broken connection would at most affect only one test.
I'll look into the implementation of some of the testing frameworks that support distributed testing and see if there isn't a way that this can be supported in both contexts (is it possible to implement in a way that the case setup would get run in each process/machine?).
Its about partitioning the state: e.g. (ugly) deep copy the class object per thread.
It also poses a difficult challenge for test runners that provide features such as running the tests in a random order. It's very hard to know if the class is actually "done" and ready to be torn down. My initial (off the top of my head) thinking was to count the number of test fixtures to be run via a class attribute set in the test case constructor, and then the test runner would either decrement this after the test is complete and call the tear down method once the counter reached zero. This doesn't seem like it would be affected by randomized testing ordering, but I'll look into some existing implementations to see how this could be affected. Any obvious issues I'm missing?
Well, this has a few undesirable facets: - its complex - you'll need to honour tests that don't have that attribute set - and tests that already have that attribute set - it breaks the abstraction of 'a test knows how to run itself'. - you can't set that attribute in test case constructors because tests can be constructed while the suite is running - using a global like that will mean that a single classes class setup stuff would *stay setup* while other classes run, with a randomised order : thats bad because things that are expensive and need to be optimised also tend to be *large*, either in memory or external resources (like TCP connections in your example).
Finally, we found that it's use often lead to hard-to-debug failures due to test isolation issues.
I think there's a distinction between "can lead to bad situations" and "encourages bad situations". The former is almost impossible to avoid without becoming java :). That latter is much subtler, but can be addressed. Do you have any suggestions for altering the semantics to discourage abuse without reducing flexibility. With a similar feature I use, we have a rule to not use the case setup unless explicitly writing integration tests, though there is no functional way to enforce this, only communicating the idea (via documentation and code reviews).
Have you seen Rusty Russell's interface usability scale? Having to have a rule strongly suggests that the API is prone to misuse :).
I'm not deeply familiar with xUnit implementations in other languages, but the problem isn't unique to Python. I imagine it would be worth doing some research on what Nunit, JUnit etc do. Both JUnit and Nunit have a class setup and teardown functionality: http://www.junit.org/apidocs/org/junit/BeforeClass.html http://www.junit.org/apidocs/org/junit/AfterClass.html http://www.nunit.org/index.php?p=fixtureSetup&r=2.5.3 http://www.nunit.org/index.php?p=fixtureTeardown&r=2.5.3 (The Nunit implementation I found a little confusing as they apparently refer to a TestCase as a Fixture).
See my other mail for a link to JUnit's Rules. Cheers, Rob