[Python-ideas] Test Class setup and teardown in unittest
robertc at robertcollins.net
Thu Jan 21 05:16:17 CET 2010
On Wed, 2010-01-20 at 22:33 -0500, Mark Roddy wrote:
> > I'd take issue with the argument that this makes the intent clearer. In your
> > motivating example, what you mean is "the test needs a connection" not "I want
> > to do a set up that spans the scope of this class". A better approach would be
> > to _declare_ the dependency in the test and let something else figure out how to
> > best provide the dependency.
> I think having the class setup is clearer than the 'if not
> self.CaseIsSetup' idiom, but I wouldn't claim that it's definitely the
> clearest way to achieve such functionality.
> I like the class setup semantics as the closely follow the existing
> fixture setup semantics. Having to declare and configure dependencies
> (while possibly being clearer as far as expressing intent) introduces
> a completely different set of semantics that have to be grasped in
> order to be used. I'm going to with hold forming an opinion as to
> whether or not this presents any meaningful barrier to entry until
> after I've had a chance to review Rob's resources library.
A data point here is that the setUp/tearDown/cleanUp semantics are not
trivial themselves: here is a self check (check the source after coming
up with answers :P)
- does tearDown run if setUp raises?
- do cleanUps?
- do cleanUps run before or after tearDown?
Also would you add classCleanUps or something, if you add a class setUp?
clean ups are _much_ nicer than tearDown, so I'd really want to see
> > It turns out that if you are building extensions to testing frameworks (like any
> > big app does), it helps a lot to have an object per runnable test. In
> > particular, it makes distributing tests across multiple processors & multiple
> > computers much, much easier.
> True, and clearly the case for highly focused unit tests. However,
> for integration tests (or whatever we could call tests that are
> explicitly designed to work with an expensive resource) it can be cost
> prohibitive to have this type of isolation (or flat out impossible in
> the case that I gave).
Its been trimmed, but here was your example:
example, I have several hundred integration tests that hit a live
database. If I create the connection object for each fixture, most of
the tests fail due to the maximum connection limit being reached. As
a work around, I create a connection object once for each test case."
Database connections can be dirtied - they need to be in a clean state
with no outstanding requests, and outside a transaction, to be valid for
the start of a test. I do appreciate that if you have a buggy TCP stack,
or are not closing your connections, you can certainly hit connection
I'd be inclined to have a pool of connections, rather than one
connection per class. That would under ideal circumstances give you one
connection for an entire test run (without concurrency), but also mean
that a broken connection would at most affect only one test.
> I'll look into the implementation of some of
> the testing frameworks that support distributed testing and see if
> there isn't a way that this can be supported in both contexts (is it
> possible to implement in a way that the case setup would get run in
> each process/machine?).
Its about partitioning the state: e.g. (ugly) deep copy the class object
> > It also poses a difficult challenge for test runners that provide features such
> > as running the tests in a random order. It's very hard to know if the class is
> > actually "done" and ready to be torn down.
> My initial (off the top of my head) thinking was to count the number
> of test fixtures to be run via a class attribute set in the test case
> constructor, and then the test runner would either decrement this
> after the test is complete and call the tear down method once the
> counter reached zero. This doesn't seem like it would be affected by
> randomized testing ordering, but I'll look into some existing
> implementations to see how this could be affected. Any obvious issues
> I'm missing?
Well, this has a few undesirable facets:
- its complex
- you'll need to honour tests that don't have that attribute set
- and tests that already have that attribute set
- it breaks the abstraction of 'a test knows how to run itself'.
- you can't set that attribute in test case constructors because tests
can be constructed while the suite is running
- using a global like that will mean that a single classes class setup
stuff would *stay setup* while other classes run, with a randomised
order : thats bad because things that are expensive and need to be
optimised also tend to be *large*, either in memory or external
resources (like TCP connections in your example).
> > Finally, we found that it's use often lead to hard-to-debug failures due to test
> > isolation issues.
> I think there's a distinction between "can lead to bad situations" and
> "encourages bad situations". The former is almost impossible to avoid
> without becoming java :). That latter is much subtler, but can be
> addressed. Do you have any suggestions for altering the semantics to
> discourage abuse without reducing flexibility. With a similar feature
> I use, we have a rule to not use the case setup unless explicitly
> writing integration tests, though there is no functional way to
> enforce this, only communicating the idea (via documentation and code
Have you seen Rusty Russell's interface usability scale? Having to have
a rule strongly suggests that the API is prone to misuse :).
> > I'm not deeply familiar with xUnit implementations in other languages, but the
> > problem isn't unique to Python. I imagine it would be worth doing some research
> > on what Nunit, JUnit etc do.
> Both JUnit and Nunit have a class setup and teardown functionality:
> (The Nunit implementation I found a little confusing as they
> apparently refer to a TestCase as a Fixture).
See my other mail for a link to JUnit's Rules.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 197 bytes
Desc: This is a digitally signed message part
More information about the Python-ideas