replacing __dict__ with an OrderedDict

[Ulrich Eckhardt]

Am 09.01.2012 15:35, schrieb Roy Smith:
> The classic unittest philosophy says that tests should be independent of
> each other, which means they should be able to be run in arbitrary
> order.  Some people advocate that the test framework should
> intentionally randomize the order, to flush out inter-test dependencies
> that the author didn't realize existed (or intend).

While I agree with the idea, I'd like to add that independence is an 
illusion. You already have possible dependencies if you run tests in the 
same process/OS-installation/computer/parallel universe. If you now 
happen to influence one test with another and the next run randomizes 
the tests differently, you will never see the fault again. Without this 
reproducability, you don't gain anything but the bad stomach feeling 
that something is wrong.


> Test independence is a good thing.  Many people don't understand this
> when writing tests, and inadvertently write tests that depend on each
> other.  I've worked with those systems.  They're a bear to debug.
> You've got some test suite that runs for 15 minutes and dies at case 37
> of 43.  If you try to run case 37 by itself, it won't run, and you can't
> figure out what state cases 1-36 were supposed to leave the system in to
> make 37 work.  You could sink days or weeks into debugging this kind of
> crap.  BTDT.

I'm sorry to hear that, debugging other peoples' feces is never a task 
to wish for. That said, there are two kinds of dependencies and in at 
least this discussion there hasn't been any mentioning of the 
differences yet, but those differences are important.


Your unfortunate case is where test X creates persistent state that must 
be present in order for test X+1 to produce meaningful results. This 
kind of dependency obviously blows, as it means you can't debug test X+1 
separately. I'd call this operational dependency.

This kind of dependency is IMHO a bug in the tests themselves. The unit 
testing framework could help you find those bugs by allowing random 
order of execution for the test cases.


There is another dependency and that I'd call a logical dependency. This 
occurs when e.g. test X tests for an API presence and test Y tests the 
API behaviour. In other words, Y has no chance to succeed if X already 
failed. Unfortunately, there is no way to express this relation, there 
is no "@unittest.depends(test_X)" to decorate test_Y with (Not yet!). 
Each test would be the root of a tree of tests that it depends on. If a 
dependency fails already, you can either skip the tree or at least mark 
the following test failures as implicit failures, so that you can easily 
distinguish them from the root cause.

This kind of dependency is quite normal although not directly supported 
by the unittest module. As a workaround, being able to define an order 
allows you to move the dependencies further to the top, so they are 
tested first.


To sum it up, in order to catch operational dependencies, you need a 
random order while in order to clearly express logical dependencies in 
the output, you want a fixed order. Neither is the one true way, though 
my gut feeling is that the fixed order is overall more useful.


> As long as they understand the consequences of their actions, don't
> try to preach unittest religion to them.  They're in the best
> position to know if what they're trying to do is the best thing for
> their particular situation.

Amen!

Uli