replacing __dict__ with an OrderedDict

[Roy Smith]

In article <11jrt8-l32.ln1 at satorlaser.homedns.org>,
 Ulrich Eckhardt <ulrich.eckhardt at dominolaser.com> wrote:

> > Some people advocate that the test framework should
> > intentionally randomize the order, to flush out inter-test dependencies
> > that the author didn't realize existed (or intend).
> 
> If you now 
> happen to influence one test with another and the next run randomizes 
> the tests differently, you will never see the fault again. Without this 
> reproducability, you don't gain anything but the bad stomach feeling 
> that something is wrong.

The standard solution to that is to print out the PRNG initialization 
state and provide a way in your test harness to re-initialize it to that 
state.  I've done things like that in test scenarios where it is 
difficult or impossible to cover the problem space deterministically.

> Your unfortunate case is where test X creates persistent state that must 
> be present in order for test X+1 to produce meaningful results. This 
> kind of dependency obviously blows, as it means you can't debug test X+1 
> separately. I'd call this operational dependency.
> 
> This kind of dependency is IMHO a bug in the tests themselves.

For the most part, I'm inclined to agree.  However, there are scenarios 
where having each test build the required state from scratch is 
prohibitively expensive.  Imagine if you worked at NASA wanted to run 
test_booster_ignition(), test_booster_cutoff(), 
test_second_stage_ignition(), and test_self_destruct().  I suppose you 
could run them in random order, but you'd use up a lot of rockets that 
way.

Somewhat more seriously, let's say you wanted to do test queries against 
a database with 100 million records in it.  You could rebuild the 
database from scratch for each test, but doing so might take hours per 
test.  Sometimes, real life is just *so* inconvenient.

> There is another dependency and that I'd call a logical dependency. This 
> occurs when e.g. test X tests for an API presence and test Y tests the 
> API behaviour. In other words, Y has no chance to succeed if X already 
> failed.

Sure.  I run into that all the time.  A trivial example would be the 
project I'm working on now.  I've come to realize that a long unbroken 
string of E's means, "Dummy, you forgot to bring the application server 
up before you ran the tests".  It would be nicer if the test suite could 
have run a single test which proved it could create a TCP connection and 
when that failed, just stop.