replacing __dict__ with an OrderedDict
roy at panix.com
Mon Jan 9 09:35:10 EST 2012
In article <mailman.4488.1325870461.27778.python-list at python.org>,
Ian Kelly <ian.g.kelly at gmail.com> wrote:
> Randomizing the order is not a bad idea, but you also need to be able
> to run the tests in a consistent order, from a specific random seed.
> In the real world, test conflicts and dependencies do happen, and if
> we observe a failure, make a change, rerun the tests and observe
> success, we need to be able to be sure that we actually fixed the bug,
> and that it didn't pass only because it was run in a different order.
I've seen this argument play out multiple times on this group.
OP: "I want to do X"
Peanut Gallery: "You're not supposed to do that"
Here's my commentary on that.
The classic unittest philosophy says that tests should be independent of
each other, which means they should be able to be run in arbitrary
order. Some people advocate that the test framework should
intentionally randomize the order, to flush out inter-test dependencies
that the author didn't realize existed (or intend).
Test independence is a good thing. Many people don't understand this
when writing tests, and inadvertently write tests that depend on each
other. I've worked with those systems. They're a bear to debug.
You've got some test suite that runs for 15 minutes and dies at case 37
of 43. If you try to run case 37 by itself, it won't run, and you can't
figure out what state cases 1-36 were supposed to leave the system in to
make 37 work. You could sink days or weeks into debugging this kind of
That being said, the unittest module, while designed to support the "all
tests must be independent" philosophy, is a useful piece of software for
lots of things. It provides an easy to use framework for writing tests,
lots of convenient assertions, reporting, test discovery, etc, etc.
If somebody (i.e. the classic "consenting adult" of the Python world)
wants to take advantage of that to write a test suite where the tests
*do* depend on each other, and have to be run in a certain order,
there's nothing wrong with that. As long as they understand the
consequences of their actions, don't try to preach unittest religion to
them. They're in the best position to know if what they're trying to do
is the best thing for their particular situation.
More information about the Python-list