Probabilistic unit tests?
Roy Smith
roy at panix.com
Thu Jan 10 21:22:29 EST 2013
In article <b312f3e7-5c73-486e-925e-da8343963fb6 at googlegroups.com>,
Nick Mellor <thebalancepro at gmail.com> wrote:
> Hi,
>
> I've got a unit test that will usually succeed but sometimes fails. An
> occasional failure is expected and fine. It's failing all the time I want to
> test for.
>
> What I want to test is "on average, there are the same number of males and
> females in a sample, give or take 2%."
> [...]
> My question is: how would you run an identical test 5 times and pass the
> group *as a whole* if only one or two iterations passed the test? Something
> like:
>
> for n in range(5):
> # self.assertAlmostEqual(...)
> # if test passed: break
> else:
> self.fail()
I would do something like:
def do_test_body():
"""Returns 1 if it passes, 0 if it fails"""
results = [do_test() for n in range(number_of_trials)]
self.assert(sum(results) > threshold)
That's the simple part.
The more complicated part is figuring out how many times to run the test
and what an appropriate threshold is. For that, you need to talk to a
statistician.
More information about the Python-list
mailing list