Probabilistic unit tests?
roy at panix.com
Sat Jan 12 15:22:32 CET 2013
<693d4bb1-8e1e-4de0-9d4d-8a136ea70ef4 at pp8g2000pbb.googlegroups.com>,
alex23 <wuwei23 at gmail.com> wrote:
> On 11 Jan, 13:34, Steven D'Aprano <steve
> +comp.lang.pyt... at pearwood.info> wrote:
> > Well, that's not really a task for unit testing. Unit tests, like most
> > tests, are well suited to deterministic tests, but not really to
> > probabilistic testing. As far as I know, there aren't really any good
> > frameworks for probabilistic testing, so you're stuck with inventing your
> > own. (Possibly on top of unittest.)
> One approach I've had success with is providing a seed to the RNG, so
> that the random results are deterministic.
Sometimes, a hybrid approach is best.
I was once working on some code which had timing-dependent behavior.
The input space was so large, there was no way to exhaustively test all
conditions. What we did was use a PRNG to drive the test scenarios,
seeded with the time. We would print out the seed at the beginning of
the test. This let us explore a much larger range of the input space
than we could have with hand-written test scenarios.
There was also a mode where you could supply your own PRNG seed. So,
the typical deal would be to wait for a failure during normal (nightly
build) testing, then grab the seed from the test logs and use that to
replicate the behavior for further study.
More information about the Python-list