[SciPy-user] Article(s) on Test Driven Development for Science

[Bruce Southey]

Really can not to what Josef said.

On Thu, Mar 12, 2009 at 8:13 AM, Timmie <timmichelsen at gmx-topmail.de> wrote:
> Hello,
> from many blogs, in the book "Expert Python Programming" and in the code I read
> about test driven development.
> It's said to help preventing future breakage or failures of the code.
>
> Is see on major diffuculty when adopting this for science scripts:
>
> In Science we not only have to control the program flow but also to validate the
> output.
> I think such a validation needs to be included in tests.

I would presume that the code was written to give the correct results
in the first place.

>
> I may change something in my code and still pass tests from the software side
I thought that it is what you hope happens

> but the result data may be totally wrong.

In that case there are really a few reasons:
1) the test is wrong such as not allowing sufficient iterations or is
too lax in setting convergence so the test output is actually
incorrect but code would have given the correct answer if given say
more iterations.
2) your original code had a bug like bad logic
3) the algorithm does not work for that data or may need more iterations.
4) the data is work or inappropriate - may not have sufficient
information or numerical issues to do what you want.

>
> Are there already concepts for such testing?
Writing even more tests that cover any special corner cases.

>
> The tests I have seen so far play mostely with generated random data.
>
> Are there presentations or articles about this?
You might find some references under software quality.

> Regards,
> Timmie

In Numpy there is usually a request for people to write new tests for
the corner cases that get found or any unexpected behavior. I am also
sure that people will ask or file bug reports when they try various
functions and do not get the expected answer.

More fundamental are tests that really test accuracy by giving complex
or difficult problems to solve (not just to show the errors in
Microsoft Excel). I only know statistics and I know there some
datasets and models that really test software. I do not recall the
papers but one attempt is The Statistical Reference Datasets Project:
http://www.itl.nist.gov/div898/strd/

Regards
Bruce