[Jeremy Hylton]
There was recently some idle chatter in Guido's living room about using a unit testing framework (like PyUnit) for the Python regression test suite. We're also writing tests for some DC projects, and need to decide what framework to use.
Does anyone have opinions on test frameworks? A quick web search turned up PyUnit (pyunit.sourceforge.net) and a script by Tres Seaver that allows implements xUnit-style unit tests. Are there other tools we should consider?
My own doctest is loved by people other than just me <wink>, but is aimed at ensuring that examples in docstrings work exactly as shown (which is why it starts with "doc" instead of "test").
Is anyone else interested in migrating the current test suite to a new framework?
Yes.
I hope the new framework will allow us to improve the test suite in a number of ways:
- run an entire test suite to completion instead of stopping on the first failure
doctest does that.
- clearer reporting of what went wrong
Ditto.
- better support for conditional tests, e.g. write a test for httplib that only runs if the network is up. This is tied into better error reporting, since the current test suite could only report that httplib succeeded or failed.
A doctest test is simply an interactive Python session pasted into a docstring (or more than one session, and/or interspersed with prose). If you can write an example in the interactive shell, doctest will verify it still works as advertised. This allows for embedding unit tests into the docs for each function, method and class. Nothing about them "looks like" an artificial test tacked on: the examples in the docs *are* the test cases. I need to try the other frameworks. I dare say doctest is ideal for computational functions, where the intended input->output relationship can be clearly explicated via examples. It's useless for GUIs. Usefulness varies accordingly between those extremes (doctest is natural exactly to the extent that a captured interactive session is helpful for documentation purposes). testing-ain't-easy-ly y'rs - tim