On Sun, Sep 30, 2018 at 11:34 AM Steven D'Aprano email@example.com wrote:
On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote: But given that in general unit tests tend to only exercise a handful of values (have a look at the tests in the Python stdlib) I think it is fair to say that in practice unit tests typically do not have anywhere near the coverage of live data used during alpha and beta testing.
I still think it's a mixture. I write tests to also address "this really shouldn't happen" cases as well (often in a loop, or using a Nose class for scaffolding).
There's some saying/joke about software testing along the lines of:
For an argument that should be in range 1-100: try 50; try 1; try 100 try 101; try 0; try -1, try 3.14 try 1+1j; try the string "fish", try a null pointer; etc.
Many of those oddball cases can easily be in a list of values to test in unit tests, but may be impossible or unlikely to make it to the function call in the normal/possible flow of the program.
I'm curious. When you write a function or method, do you include input checks? Here's an example from the Python stdlib (docstring removed for brevity):
# bisect.py def insort_right(a, x, lo=0, hi=None): if lo < 0: raise ValueError('lo must be non-negative') if hi is None: hi = len(a) while lo < hi: mid = (lo+hi)//2 if x < a[mid]: hi = mid else: lo = mid+1 a.insert(lo, x)
Do you consider that check for lo < 0 to be disruptive? How would you put that in a unit test?
I definitely put in checks like that. However, I might well write a test like:
assert lo >= 0, "lo must be non-negative"
That would allow disabling the check for production. However, I presume the stdlib does this for a reason; it presumably wants to allow callers to catch a specific exception. I haven't seen anything in the contract discussion that allows raising a particular exception when a contract is violated, only a general one for all contracts.
The case of 'if hi is None' is different. It's remediation of a missing value where it's perfectly fine to impute an unspecified value. So that would be a poor contract.