On Sun, Sep 30, 2018 at 10:29:50AM -0400, David Mertz wrote:
But given that in general unit tests tend to only exercise a handful of
values (have a look at the tests in the Python stdlib) I think it is
fair to say that in practice unit tests typically do not have anywhere
near the coverage of live data used during alpha and beta testing.
I still think it's a mixture. I write tests to also address "this really shouldn't happen" cases as well (often in a loop, or using a Nose class for scaffolding).
There's some saying/joke about software testing along the lines of:
For an argument that should be in range 1-100:
try 50; try 1; try 100
try 101; try 0;
try -1,
try 3.14
try 1+1j;
try the string "fish",
try a null pointer;
etc.
Many of those oddball cases can easily be in a list of values to test in unit tests, but may be impossible or unlikely to make it to the function call in the normal/possible flow of the program.
I'm curious. When you write a function or method, do you include input
checks? Here's an example from the Python stdlib (docstring removed for
brevity):
# bisect.py
def insort_right(a, x, lo=0, hi=None):
if lo < 0:
raise ValueError('lo must be non-negative')
if hi is None:
hi = len(a)
while lo < hi:
mid = (lo+hi)//2
if x < a[mid]: hi = mid
else: lo = mid+1
a.insert(lo, x)
Do you consider that check for lo < 0 to be disruptive? How would you
put that in a unit test?
I definitely put in checks like that. However, I might well write a test like:
assert lo >= 0, "lo must be non-negative"
That would allow disabling the check for production. However, I presume the stdlib does this for a reason; it presumably wants to allow callers to catch a specific exception. I haven't seen anything in the contract discussion that allows raising a particular exception when a contract is violated, only a general one for all contracts.
The case of 'if hi is None' is different. It's remediation of a missing value where it's perfectly fine to impute an unspecified value. So that would be a poor contract.