[New-bugs-announce] [issue26040] Improve coverage and rigour of test.test_math

[Jeff Allen]

New submission from Jeff Allen:

In many cases, test_math does not exercise the functions defined by the math module, other than at a few values needing special handling. Where it does test a function with values representative of the domain, the accuracy demanded is often loose in one place and tight in another. By comparison, test_cmath uses more interesting values and a rational approach to accuracy.

In most implementations, math simply proxies an expertly built (and tested) platform library or firmware, so it is unlikely we are the victims of mathematical inexactitude.

We observed this as a problem in the Jython project, where failures in cmath were traced eventually to the naive (but passing) implementation of some functions in math. For Jython, we now supplement test_math with stronger tests more like those in test_cmath. So this issue is really an offer to fold that work back into the CPython test_math, to strengthen testing for all.


Coverage:
--------
In the attached program, in the coverge section, I logged the calls to selected functions from test_math to see what range of values they used. It produces results like:

sqrt
 [9.881e-324, 7.905e-323, 1.862e-309, 1e-305, 1e-150, ..., 2, 23, 1e+16, 1e+150, 1e+299]
 [-1]
 [-0.0, -inf, 0.0, inf, nan]

sin
 [1.571]
 [-1.571]
 [-0.0, -inf, 0.0, inf, nan]

exp
 []
 [-745]
 [-0.0, -inf, 0.0, inf, nan]

sinh
 []
 []
 [-0.0, -inf, 0.0, inf, nan]

The three rows here are positive, negative and "special" arguments, and "..." means I abridged the list.


Accuracy:
--------
Where functions are tested, results are expected with a maximum error of 1e-5. On sqrt(1e150), this demands bit-perfect accuracy; on sqrt(1e-150) it demands virtually nothing. 

In the attached program, in the accuracy section, I wrap most of the functions so as to fuzz their results by plus or minus 2e-6, test_math will still pass them. test_cmath uses a relative measure of error (so many ulps of the expected value), which is worth emulating.


If people think it better, coverage and accuracy can be tracked as separate issues.

----------
components: Tests
files: stat_math.py
messages: 257695
nosy: jeff.allen, mark.dickinson
priority: normal
severity: normal
status: open
title: Improve coverage and rigour of test.test_math
type: behavior
versions: Python 2.7, Python 3.6
Added file: http://bugs.python.org/file41526/stat_math.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26040>
_______________________________________