[Numpy-discussion] Adding "maximum difference" to np.testing.assert_array_equal errors

[Stephan Hoyer]

I have a PR up for review that adds "maximum difference" to the error
messages produced by NumPy's testing functions for comparing arrays:
https://github.com/numpy/numpy/pull/12591

Because this changes NumPy's public interface, I'm running it by the
mailing list to see if there are any objections or suggestions for
improvement.

Example behavior:

>>> x = np.array([1, 2, 3])
>>> y = np.array([1, 2, 3.0001])
>>> np.testing.assert_allclose(x, y)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
 x: array([1, 2, 3])
 y: array([1. , 2. , 3.0001])

Motivation: when writing numerical algorithms, I frequently find myself
experimenting to pick the right value of atol and rtol for
np.testing.assert_allclose(). If I make the tolerance too generous, I risk
missing regressions in accuracy, so I usually try to pick the smallest
values for which tests pass. This change immediately reveals appropriate
values to use for these parameters, so I don't need to guess and check.

One alternative would be print both "atol" and "rtol" numbers directly that
could be immediately used in assert_allclose(). Here we effectively only
have "atol". The main reason why I didn't do this is that I felt it would
add more clutter (rtol is slightly redundant with the displayed array
values). But we could probably do this if we're willing to split the error
message onto a few more lines, e.g.,

AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 33.333333333333336%,
 maximum absolute difference 0.00010000000000021103,
 maximum relative difference 3.333333333340368e-05)
 x: array([1, 2, 3])
 y: array([1. , 2. , 3.0001])

Best,
Stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20181219/ef28334f/attachment.html>