Adding "maximum difference" to np.testing.assert_array_equal errors
I have a PR up for review that adds "maximum difference" to the error messages produced by NumPy's testing functions for comparing arrays: https://github.com/numpy/numpy/pull/12591 Because this changes NumPy's public interface, I'm running it by the mailing list to see if there are any objections or suggestions for improvement. Example behavior:
x = np.array([1, 2, 3]) y = np.array([1, 2, 3.0001]) np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103) x: array([1, 2, 3]) y: array([1. , 2. , 3.0001]) Motivation: when writing numerical algorithms, I frequently find myself experimenting to pick the right value of atol and rtol for np.testing.assert_allclose(). If I make the tolerance too generous, I risk missing regressions in accuracy, so I usually try to pick the smallest values for which tests pass. This change immediately reveals appropriate values to use for these parameters, so I don't need to guess and check. One alternative would be print both "atol" and "rtol" numbers directly that could be immediately used in assert_allclose(). Here we effectively only have "atol". The main reason why I didn't do this is that I felt it would add more clutter (rtol is slightly redundant with the displayed array values). But we could probably do this if we're willing to split the error message onto a few more lines, e.g., AssertionError: Not equal to tolerance rtol=1e07, atol=0 (mismatch 33.333333333333336%, maximum absolute difference 0.00010000000000021103, maximum relative difference 3.333333333340368e05) x: array([1, 2, 3]) y: array([1. , 2. , 3.0001]) Best, Stephan
On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
Example behavior:
x = np.array([1, 2, 3]) y = np.array([1, 2, 3.0001]) np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
This is a helpful addition; thank you! I don't have a strong preference around whether to also include rtol. Stéfan
On 19. Dec 2018, at 19:50, Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
Example behavior:
x = np.array([1, 2, 3]) y = np.array([1, 2, 3.0001]) np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
This is a helpful addition; thank you! I don't have a strong preference around whether to also include rtol.
Stéfan _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
+1 to this addition. And +1 to also print rtol. I frequently use rtol with assert_allclose, and having the value printed helps quickly choose an appropriate value. Christoph
Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan! On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:
On 19. Dec 2018, at 19:50, Stefan van der Walt <stefanv@berkeley.edu> wrote:
On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
Example behavior:
x = np.array([1, 2, 3]) y = np.array([1, 2, 3.0001]) np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
This is a helpful addition; thank you! I don't have a strong preference around whether to also include rtol.
Stéfan _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
+1 to this addition.
And +1 to also print rtol. I frequently use rtol with assert_allclose, and having the value printed helps quickly choose an appropriate value.
Christoph _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
OK, it sounds like a popular change :). I'll add relative error to the message as well. Please comment on the pull request if you have strong feelings about exactly what it should look like. On Wed, Dec 19, 2018 at 4:41 PM Juan NunezIglesias <jni.soma@gmail.com> wrote:
Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan!
On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:
On 19. Dec 2018, at 19:50, Stefan van der Walt <stefanv@berkeley.edu>
On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
Example behavior:
> x = np.array([1, 2, 3]) > y = np.array([1, 2, 3.0001]) > np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
This is a helpful addition; thank you! I don't have a strong
wrote: preference
around whether to also include rtol.
Stéfan _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
+1 to this addition.
And +1 to also print rtol. I frequently use rtol with assert_allclose, and having the value printed helps quickly choose an appropriate value.
Christoph _______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
_______________________________________________ NumPyDiscussion mailing list NumPyDiscussion@python.org https://mail.python.org/mailman/listinfo/numpydiscussion
participants (4)

Christoph Deil

Juan NunezIglesias

Stefan van der Walt

Stephan Hoyer