suggestion: improve text of failing test
The assert_allclose text is not precise enough to be helpful to fix a test failure that cannot be replicated on every machine, and we cannot just quickly grab --pdb-failures.
By how much do I have to lower the precision to make it pass on this continuous integration machine?
assert_allclose(he, hefd, rtol=5e-10) File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py", line 1297, in assert_allclose verbose=verbose, header=header) File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py", line 665, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=5e-10, atol=0
(mismatch 100.0%) x: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00, -1.003654e+01], [ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]]) y: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00, -1.003654e+01], [ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]])
the suggestion is to add rtol and atol to the mismatch summary, so we can see if it's just a precision issue or something serious
rtol = np.max(np.abs(x / y - 1) atol = np.max(np.abs(x - y)
(mismatch 100.0% rtol=xxx atol=xxx)
(and as aside to the "all close" discussion: I do set the tolerances very carefully especially if the agreement with comparison numbers is below 1e-6 or so)
Josef
On 5 Feb 2015 12:15, josef.pktd@gmail.com wrote:
The assert_allclose text is not precise enough to be helpful to fix a
test failure that cannot be replicated on every machine, and we cannot just quickly grab --pdb-failures.
By how much do I have to lower the precision to make it pass on this
continuous integration machine?
assert_allclose(he, hefd, rtol=5e-10) File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py",
line 1297, in assert_allclose
verbose=verbose, header=header)File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py",
line 665, in assert_array_compare
raise AssertionError(msg)AssertionError: Not equal to tolerance rtol=5e-10, atol=0
(mismatch 100.0%) x: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00,
-1.003654e+01],
[ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]])y: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00,
-1.003654e+01],
[ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]])the suggestion is to add rtol and atol to the mismatch summary, so we can
see if it's just a precision issue or something serious
rtol = np.max(np.abs(x / y - 1) atol = np.max(np.abs(x - y)
(mismatch 100.0% rtol=xxx atol=xxx)
So basically just printing what rtol and/or atol would have to be to make the test pass? Sounds useful to me. (There is a bit of an infelicity in that if you're using both atol and rtol in the same test then there's no easy way to suggest how to fix both simultaneously, but I'm not sure how to fix that. Maybe we should also print max(abs(x[y == 0]))?)
Want to submit a pull request?
-n
On Thu, Feb 5, 2015 at 3:39 PM, Nathaniel Smith njs@pobox.com wrote:
On 5 Feb 2015 12:15, josef.pktd@gmail.com wrote:
The assert_allclose text is not precise enough to be helpful to fix a
test failure that cannot be replicated on every machine, and we cannot just quickly grab --pdb-failures.
By how much do I have to lower the precision to make it pass on this
continuous integration machine?
assert_allclose(he, hefd, rtol=5e-10) File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py",
line 1297, in assert_allclose
verbose=verbose, header=header)File "C:\Python27\envs\py3\lib\site-packages\numpy\testing\utils.py",
line 665, in assert_array_compare
raise AssertionError(msg)AssertionError: Not equal to tolerance rtol=5e-10, atol=0
(mismatch 100.0%) x: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00,
-1.003654e+01],
[ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]])y: array([[ -2.965667e+01, -1.988865e+02, -2.370194e+00,
-1.003654e+01],
[ -1.988865e+02, -1.383377e+03, -1.592292e+01, -6.800266e+01], [ -2.370194e+00, -1.592292e+01, -8.301699e-01, -8.301699e-01], [ -1.003654e+01, -6.800266e+01, -8.301699e-01, -3.449885e+00]])the suggestion is to add rtol and atol to the mismatch summary, so we
can see if it's just a precision issue or something serious
rtol = np.max(np.abs(x / y - 1) atol = np.max(np.abs(x - y)
(mismatch 100.0% rtol=xxx atol=xxx)
So basically just printing what rtol and/or atol would have to be to make the test pass? Sounds useful to me. (There is a bit of an infelicity in that if you're using both atol and rtol in the same test then there's no easy way to suggest how to fix both simultaneously, but I'm not sure how to fix that. Maybe we should also print max(abs(x[y == 0]))?)
I usually check the rtol and atol as above in pdb on failure, Most of the time it's enough information to figure out how to twist the numbers. There are only a few cases where I'm fine tuning both rtol and atol at the same time. I guess there is the sum of the tol from the definition of allclose.
We don't have many cases with y == 0 mixed together with large numbers, because our reference numbers usually also have numerical noise.
One point is also to just make the test output more informative, to see if the test machine is just a bit off even if the mismatch=100%.
Want to submit a pull request?
Not really, I'd rather stick to my corner and let someone else get on the numpy contributor list :) (header was "suggestion" not "proposal")
Thanks,
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (2)
-
josef.pktd＠gmail.com -
Nathaniel Smith