[Python-ideas] PEP 485: A Function for testing approximate equality
steve at pearwood.info
Mon Jan 26 06:54:38 CET 2015
On Mon, Jan 26, 2015 at 01:17:02AM +0000, Nathaniel Smith wrote:
> On Mon, Jan 26, 2015 at 12:02 AM, Chris Barker <chris.barker at noaa.gov> wrote:
> > Though I guess I'd rather a zero_tol that defaulted to non-zero that an
> > abs_tol that did. So we might be able to satisfy your observation that a lot
> > of use cases call for testing against zero.
> Yes, that's the idea -- defaulting rel_tol and zero_tol to non-zero
> values, and abs_tol to zero, gives you a set of defaults that will
> just work for people who want to write useful tests without having to
> constantly be distracted by floating point arcana.
I really don't think that setting one or two error tolerances is
"floating point arcana". I don't think that having to explicitly decide
on what counts as "close" (as either an absolute difference or a
relative difference) is especially onerous: surely anyone writing code
will be able to cope with one or two decisions:
- close enough means they differ by no more than X
- close enough means they differ by no more than X%, expressed
as a fraction
This isn't ULPs :-)
I'm almost inclined to not set any defaults, except perhaps zero for
both (in which case "close to" cleanly degrades down to "exactly equal"
except slower) and force the user to explicitly choose a value.
Arguments in favour of setting some defaults:
- People who want a "zero-thought" solution will get one, even
if it does the wrong thing for their specific application, but
at least they didn't have to think about it.
- The defaults might occasionally be appropriate.
- There is no default error tolerance we can pick, whether relative
or absolute, which will suit everyone all the time. Unless the
defaults are appropriate (say) 50% of the time or more, they will
just be an attractive nuisance (see zero-thought above).
In the statistics tests, I had the opportunity to set my own global
defaults, but I don't think I ever actually used them. Maybe I could
have picked better defaults? I don't know.
I did use defaults per test suite, so that's an argument in favour of
having is_close (approx_equal) not use defaults, but assertIsClose
(assertApproxEqual) use per-instance defaults.
[Context: in the tests, I had an assertApproxEqual method that relied on
approx_equal function. The function had defaults, but I never used them.
The method defaulted to reading defaults from self.rel and self.tol, and
I did use them.]
> This does require that zero_tol is only applied for expected == 0.0,
> *not* for actual == 0.0, though. If you expected 1e-10 and got 0.0
> then this *might* be okay in your particular situation but it really
> requires the user to think things through; a generic tool should
> definitely flag this by default.
I really think that having three tolerances, once of which is nearly
always ignored, is poor API design. The user usually knows when they are
comparing against an expected value of zero and can set an absolute
How about this?
- Absolute tolerance defaults to zero (which is equivalent to
- Relative tolerance defaults to something (possibly zero) to be
determined after sufficient bike-shedding.
- An argument for setting both values to zero by default is that
it will make it easy to choose one of "absolute or relative". You
just supply a value for the one that you want, and let the other
take the default of zero.
- At the moment, I'm punting on the behaviour when both abs and rel
tolerances are provided. That can be bike-shedded later.
Setting both defaults to zero means that the zero-thought version:
if is_close(x, y): ...
will silently degrade to x == y, which is no worse than what people
do now (except slower). We can raise a warning in that case.
The only tricky situation might be if you *may* be comparing against
zero, but don't know so in advance. There are some solutions to that:
- The suggested "zero_tol" parameter, which I dislike. I think it is
an ugly and confusing API.
- Some versions of is_close may not require any special treatment
for zero, depending on how it treats the situation where both abs
and rel tolerances are given. Further discussion needed.
- Worst case, people write this:
if (expected == 0 and is_close(actual, expected, tol=1e-8)
or is_close(actual, expected, rel=1e-5)):
but I don't think that will come up in practice very often. In the
test_statistics module, I had tests that looked like this:
for x in [bunch of non-zero values]:
y = do_some_calculation(x)
self.assertApproxEqual(x, y, rel=0.01)
y = do_some_calculation(0)
self.assertApproxEqual(0, y, tol=0.000001)
which isn't hard to do, so I don't think this is a real issue in
practice. I think the case of "my expected value might be zero, but I'm
not sure in advance" is rare and unusual enough that we don't need to
worry about it.
> My claim wasn't that is_close_to(x, 0.0) provides a mathematically
> ill-defined result. I agree that that's a reasonable definition of
> "relatively close to" (though one could make an argument that zero is
> not relatively close to itself -- after all, abs(actual -
> expected)/expected is ill-defined).
Don't write it that way. Write it this way:
abs(actual - expected) <= relative_tolerance*expected
Now if expected is zero, the condition is true if and only if
It would be bizarre for is_close(a, a) to return False (or worse,
raise an exception!) for any finite number. NANs, of course, are
allowed to be bizarre. Zero is not :-)
> Instead, my point was that if the
> user is asking "is this close to 0?" instead of "is this exactly equal
> to zero?" then they probably are expecting that there exist some
> inputs for which those two questions give different answers. Saying
> "well TECHNICALLY this is a valid definition of 'close to'" is
> certainly true but somewhat unkind.
I agree, but I think this is a symptom of essential complexity in the
problem domain. Ultimately, "is close" is ill-defined, and *somebody*
has to make the decision what that will be, and that decision won't
satisfy everyone always. We can reduce the complexity in one place:
* provide sensible default values that work for expected != 0
but only by increasing the complexity elsewhere:
* when expected == 0 the intuition that is_close is different
from exact equality fails
We can get rid of that complexity, but only by adding it back somewhere
* is_close(x, y, zero_tol=0.1) and is_close(x, y, zero_tol=0.00001)
give the same result for all the x,y I tested!
that is, zero_tol is nearly always ignored. Since people will often need
to think about what they want "is close" to mean no matter what we do, I
would prefer not to add the complexity of a third tolerance value. If
that means that "zero thought" users end up inadvertently testing for
exact equality without realising it, I think that's a price worth
paying for a clean API.
(As I said earlier, we can raise a warning in that case.)
> > I think this points to having a separate function for absolute tolerance
> > compared to zero -- but that's just abs(val) > zero_tolerance, so why
> > bother?
> Or it could just be the same function :-). Who wants to keep track of
> two functions that conceptually do the same thing?
More information about the Python-ideas