Fwd: PEP 485: A Function for testing approximate equality

OOPS, sent to only Paul by mistake the first time. -Chris On Sun, Jan 25, 2015 at 11:07 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Any of the approaches on the table will do something reasonable in this case: In [4]: is_close_to.is_close_to(sum([0.1]*10), 1) testing: 0.9999999999999999 1 Out[4]: True Note that the 1e-8 d3fault I chose (which I am not committed to) is not ENTIRELY arbitrary -- it's about half the digits carried by a python float (double) -- essentially saying the values are close to about half of the precision available. And we are constrained here, the options are between 0.1 (which would be crazy, if you ask me!) and 1e-14 -- any larger an it would meaningless, and any smaller, and it would surpass the precision of a python float. PIcking a default near the middle of that range seems quite sane to me. This is quite different than setting a value for an absolute tolerance -- saying something is close to another number if the difference is less than 1e-8 would be wildly inappropriate when the smallest numbers a float can hold are on order of 1e-300!
arcana, maybe, not it's not a floating point issue -- X% of zero is zero absolutely precisely. But back to a point made earlier -- the idea here is to provide something better than naive use of x == y for floating point. A default for the relative tolerance provides that. There is no sane default for absolute tolerance, so I dont think we should set one. Note that the zero_tolerance idea provides a way to let is_close_to(something, 0.0) work out of the box, but I'm not sure we can come up with a sane default for that either -- in which case, what's the point?
I for one, think making it mandatory to set one would be better than just letting the zeros get used. I've used the numpy version of this a lot for tests, , and my work flow is usually: write a test with the defaults if it pases, I'm done. If it fails, then I look and see if my code is broken, or if I can accept a larger tolerance. So I'm quite happy to have a default.
also agreed -- Nathanial -- can you live with this? Note that Nathaniel found a LOT of examples of people using assertAlmostEqual to compare to zero -- I think that's why he thinks it's important to have defaults that do something sane for that case. However, that is an absolute comparison function -- inherently different anyway -- it's also tied to "number of digits after the decimal place", so only appropriate for values near 1 anyway -- so you can define a sensible default there. not so here.
- Absolute tolerance defaults to zero (which is equivalent to
exact equality).
yup
1e-8 -- but you already know that ;-) -- anything between 1e-8 and 1e-12 would be fine with me.
sorry about the head -- but setting both is a pretty useful use-case -- you have a bunch of values you want to do a realive test on. Some of them maybe exactly zero (note -- it could be either expected or actual) -- and you know what "close to zero" means to you. so you set that for abs_tolerance. Done. And I actually didn't think anyone objected to that approach -- though maybe the exploded heads weren't able to write ;-) In fact, an absolute tolerance is so easy that I wouldn't write a function for it: abs(expected - actual) <= abs_tolerance I ended up adding it to my version to deal with the zero case.
I now have "expected" and "actual", but I think those makes are too unclear -- I like "expected" -- anyone have a better idea for the other one?
It is called out, but I guess I need to make that clearer. Overall, I think it would be better to simplify the proposed function
right -- that is why I've been resisting adding flags for all the various options -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Jan 27, 2015 at 12:16 AM, Chris Barker <chris.barker@noaa.gov> wrote:
I can live with it, but I'd ignore the function and use allclose instead :-). In your workflow above, I'm guessing that with allclose then it's <1% of the time that you have a failure due to too-restrictive default tolerances, where you then have to switch from thinking about your problem to thinking about floating point details. With no absolute tolerance, this rises to ~30% (assuming my super quick and dirty github statistics are typical). That's a lot of workflow disruption. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

On Mon, Jan 26, 2015 at 4:42 PM, Nathaniel Smith <njs@pobox.com> wrote:
why not just set an appropriate abs_tolerance?
well they by definition aren't typical, as assertAlmostEqual isn't the same function -- it is only useful for an absolute tolerance for values near magnitude 1 -- so anyone that needed a relative tolerance (which I"m trying to provide) wouldn't have used it. But your point about numpy.allclose is taken -- it sets a non-zero default abs_tol ( atol=1e-08 ) -- and apparently that's worked well for you and others. Honestly, I've been using numpy's allclose for ages without thinking about abs_tol -- but after all this, I need to go back an look at all my tests -- I"m not at all sure that that was always appropriate! I guess the trade-offs for a default are: What are the most common cases? - so here -- do people only rarely work with numbers much smaller than 1? (i.e. less than 1e-4 or so?) How disastrous is it to have an inappropriate default? - Here is where I think it's a bad idea to have default: A) If there is a default: Someone writes a test where they compare their very small value to 0.0, and there is a default, the test will pass, even if the very small value really is far away from zero for their use-case. The test passes, they are happy, even though the result they are testing are totally bogus -- they very well might not notice. B) If there is no (non-zero) default: Someone writes a test where they compare their very small value to 0.0, the test will fail, even if the very small value really is close to zero for their use case. The test has failed, so they will examine the results, determine that it's as good as it can get, adjust the abs_tolerance value in the test, and away they go. Maybe B is rare -- I have no idea, but it does seem a substantially worse outcome -- the attractive nuisance referred to earlier. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Jan 27, 2015 at 12:16 AM, Chris Barker <chris.barker@noaa.gov> wrote:
I can live with it, but I'd ignore the function and use allclose instead :-). In your workflow above, I'm guessing that with allclose then it's <1% of the time that you have a failure due to too-restrictive default tolerances, where you then have to switch from thinking about your problem to thinking about floating point details. With no absolute tolerance, this rises to ~30% (assuming my super quick and dirty github statistics are typical). That's a lot of workflow disruption. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

On Mon, Jan 26, 2015 at 4:42 PM, Nathaniel Smith <njs@pobox.com> wrote:
why not just set an appropriate abs_tolerance?
well they by definition aren't typical, as assertAlmostEqual isn't the same function -- it is only useful for an absolute tolerance for values near magnitude 1 -- so anyone that needed a relative tolerance (which I"m trying to provide) wouldn't have used it. But your point about numpy.allclose is taken -- it sets a non-zero default abs_tol ( atol=1e-08 ) -- and apparently that's worked well for you and others. Honestly, I've been using numpy's allclose for ages without thinking about abs_tol -- but after all this, I need to go back an look at all my tests -- I"m not at all sure that that was always appropriate! I guess the trade-offs for a default are: What are the most common cases? - so here -- do people only rarely work with numbers much smaller than 1? (i.e. less than 1e-4 or so?) How disastrous is it to have an inappropriate default? - Here is where I think it's a bad idea to have default: A) If there is a default: Someone writes a test where they compare their very small value to 0.0, and there is a default, the test will pass, even if the very small value really is far away from zero for their use-case. The test passes, they are happy, even though the result they are testing are totally bogus -- they very well might not notice. B) If there is no (non-zero) default: Someone writes a test where they compare their very small value to 0.0, the test will fail, even if the very small value really is close to zero for their use case. The test has failed, so they will examine the results, determine that it's as good as it can get, adjust the abs_tolerance value in the test, and away they go. Maybe B is rare -- I have no idea, but it does seem a substantially worse outcome -- the attractive nuisance referred to earlier. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (2)
-
Chris Barker
-
Nathaniel Smith