<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jan 14, 2015 at 11:29 PM, Steven D'Aprano <span dir="ltr"><<a href="mailto:steve@pearwood.info" target="_blank">steve@pearwood.info</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The question of which to use as the denominator is more subtle. Like<br>
you, I used to think that you should choose ahead of time which value<br>
was expected and which was actual, and divide by the actual. Or should<br>
that be the expected? I could never decide which I wanted: error<br>
relative to the expected, or error relative to the actual? And then I<br>
could never remember which order the two arguments went.<br></blockquote><div><br></div><div>I'm on the fence about this -- it seems clear to me that if the user has specified an "expected" value, that tolerance would clearly be based on that magnitude. If nothing else, that is because you would more likely have many computed values for each expected value than the other way around. </div><div><br></div><div>And I was thinking that calling the arguments something like "actual" and "expected" would make that clear in the API, and would certainly be documentable.</div><div><br></div><div>But the fact that it was never clear to you , even as you were writing the code, is good evidence that it wouldn't be clear to everyone ;-)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Error<br>
calculations should be symmetrical, so that<br>
<br>
error(a, b) == error(b, a)<br></blockquote><div><br></div><div>That does make it simpler to think about and reason about, and makes the use-cased more universal (or at least appear more universal) "are these two values close" is a simple question to ask, if not to answer.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">regardless of whether you have absolute or relative error. Furthermore,<br>
for safety you normally want the larger estimate of error, not the<br>
smaller: given the choice between<br>
<br>
(abs(a - b))/abs(a)<br>
<br>
versus<br>
<br>
(abs(a - b))/abs(b)<br>
<br>
<br>
you want the *larger* error estimate, which means the *smaller*<br>
denominator. That's the conservative way of doing it.<br></blockquote><div><br></div><div>Which is what the Boost "strong" method does -- rather than compute teh max and use that, it computes both and does an "and" check -- but same result. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
A concrete example: given a=5 and b=7, we have:<br>
<br>
absolute error = 2<br>
relative error (calculated relative to a) = 0.4<br>
relative error (calculated relative to b) = 0.286<br>
<br>
That is, b is off by 40% relative to a; or a is off by 28.6% relative to<br>
b. Or another way to put it, given that a is the "true" value, b is 40%<br>
too big; or if you prefer, 28.6% of b is in error.<br></blockquote><div><br></div><div>The think is, in general, we use this to test for small errors, with low tolerance. Which value you use to scale only makes a big difference if the values are far apart, in which case the error will be larger than the tolerance anyway.</div><div><br></div><div>In your above example, if the tolerance is, say, 1%, then if makes no difference which you use -- you are way off anyway. And in the common use cases, comparing a double precision floating point calculation, tolerances are more likely to be around 1e-12, not 1e-2 anyway!</div><div><br></div><div>So I think that which relative tolerance you use makes little difference in practice, but it might as well be robust and symmetrical.</div><div><br></div><div>(another option is to use the average of the two values to scale the tolerance, but why bother?)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">> Note that you would never compare to an expected value of zero.<br>
<br>
</span>You *cannot* compare to an expected value of zero, but you certainly can<br>
be in a situation where you would like to: math.sin(math.pi) should<br>
return 0.0, but doesn't, it returns 1.2246063538223773e-16 instead. What<br>
is the relative error of the sin function at x = math.pi?<br></blockquote><div><br></div><div>there isn't one -- that's the whole point -- but there is an absolute error, so that's what you should check.</div><div><br></div><div>We all agree a relative error involving zero is not defined / possible. So the question is what to do?</div><div><br></div><div>1) Raise a ValueError</div><div>2) Let it return "not close" regardless of the other input -- that's mathematically correct, nothing is relatively close to zero.</div><div>3) Automagically switch to an absolute tolerance near zero -- user specified what it should be.</div><div> </div><div>It seems the implementations (Boost's, for instance) I've seen simply do (2). But if the point of putting this in the standard library is that people will have something that can be used for common use cases without thinking about it, I think maybe (1) or (3) would be better. Probably (3), as raising an Exception would make a mess of this if it were inside a comprehension or something.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">> What Chris is looking for is a way to get a closeness function that works<br>
> most of the time. (He posted while I'm writing this.)<br>
<br>
</span>I think the function I have in the statistics test suite is that<br>
function. </blockquote><div><br></div><div>I'll take a look -- it does sound like you've already done pretty much what I have in mind.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I would like to see ULP calculations offered as well, but<br>
Mark thinks that's unnecessary and I'm not going to go to the<br>
battlements to fight for ULPs.</blockquote><div><br></div><div>I suppose it could be added later -- I agree that it could be pretty useful, but that it's also much harder to wrap your brain around, and really for a different use-case.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">* you provide two values, and at least one of an absolute error<br>
tolerance and a relative error;<br>
* if the error is less than the error(s) you provided, the test<br>
passes, otherwise it fails;<br>
* NANs and INFs are handled appropriately.</blockquote><div><br></div><div>Is this different than the numpy implementation:</div><div><br></div><div><a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html">http://docs.scipy.org/doc/numpy/reference/generated/numpy.allclose.html</a> </div><div><br></div><div>In that (according to the docs, I haven't looked at the code):</div><div><br></div><div><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">"""</span></div><div><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">The relative difference (</span><em class="" style="font-style:italic;color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">rtol</em><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px"> * abs(</span><em class="" style="font-style:italic;color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">b</em><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">)) and the absolute difference </span><em class="" style="font-style:italic;color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">atol</em><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px"> are added together to compare against the absolute difference between </span><em class="" style="font-style:italic;color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">a </em><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">and </span><em class="" style="font-style:italic;color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">b</em><span style="color:rgb(51,51,51);font-family:'Open Sans',sans-serif;font-size:13px;line-height:19px">.</span><br></div><div>"""</div><div><br></div><div>I think it should either be: </div><div><br></div><div>- you specify atol or rtol, but only one is used.</div><div><br></div><div>or</div><div><br></div><div>- some way to transition from a relative tolerance to an absolute on near zero -- I a haven't figured out if that can be done smoothly yet.</div><div><br></div><div>[Also, it looks like numpy computes the tolerance from the second input, rather than looking at both, resulting in an asymetric result -- discussed above.)</div><div><br></div><div>I've always thought the numpy approach is weird, but now that I think about it, it would be really horrible (with defaults) for small numbers:</div><div><br></div><div>rtol defaults to 1e-5, atol to 1e-8 -- too big, I think, but not the point here.</div><div><br></div><div><div>In [23]: a, b = 1.1, 1.2</div><div><br></div><div>In [24]: np.allclose(a,b)</div><div>Out[24]: False</div></div><div>## that's good there are pretty far apart</div><div><br></div><div><div>In [27]: a, b = 1.1e15, 1.2e15</div><div><br></div><div>In [28]: np.allclose(a,b)</div><div>Out[28]: False</div></div><div><br></div><div># same thing for large values -- still good.</div><div><br></div><div><div>In [25]: a, b = 1.1e-15, 1.2e-15</div><div><br></div><div>In [26]: np.allclose(a,b)</div><div>Out[26]: True</div></div><div><br></div><div>OOPS! this is NOT what most people would expect!!</div><div><div><br></div><div>In [30]: np.allclose(a,b, atol=0.0)</div><div>Out[30]: False</div></div><div><br></div><div>There we go. But with a default atol as large as 1e-8, this is a rally bad idea!</div><div><br></div><div><div>I can only imagine whoever wrote this was thinking about really large values, but not really small values...</div></div><div><br></div><div>(I think this has been brought up in the numpy community, but I'll make sure)</div><div><br></div><div>-Chris</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<br>
> is_close(218.345, 220, 1, .05) # OHMs<br>
> is_close(a, b, ULP, 2) # ULPs<br>
> is_close(a, b, AU, .001) # astronomical units<br>
><br>
><br>
> I don't see anyway to generalise those with just a function.<br>
<br>
</span>Generalise in what way?<br>
<span class=""><br>
<br>
> By using objects we can do a bit more. I seem to recall coming across<br>
> measurement objects some place. They keep a bit more context with them.<br>
<br>
</span>A full system of <value + unit> arithmetic is a *much* bigger problem<br>
than just calculating error estimates correctly, and should be a<br>
third-party library before even considering it for the std lib.<br>
<span class=""><font color="#888888"><br>
<br>
--<br>
Steve<br>
</font></span><div class=""><div class="h5">_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" target="_blank">http://python.org/psf/codeofconduct/</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div>