On Fri, Jul 18, 2014 at 2:29 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Jul 18, 2014 at 7:03 PM,  <josef.pktd@gmail.com> wrote:
>
> On Fri, Jul 18, 2014 at 12:53 PM, Nathaniel Smith <njs@pobox.com> wrote:
>>
>> For cases like this, you need *some* non-zero atol or the thing just
>> doesn't work, and one could quibble over the exact value as long as
>> it's larger than "normal" floating point error. These calculations
>> usually involve "normal" sized numbers, so atol should be comparable
>> to eps * these values.  eps is 2e-16, so atol=1e-8 works for values up
>> to around 1e8, which is a plausible upper bound for where people might
>> expect assert_allclose to just work. I'm trying to figure out some way
>> to support your use cases while also supporting other use cases.
>
>
> my problem is that there is no "normal" floating point error.
> If I have units in 1000 or units in 0.0001 depends on the example and
> dataset that we use for testing.
>
> this test two different functions/methods that calculate the same thing
>
> (Pdb) pval
> array([  3.01270184e-42,   5.90847367e-02,   3.00066946e-12])
> (Pdb) res2.pvalues
> array([  3.01270184e-42,   5.90847367e-02,   3.00066946e-12])
> (Pdb) assert_allclose(pval, res2.pvalues, rtol=5 * rtol, atol=1e-25)
>
> I don't care about errors that are smaller that 1e-25
>
> for example testing p-values against Stata
>
> (Pdb) tt.pvalue
> array([  5.70315140e-30,   6.24662551e-02,   5.86024090e-11])
> (Pdb) res2.pvalues
> array([  5.70315140e-30,   6.24662551e-02,   5.86024090e-11])
> (Pdb) tt.pvalue - res2.pvalues
> array([  2.16612016e-40,   2.51187959e-15,   4.30027936e-21])
> (Pdb) tt.pvalue / res2.pvalues - 1
> array([  3.79811738e-11,   4.01900735e-14,   7.33806349e-11])
> (Pdb) rtol
> 1e-10
> (Pdb) assert_allclose(tt.pvalue, res2.pvalues, rtol=5 * rtol)
>
>
> I could find a lot more and maybe nicer examples, since I spend quite a bit
> of time fine tuning unit tests.

...these are all cases where there are not exact zeros, so my proposal
would not affect them?

I can see the argument that we shouldn't provide any default rtol/atol
at all because there is no good default, but... I don't think putting
that big of a barrier in front of newbies writing their first tests is
a good idea.

I think atol=0 is **very** good for newbies, and everyone else.
If expected is really zero or very small, then it immediately causes a test failure, and it's relatively obvious how to fix it.

I worry a lot more about unit tests that don't "bite" written by newbies or not so newbies who just use a default.

That's one of the problems we had with assert_almost_equal, and why I was very happy to switch to assert_allclose with it's emphasis on relative tolerance.

Josef

 

-n

--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion