[Numpy-discussion] Mysterious test_pareto failure on Travis

[Ondřej Čertík]

On Tue, Sep 4, 2012 at 2:58 PM,  <josef.pktd at gmail.com> wrote:
> On Tue, Sep 4, 2012 at 5:49 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>> On Tue, Sep 4, 2012 at 1:48 PM,  <josef.pktd at gmail.com> wrote:
>>> On Tue, Sep 4, 2012 at 4:37 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>>> On Tue, Sep 4, 2012 at 9:17 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>>>> On Tue, Sep 4, 2012 at 12:41 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>>>>> On Tue, Sep 4, 2012 at 12:31 PM, Ondřej Čertík <ondrej.certik at gmail.com> wrote:
>>>>>>> On Tue, Sep 4, 2012 at 3:15 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>>>>>>> The last two Travis builds of master have failed consistently with the
>>>>>>>> same error:
>>>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds
>>>>>>>> It looks like a real failure -- we're getting the same error on every
>>>>>>>> build variant, some sort of problem in test_pareto. Example:
>>>>>>>>   http://travis-ci.org/#!/numpy/numpy/jobs/2328823
>>>>>>>>
>>>>>>>> The obvious culprit would be the previous commit, which regenerated
>>>>>>>> mtrand.c with Cython 0.17:
>>>>>>>>   http://github.com/numpy/numpy/commit/cd9092aa71d23359b33e89d938c55fb14b9bf606
>>>>>>>>
>>>>>>>> What's weird, though, is that that commit passed just fine on Travis:
>>>>>>>>   http://travis-ci.org/#!/numpy/numpy/builds/2313124
>>>>>>>>
>>>>>>>> It's just the two commits since then that failed. But these commits
>>>>>>>> have been 1-line docstring changes, so I don't see how they could have
>>>>>>>> possibly created the problem.
>>>>>>>>
>>>>>>>> Also, the test passes fine with python 2.7 on my laptop with current master.
>>>>>>>>
>>>>>>>> Can anyone reproduce this failure? Any ideas what might be going on?
>>>>>>>
>>>>>>> I made this:
>>>>>>>
>>>>>>> https://github.com/numpy/numpy/issues/424
>>>>>>>
>>>>>>> It was me who updated the Cython file. It seemed to be working. I've
>>>>>>> added the issue
>>>>>>> to the release TODO.
>>>>>>
>>>>>> Ok, here is how to reproduce the problem:
>>>>>>
>>>>>> 1) install my numpy-vendor vagrant image (32 bit Ubuntu), as directed
>>>>>> in the README:
>>>>>>
>>>>>> https://github.com/certik/numpy-vendor
>>>>>>
>>>>>> 2) run tests, you'll get:
>>>>>>
>>>>>> https://gist.github.com/3625509
>>>>>
>>>>> So the problem was actually introduced much earlier. Most probably it
>>>>> has never worked
>>>>> in 32bit Ubuntu 12.04. I tried for example this old commit:
>>>>>
>>>>> https://github.com/numpy/numpy/commit/3882d65c42acf6d5fff8cc9b3f410bb3e49c8af8
>>>>>
>>>>> and it still fails:
>>>> [...]
>>>>> But in any case, this seems to be a problem with the actual 32bit
>>>>> Ubuntu 12.04 itself. So maybe something in gcc
>>>>> has changed that now triggers the problem.
>>>>
>>>> To be clear, the mismatching value is:
>>>>
>>>>> /home/njs/numpy/.tox/py27/local/lib/python2.7/site-packages/numpy/random/tests/test_random.py(363)test_pareto()
>>>> -> np.testing.assert_array_almost_equal(actual, desired, decimal=15)
>>>> (Pdb) actual[1, 0]
>>>> 52828779.702948704
>>>> (Pdb) desired[1, 0]
>>>> 52828779.702948518
>>>>
>>>> So they do match to 14 decimal points, and it's entirely possible that
>>>> this is just a problem of the test being too stringent in requiring 15
>>>> decimal points of match. Maybe the 32-bit GCC is spilling registers
>>>> differently, tripping the famous x86 idiosyncrasy where register
>>>> spilling triggers rounding. But I'd feel better if someone familiar
>>>> with the pareto code could confirm.
>>>
>>> I don't understand this. Isn't assert_almost_equal testing decimals
>>> not significant digits?
>>
>> Yes it is, see the docs:
>>
>>     The test is equivalent to ``abs(desired-actual) < 0.5 * 10**(-decimal)``.
>>
>> and the unit test is badly written, because that floating point
>> number has maybe 14 or 15 good significant digits, but it simply does not have
>> 15 digits after the decimal point, so in particular, that test is
>> testing that the two
>> floating point numbers are exactly equal.
>>
>>
>> Here is a fix:
>>
>> https://github.com/numpy/numpy/pull/425
>>
>> Let me know if it is ok to merge it, so that we can work on other
>> issues and have a working test suite.
>
> I think  all the tests should be changed to use the new assert with
> relative tolerance (similar to approx equal)
> https://gist.github.com/3625943  shows also the other distributions
> with 15 decimals when the values are larger than 10 for example

That sounds reasonable. Josef, would you mind sending a PR with this better fix?
The PR #425 fixes it for now and for me it's now very important to get
all the tests pass again on Travis due to their upgrade.

I am now working on other failures at Travis:

https://github.com/numpy/numpy/issues/394
https://github.com/numpy/numpy/issues/426

As those seem to be causing all the Debian buildbots test failures
reported by Sandro.

Ondrej