[Numpy-discussion] Warnings in numpy.ma.test()

[Bruce Southey]

On 03/17/2010 01:07 AM, Pierre GM wrote:
> All,
> As you're probably aware, the current test suite for numpy.ma raises some nagging warnings such as "invalid value in ...". These warnings are only issued when a standard numpy ufunc (eg., np.sqrt) is called on a MaskedArray, instead of its numpy.ma (eg., np.ma.sqrt) equivalent. The reason is that the masked versions of the ufuncs temporarily set the numpy error status to 'ignore' before the operation takes place, and reset the status to its original value.
>    
Perhaps naive question, what is really being tested here?

That is, it appears that you are testing both the generation of the 
invalid values and function. So if the generation fails, then the 
function will also fail. However, the test for the generation of invalid 
values  should be elsewhere so you have to assume that the generation of 
values will work correctly.

I think that you should be only testing that the specific function 
passes the test. Why not just use 'invalid' values like np.inf directly?

For example, in numpy/ma/tests/test_core.py
We have this test:
     def test_fix_invalid(self):
         "Checks fix_invalid."
         data = masked_array(np.sqrt([-1., 0., 1.]), mask=[0, 0, 1])
         data_fixed = fix_invalid(data)

If that is to test that fix_invalid Why not create the data array as:
data = masked_array([np.inf, 0., 1.]), mask=[0, 0, 1])

However, I am not sure the output should be for the test_ndarray_mask 
test because ma automatically masks the value resulting from sqrt(-1):
 >>> a = masked_array([-1, 0, 1, 2, 3], mask=[0, 0, 0, 0, 1])
 >>> np.sqrt(a)
Warning: invalid value encountered in sqrt
masked_array(data = [-- 0.0 1.0 1.41421356237 --],
              mask = [ True False False False  True],
        fill_value = 999999)

Note the warning is important because it does indicate that the result 
might not be as expected.
But if the -1 is replaced by np.inf, then it is not automatically masked:
 >>> b = masked_array([np.inf, 0, 1, 2, 3], mask=[0, 0, 0, 0, 1])
 >>> np.sqrt(b)
masked_array(data = [inf 0.0 1.0 1.41421356237 --],
              mask = [False False False False  True],
        fill_value = 1e+20)

Bruce


> I thought I could use the new __array_prepare__ method to intercept the call of a standard ufunc. After actual testing, that can't work. __array_prepare only help to prepare the *output* of the operation, not to change the input on the fly, just for this operation. Actually, you can modify the input in place, but it's usually not what you want.
> Then, I tried to use  __array_prepare__ to store the current error status in the input, force it to ignore divide/invalid errors and send the input to the ufunc. Doesn't work either: np.seterr in __array_prepare__ does change the error status, but as far as I understand, the ufunc is called is still called with the original error status. That means that if something goes wrong, your error status can stay stuck. Not a good idea either.
> I'm running out of ideas at this point. For the test suite, I'd suggest to disable the warnings in test_fix_invalid and test_basic_arithmetic.
> An additional issue is that if one of the error status is set to 'raise', the numpy ufunc will raise the exception (as expected), while its numpy.ma version will not. I'll put also a warning in the docs to that effect.
> Please send me your comments before I commit any changes.
> Cheers,
> P.
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>