<p dir="ltr">If you want a proper self-consistent correlation/covariance matrix, then pairwise deletion just makes no sense period, I don't see how postprocessing can help.</p>
<p dir="ltr">If you want a matrix of correlations, then pairwise deletion makes sense. It's an interesting point that arguably the current ma.corrcoef code may actually give you a better estimator of the individual correlation coefficients than doing full pairwise deletion, but it's pretty surprising and unexpected... when people call corrcoef I think they are asking "please compute the textbook formula for 'sample correlation'" not "please give me some arbitrary good estimator for the population correlation", so we probably have to change it.</p>
<p dir="ltr">(Hopefully no-one has published anything based on the current code.)</p>
<p dir="ltr">-n</p>
<div class="gmail_quote">On 26 Sep 2013 04:19, <<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Wed, Sep 25, 2013 at 11:05 PM, <<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>> wrote:<br>
> On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei <<a href="mailto:fmmirzaei@gmail.com">fmmirzaei@gmail.com</a>> wrote:<br>
>> Hi everyone,<br>
>><br>
>> I'm using np.ma.corrcoef to compute the correlation coefficients among rows<br>
>> of a masked matrix, where the masked elements are the missing data. I've<br>
>> observed that in some cases, the np.ma.corrcoef gives invalid coefficients<br>
>> that are greater than 1 or less than -1.<br>
>><br>
>> Here's an example:<br>
>><br>
>> x = array([[ 7, -4, -1, -7, -3, -2],<br>
>> [ 6, -3, 0, 4, 0, 5],<br>
>> [-4, -5, 7, 5, -7, -7],<br>
>> [-5, 5, -8, 0, 1, 4]])<br>
>><br>
>> x_ma = np.ma.masked_less_equal(x , -5)<br>
>><br>
>> C = np.round(np.ma.corrcoef(x_ma), 2)<br>
>><br>
>> print C<br>
>><br>
>> [[1.0 0.73 -- -1.68]<br>
>> [0.73 1.0 -0.86 -0.38]<br>
>> [-- -0.86 1.0 --]<br>
>> [-1.68 -0.38 -- 1.0]]<br>
>><br>
>> As you can see, the [0,3] element is -1.68 which is not a valid correlation<br>
>> coefficient. (Valid correlation coefficients should be between -1 and 1).<br>
>><br>
>> I looked at the code for np.ma.corrcoef, and this behavior seems to be due<br>
>> to the way that mean values of the rows of the input matrix are computed and<br>
>> subtracted from them. Apparently, the mean value is individually computed<br>
>> for each row, without masking the elements corresponding to the masked<br>
>> elements of the other row of the matrix, with respect to which the<br>
>> correlation coefficient is being computed.<br>
>><br>
>> I guess the right way should be to recompute the mean value for each row<br>
>> every time that a correlation coefficient is being computed for two rows<br>
>> after propagating pair-wise masked values.<br>
>><br>
>> Please let me know what you think.<br>
><br>
> just general comments, I have no experience here<br>
><br>
> From what you are saying it sounds like <a href="http://np.ma" target="_blank">np.ma</a> is not doing pairwise<br>
> deletion in calculating the mean (which only requires ignoring<br>
> missings in one array), however it does (correctly) do pairwise<br>
> deletion in calculating the cross product.<br>
<br>
Actually, I think the calculation of the mean is not relevant for<br>
having weird correlation coefficients without clipping.<br>
<br>
With pairwise deletion you use different samples, subsets of the data,<br>
for the variances and the covariances.<br>
It should be easy (?) to construct examples where the pairwise<br>
deletion for the covariance produces a large positive or negative<br>
number, and both variances and standard deviations are small, using<br>
two different subsamples.<br>
Once you calculate the correlation coefficient, it could be all over<br>
the place, independent of the mean calculations.<br>
<br>
conclusion: pairwise deletion requires post-processing if you want a<br>
proper correlation matrix.<br>
<br>
Josef<br>
<br>
><br>
> covariance or correlation matrices with pairwise deletion are not<br>
> necessarily "proper" covariance or correlation matrices.<br>
> I've read that they don't need to be positive semi-definite, but I've<br>
> never heard of values outside of [-1, 1]. It might only be a problem<br>
> if you have a large fraction of missing values..<br>
><br>
> I think the current behavior in <a href="http://np.ma" target="_blank">np.ma</a> makes sense in that it uses all<br>
> the information available in estimating the mean, which should be more<br>
> accurate if we use more information. But it makes cov and corrcoef<br>
> even weirder than they already are with pairwise deletion.<br>
><br>
> Row-wise deletion (deleting observations that have at least one<br>
> missing), which would create "proper" correlation matrices, wouldn't<br>
> produce much in your example.<br>
><br>
> I would check what R or other packages are doing and follow their<br>
> lead, or add another option.<br>
> (similar: we had a case in statsmodels where I used initially all<br>
> information for calculating the mean, but then we dropped some<br>
> observations to match the behavior of Stata, and to use the same<br>
> observations for calculating the mean and the follow up statistics.)<br>
><br>
><br>
> looks like pandas might be truncating the correlations to [-1, 1] (I<br>
> didn't check)<br>
><br>
>>>> import pandas as pd<br>
>>>> x_pd = pd.DataFrame(x_ma.T)<br>
>>>> x_pd.corr()<br>
> 0 1 2 3<br>
> 0 1.000000 0.734367 -1.000000 -0.240192<br>
> 1 0.734367 1.000000 -0.856565 -0.378777<br>
> 2 -1.000000 -0.856565 1.000000 NaN<br>
> 3 -0.240192 -0.378777 NaN 1.000000<br>
><br>
>>>> np.round(np.ma.corrcoef(x_ma), 6)<br>
> masked_array(data =<br>
> [[1.0 0.734367 -1.190909 -1.681346]<br>
> [0.734367 1.0 -0.856565 -0.378777]<br>
> [-1.190909 -0.856565 1.0 --]<br>
> [-1.681346 -0.378777 -- 1.0]],<br>
> mask =<br>
> [[False False False False]<br>
> [False False False False]<br>
> [False False False True]<br>
> [False False True False]],<br>
> fill_value = 1e+20)<br>
><br>
><br>
> Josef<br>
><br>
><br>
>><br>
>> Thanks,<br>
>><br>
>> Faraz<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> NumPy-Discussion mailing list<br>
>> <a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
>> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
>><br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
</blockquote></div>