[Numpy-discussion] Inversion of near singular matrices.
akabaila at pcug.org.au
Sun Jan 30 00:11:23 EST 2011
On Sunday 30 January 2011 09:10:30 Sturla Molden wrote:
> Den 29.01.2011 12:40, skrev Algis Kabaila:
> > So my question is: how can one reliably detect singularity
> > (or near singularity) and raise an exception?
> Use an SVD, examine the singular values.
I gather that SVD is the Singular Value Decomposition, but I
have no idea how to perform such decomposition. Would you care
to refer me to some simple source material? I have been advised
to watch the condition numbers. No doubt, SVD and condition
numbers are related. The references about condition numbers are
very interesting and I intend to follow them in the first
> In statistics we sometimes see ill-conditioning of covariance
> matrices. Another way to deal with multicollinearity besides
> SVD/PCA is regularisation. Simply adding a small bias k*I to
> the diagonal might fix the problem (cf. ridge regression).
> In the Levenberg-Marquardt algorithm used to fit non-linear
> least squares models (cf.
> scipy.optimize.leastsq), the bias k to the diagonal of the
> Jacobian is changed adaptively. One might also know in
> advance if a covariance matrix could be ill-conditioned (the
> number of samples is small compared to the number of
> dimensions) or singular (less data than parameters). That
> is, sometimes we don't even need to look at the matrix to
> give the correct diagnosis. Another widely used strategy is
> to use Cholesky factorization on covariance matrices. It is
> always stable unless there is a singularity, for which it
> will fail (NumPy will raise a LinAlgError exception).
> Cholesky is therefore safer to use for inverting covariance
> matrices than LU (as well as faster). If Cholesky fails one
> might fallback to SVD or regularisation to correct the
My knowledge of statistics is rather limited, though our son Dr.
Paul Kabaila is a specialist in that area. My interests lie in
the area of Analysis of Engineering Structures - it saves my
brain from falling to a permafrost like sleep :)
Thank you for your reply - greatly appreciated.
PS: Paul, I thought there is a minuscule chance that this is of
some interest to you.
More information about the NumPy-Discussion