[scikit-learn] How does multiple target Ridge Regression work in scikit learn?

[Michael Eickenberg]

By the linear nature of the problem the targets are always separately
treated (even if there was a matrix-variate normal prior indicating
covariance between target columns, you could do that adjustment before or
after fitting).
As for different alpha parameters, I think you can specify a different
alpha per target if you pass in an array of shape (n_targets,). Maybe this
is not implemented for all solvers, but it should be at least for some.

If you grid search, then the scikit-learn API requires the score to be one
number, so it's non-trivial to optimize different alphas for different
voxels easily (even though selecting the best alpha for each voxel will of
course make the sum of errors go down, too).

Depending on what your use case is, it may be easier to just write your own:

If X = U S VT (svd), then weights = VT.T.dot((1 / (S ** 2 + alpha) *
U).T.dot(Y))

For more than one alpha:
alphas.shape == (n_alphas, n_targets)
Y.shape == (n_samples, n_targets)
X.shape == (n_samples, n_features)


U, S, VT = np.linalg.svd(X)
diags = 1 / (S[np.newaxis, :, np.newaxis] ** 2 + alphas[:, np.newaxis, :])
UTY = U.T.dot(Y)
weights = np.zeros([n_alphas, n_features, n_targets])
for i in range(alphas.shape[0]):
    weights[i] = VT.T.dot(diags[i] * UTY)

Then use those weights to predict.

Michael


On Wed, May 2, 2018 at 6:02 AM, Peer Nowack <peer.j.nowack at gmail.com> wrote:

> Thanks, Bertrand - very helpful. Needed to consolidate this.
>
> Peter
>
> On 2 May 2018 at 13:07, bthirion <bertrand.thirion at inria.fr> wrote:
>
>> The alpha parameter is shared for all problems; If you wnat to use
>> differnt parameters, you probably want to perform seprate fits.
>> Best,
>>
>> Bertrand
>>
>> On 02/05/2018 13:08, Peer Nowack wrote:
>>
>> Hi all,
>>
>> I am struggling to understand the following:
>>
>> Scikit-learn offers a multiple output version for Ridge Regression,
>> simply by handing over a 2D array [n_samples, n_targets], but how is it
>> implemented?
>>
>> http://scikit-learn.org/stable/modules/generated/sklearn.
>> linear_model.Ridge.html
>>
>> Is it correct to assume that each regression for each target is
>> independent? Under these circumstances, how can I adapt this to use
>> individual alpha regularization parameters for each regression? If I use
>> GridSeachCV, I would have to hand over a matrix of possible regularization
>> parameters, or how would that work?
>>
>> Thanks in advance - I have been searching for hours but could not find
>> anything on this topic.
>> Peter
>>
>>
>> _______________________________________________
>> scikit-learn mailing listscikit-learn at python.orghttps://mail.python.org/mailman/listinfo/scikit-learn
>>
>>
>>
>> _______________________________________________
>> scikit-learn mailing list
>> scikit-learn at python.org
>> https://mail.python.org/mailman/listinfo/scikit-learn
>>
>>
>
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20180502/e502eb20/attachment.html>