[scikit-learn] Problem with nested cross-validation example?

[Joel Nothman]

If that clarifies, please offer changes to the example (as a pull request)
that make this clearer.

On 29 November 2016 at 11:06, Joel Nothman <joel.nothman at gmail.com> wrote:

> Briefly:
>
> clf = GridSearchCV <http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV>(estimator=svr, param_grid=p_grid, cv=inner_cv)nested_score = cross_val_score <http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score>(clf, X=X_iris, y=y_iris, cv=outer_cv)
>
>
> Each train/test split in cross_val_score holds out test data. GridSearchCV
> then splits each train set into (inner-)train and validation sets. There is
> no leakage of test set knowledge from the outer loop into the grid search
> optimisation; no leakage of validation set knowledge into the SVR
> optimisation. The outer test data are reused as training data, but within
> each split are only used to measure generalisation error.
>
> Is that clear?
>
> On 29 November 2016 at 10:30, Daniel Homola <dani.homola at gmail.com> wrote:
>
>> Dear all,
>>
>>
>> I was wondering if the following example code is valid:
>>
>> http://scikit-learn.org/stable/auto_examples/model_selection
>> /plot_nested_cross_validation_iris.html
>>
>> My understanding is, that the point of nested cross-validation is to
>> prevent any data leakage from the inner grid-search/param optimization CV
>> loop into the outer model evaluation CV loop. This could be achieved if the
>> outer CV loop's test data is completely separated from the inner loop's CV,
>> as shown here:
>>
>> https://mlr-org.github.io/mlr-tutorial/release/html/img/nest
>> ed_resampling.png
>>
>>
>> The code in the above example however doesn't seem to achieve this in any
>> way.
>>
>>
>> Am I missing something here?
>>
>>
>> Thanks a lot,
>>
>> dh
>>
>> _______________________________________________
>> scikit-learn mailing list
>> scikit-learn at python.org
>> https://mail.python.org/mailman/listinfo/scikit-learn
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20161129/6db54f94/attachment-0001.html>