<div dir="ltr">Wait an hour for the docs to build and you won't get artifact not found :)<div><br></div><div>If you'd looked at the PR diff, you'd see I've modified the description to refer directly to GridSearchCV and cross_val_score:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(29,31,34);font-family:helvetica,arial,sans-serif;font-size:14.4px">In the inner loop (here executed by </span><code class="gmail-xref gmail-py gmail-py-class gmail-docutils gmail-literal" style="padding:2px 4px;font-family:monospace;font-size:1.1em;border-radius:3px;border:none;white-space:nowrap;font-weight:bold"><span class="gmail-pre">GridSearchCV</span></code><span style="color:rgb(29,31,34);font-family:helvetica,arial,sans-serif;font-size:14.4px">), the score is approximately maximized by fitting a model to each training set, and then directly maximized in selecting (hyper)parameters over the validation set. In the outer loop (here in </span><code class="gmail-xref gmail-py gmail-py-func gmail-docutils gmail-literal" style="padding:2px 4px;font-family:monospace;font-size:1.1em;border-radius:3px;border:none;white-space:nowrap;font-weight:bold"><span class="gmail-pre">cross_val_score</span></code><span style="color:rgb(29,31,34);font-family:helvetica,arial,sans-serif;font-size:14.4px">), ...</span></blockquote><div><span style="color:rgb(29,31,34);font-family:helvetica,arial,sans-serif;font-size:14.4px"><br></span></div><div>Further comments in the code are welcome.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 29 November 2016 at 21:42, Albert Thomas <span dir="ltr"><<a href="mailto:albertthomas88@gmail.com" target="_blank">albertthomas88@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I also get "artifact not found". And I agree with Daniel.<br><br>Once you decompose what the code is doing you realize that it does the job. The simplicity of the code to perform nested cross validation using scikit learn objects is impressive but I guess it also makes it less obvious. So making the example clearer by explaining what the code does or by adding a few comments can be useful for others.<span class="HOEnZb"><font color="#888888"><br><br>Albert <br></font></span><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, 29 Nov 2016 at 11:19, Daniel Homola <<a href="mailto:daniel.homola11@imperial.ac.uk" target="_blank">daniel.homola11@imperial.ac.<wbr>uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_6097130876457162951gmail_msg">
<p class="m_6097130876457162951gmail_msg">Hi Joel,</p>
<p class="m_6097130876457162951gmail_msg">Thanks a lot for the answer.<br class="m_6097130876457162951gmail_msg">
</p></div><div bgcolor="#FFFFFF" text="#000000" class="m_6097130876457162951gmail_msg">
<p class="m_6097130876457162951gmail_msg">"Each train/test split in cross_val_score holds out test data.
GridSearchCV then splits each train set into (inner-)train and
validation sets. "</p>
</div><div bgcolor="#FFFFFF" text="#000000" class="m_6097130876457162951gmail_msg"><p class="m_6097130876457162951gmail_msg">I know this is what nested CV supposed to do but the code is
doing an excellent job at obscuring this. I'll try and add some
clarification in as comments later today.</p>
<p class="m_6097130876457162951gmail_msg">Cheers,</p>
<p class="m_6097130876457162951gmail_msg">d<br class="m_6097130876457162951gmail_msg">
</p></div><div bgcolor="#FFFFFF" text="#000000" class="m_6097130876457162951gmail_msg">
<p class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</p>
<div class="m_6097130876457162951m_1181111257658593422moz-cite-prefix m_6097130876457162951gmail_msg">On 29/11/16 00:07, Joel Nothman wrote:<br class="m_6097130876457162951gmail_msg">
</div>
<blockquote type="cite" class="m_6097130876457162951gmail_msg">
<div dir="ltr" class="m_6097130876457162951gmail_msg">If that clarifies, please offer changes to the
example (as a pull request) that make this clearer.</div>
<div class="gmail_extra m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
<div class="gmail_quote m_6097130876457162951gmail_msg">On 29 November 2016 at 11:06, Joel
Nothman <span dir="ltr" class="m_6097130876457162951gmail_msg"><<a href="mailto:joel.nothman@gmail.com" class="m_6097130876457162951gmail_msg" target="_blank">joel.nothman@gmail.com</a>></span>
wrote:<br class="m_6097130876457162951gmail_msg">
<blockquote class="gmail_quote m_6097130876457162951gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_6097130876457162951gmail_msg">Briefly:
<div class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</div>
<div class="m_6097130876457162951gmail_msg">
<pre style="padding:5px 10px;font-family:monaco,menlo,consolas,"courier new",monospace;font-size:13px;border-radius:4px;margin-top:0.1em;margin-bottom:0.5em;line-height:1.2em;word-break:break-all;word-wrap:break-word;white-space:pre-wrap;background-color:rgb(248,248,248);border:1px solid rgb(221,221,221);overflow-x:auto;overflow-y:hidden" class="m_6097130876457162951gmail_msg"><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">clf</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span> <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" style="color:rgb(40,120,162);word-wrap:break-word" class="m_6097130876457162951gmail_msg" target="_blank"><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">GridSearchCV</span></a><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">(</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">estimator</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">svr</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">,</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">param_grid</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">p_grid</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">,</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">cv</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">inner_cv</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">)</span>
<span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">nested_score</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span> <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score" style="color:rgb(40,120,162);word-wrap:break-word" class="m_6097130876457162951gmail_msg" target="_blank"><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">cross_val_score</span></a><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">(</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">clf</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">,</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">X</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">X_iris</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">,</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">y</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">y_iris</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">,</span> <span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">cv</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-o m_6097130876457162951gmail_msg" style="color:rgb(102,102,102)">=</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-n m_6097130876457162951gmail_msg">outer_cv</span><span class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897gmail-p m_6097130876457162951gmail_msg">)</span></pre>
</div>
<div class="gmail_extra m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
<div class="gmail_quote m_6097130876457162951gmail_msg">Each train/test split in
cross_val_score holds out test data. GridSearchCV then
splits each train set into (inner-)train and
validation sets. There is no leakage of test set
knowledge from the outer loop into the grid search
optimisation; no leakage of validation set knowledge
into the SVR optimisation. The outer test data are
reused as training data, but within each split are
only used to measure generalisation error.</div>
<div class="gmail_quote m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
Is that clear?</div>
<div class="gmail_quote m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</div>
<div class="gmail_quote m_6097130876457162951gmail_msg">
<div class="m_6097130876457162951gmail_msg">
<div class="m_6097130876457162951m_1181111257658593422h5 m_6097130876457162951gmail_msg">On 29 November 2016 at 10:30, Daniel
Homola <span dir="ltr" class="m_6097130876457162951gmail_msg"><<a href="mailto:dani.homola@gmail.com" class="m_6097130876457162951gmail_msg" target="_blank">dani.homola@gmail.com</a>></span>
wrote:<br class="m_6097130876457162951gmail_msg">
</div>
</div>
<blockquote class="gmail_quote m_6097130876457162951gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="m_6097130876457162951gmail_msg">
<div class="m_6097130876457162951m_1181111257658593422h5 m_6097130876457162951gmail_msg">
<div dir="ltr" class="m_6097130876457162951gmail_msg">
<div class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897m_-6276298983508018080gmail_signature m_6097130876457162951gmail_msg">
<div class="m_6097130876457162951gmail_msg">
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">Dear
all,</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">I
was wondering if the following example
code is valid:</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><a href="http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html" class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897m_-6276298983508018080gmail-m_8935104698084405240gmail-x_OWAAutoLink m_6097130876457162951gmail_msg" id="m_6097130876457162951m_1181111257658593422m_-1140303242554994897m_-6276298983508018080gmail-m_8935104698084405240gmail-LPlnk436896" target="_blank">http://scikit-learn.org/<wbr>stable/auto_examples/model_<wbr>selection/plot_nested_cross_<wbr>validation_iris.html</a><br class="m_6097130876457162951gmail_msg">
<br class="m_6097130876457162951gmail_msg">
</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">My
understanding is, that the point of
nested cross-validation is to prevent
any data leakage from the
inner grid-search/param optimization CV
loop into the outer model evaluation CV
loop. This could be achieved if the
outer CV loop's test data is completely
separated from the inner loop's CV, as
shown here:</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><a href="https://mlr-org.github.io/mlr-tutorial/release/html/img/nested_resampling.png" class="m_6097130876457162951m_1181111257658593422m_-1140303242554994897m_-6276298983508018080gmail-m_8935104698084405240gmail-x_OWAAutoLink m_6097130876457162951gmail_msg" id="m_6097130876457162951m_1181111257658593422m_-1140303242554994897m_-6276298983508018080gmail-m_8935104698084405240gmail-LPlnk683151" target="_blank">https://mlr-org.github.io/mlr-<wbr>tutorial/release/html/img/<wbr>nested_resampling.png</a></p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><span class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</span></p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><span class="m_6097130876457162951gmail_msg">The code in
the above example however doesn't seem
to achieve this in any way.</span><br class="m_6097130876457162951gmail_msg">
</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">Am
I missing something here? </p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg"><br class="m_6097130876457162951gmail_msg">
</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">Thanks
a lot,</p>
<p style="margin-top:0px;margin-bottom:0px;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif;font-size:16px" class="m_6097130876457162951gmail_msg">dh</p>
</div>
</div>
</div>
<br class="m_6097130876457162951gmail_msg">
</div>
</div>
______________________________<wbr>_________________<br class="m_6097130876457162951gmail_msg">
scikit-learn mailing list<br class="m_6097130876457162951gmail_msg">
<a href="mailto:scikit-learn@python.org" class="m_6097130876457162951gmail_msg" target="_blank">scikit-learn@python.org</a><br class="m_6097130876457162951gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" class="m_6097130876457162951gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/scikit-learn</a><br class="m_6097130876457162951gmail_msg">
<br class="m_6097130876457162951gmail_msg">
</blockquote>
</div>
<br class="m_6097130876457162951gmail_msg">
</div>
</div>
</blockquote>
</div>
<br class="m_6097130876457162951gmail_msg">
</div>
<br class="m_6097130876457162951gmail_msg">
<fieldset class="m_6097130876457162951m_1181111257658593422mimeAttachmentHeader m_6097130876457162951gmail_msg"></fieldset>
<br class="m_6097130876457162951gmail_msg">
<pre class="m_6097130876457162951gmail_msg">______________________________<wbr>_________________
scikit-learn mailing list
<a class="m_6097130876457162951m_1181111257658593422moz-txt-link-abbreviated m_6097130876457162951gmail_msg" href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>
<a class="m_6097130876457162951m_1181111257658593422moz-txt-link-freetext m_6097130876457162951gmail_msg" href="https://mail.python.org/mailman/listinfo/scikit-learn" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/scikit-learn</a>
</pre>
</blockquote>
<br class="m_6097130876457162951gmail_msg">
</div>
______________________________<wbr>_________________<br class="m_6097130876457162951gmail_msg">
scikit-learn mailing list<br class="m_6097130876457162951gmail_msg">
<a href="mailto:scikit-learn@python.org" class="m_6097130876457162951gmail_msg" target="_blank">scikit-learn@python.org</a><br class="m_6097130876457162951gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" class="m_6097130876457162951gmail_msg" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/scikit-learn</a><br class="m_6097130876457162951gmail_msg">
</blockquote></div>
</div></div><br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>