<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Am 03.04.19 um 23:46 schrieb Joel
Nothman:<br>
</div>
<blockquote type="cite"
cite="mid:CAAkaFLUYWHrpkg5OYr-ZejnG93Wzo=rasy3-RDxVGFdFkZ4S8g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div class="moz-text-html" lang="x-unicode">
<div dir="auto">Pull requests improving the documentation are
always welcome. At a minimum, users need to know that these
compute different things.
<div dir="auto"><br>
</div>
<div dir="auto">Accuracy is not precision. Precision is the
number of true positives divided by the number of true
positives plus false positives. It therefore cannot be
decomposed as a sample-wise measure without knowing the rate
of positive predictions. This rate is dependent on the
training data and algorithm. <br>
</div>
</div>
</div>
</blockquote>
<p>In my last post, I referred to your remark that "for precision
... you can't say the same". Since precision can't be computed
with formula (*), even with a different loss function, I pointed
out that (*) can be used to compute the accuracy if the loss
function is an indicator function. <br>
</p>
<p>It is still not clear to me what your point is with your remark
that "for precision ... you can't say the same". I assume that you
want to tell that it is not wise to compute TP, FP, FN and then
precision and recall using cross_val_predict. If this is what you
mean, I'd like you to explain why.</p>
<blockquote type="cite"
cite="mid:CAAkaFLUYWHrpkg5OYr-ZejnG93Wzo=rasy3-RDxVGFdFkZ4S8g@mail.gmail.com">
<div class="moz-text-html" lang="x-unicode">
<div dir="auto">
<div dir="auto">
<div dir="auto">I'm not a statistician and cannot speak to
issues of computing a mean of means, but if what we are
trying to estimate is the performance on a sample of size
approximately n_t of a model trained on a sample of size
approximately N - n_t, then I wouldn't have thought taking
a mean over such measures (with whatever score function)
to be unreasonable.</div>
</div>
</div>
<br>
</div>
</blockquote>
<p>In general, a mean of means is not the mean of the original data.
The pooled mean is the correct metric in this case. However, the
pooled mean equals the mean of means if all folds are exactly the
same size.<br>
</p>
<blockquote type="cite"
cite="mid:CAAkaFLUYWHrpkg5OYr-ZejnG93Wzo=rasy3-RDxVGFdFkZ4S8g@mail.gmail.com">
<div class="moz-text-html" lang="x-unicode">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu., 4 Apr. 2019, 3:51
am Boris Hollas, <<a
href="mailto:hollas@informatik.htw-dresden.de"
target="_blank" rel="noreferrer" moz-do-not-send="true">hollas@informatik.htw-dresden.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div
class="m_-5860468016156986646m_-9147584446002501114moz-cite-prefix">Am
03.04.19 um 13:59 schrieb Joel Nothman:<br>
</div>
<blockquote type="cite">
<div
class="m_-5860468016156986646m_-9147584446002501114moz-text-plain"
style="font-family:-moz-fixed;font-size:12px"
lang="x-western">
<pre class="m_-5860468016156986646m_-9147584446002501114moz-quote-pre">The equations in Murphy and Hastie very clearly assume a metric
decomposable over samples (a loss function). Several popular metrics
are not.
For a metric like MSE it will be almost identical assuming the test
sets have almost the same size. </pre>
</div>
</blockquote>
What will be almost identical to what? I suppose you mean
that (*) is consistent with the scores of the models in
the fold (ie, the result of cross_val_score) if the loss
function is (x-y)².<br>
<blockquote type="cite">
<div
class="m_-5860468016156986646m_-9147584446002501114moz-text-plain"
style="font-family:-moz-fixed;font-size:12px"
lang="x-western">
<pre class="m_-5860468016156986646m_-9147584446002501114moz-quote-pre">For something like Recall
(sensitivity) it will be almost identical assuming similar test set
sizes <b class="m_-5860468016156986646m_-9147584446002501114moz-txt-star"><span class="m_-5860468016156986646m_-9147584446002501114moz-txt-tag">*</span>and<span class="m_-5860468016156986646m_-9147584446002501114moz-txt-tag">*</span></b> stratification. For something like precision whose
denominator is determined by the biases of the learnt classifier on
the test dataset, you can't say the same.</pre>
</div>
</blockquote>
I can't follow here. If the loss function is L(x,y) = 1_{x
= y}, then (*) gives the accuracy.<br>
<blockquote type="cite">
<div
class="m_-5860468016156986646m_-9147584446002501114moz-text-plain"
style="font-family:-moz-fixed;font-size:12px"
lang="x-western">
<pre class="m_-5860468016156986646m_-9147584446002501114moz-quote-pre"> For something like ROC AUC
score, relying on some decision function that may not be equivalently
calibrated across splits, evaluating in this way is almost
meaningless.</pre>
</div>
</blockquote>
<p>In any case, I still don't see what may be wrong with
(*). Otherwise, the warning in the documentation about
the use of cross_val_predict should be removed or
revised.</p>
<p>On the other hand, an example in the documentation uses
cross_val_scores.mean(). This is debatable since this
computes a mean of means.<br>
</p>
<blockquote type="cite">
<div
class="m_-5860468016156986646m_-9147584446002501114moz-text-plain"
style="font-family:-moz-fixed;font-size:12px"
lang="x-western">
<pre class="m_-5860468016156986646m_-9147584446002501114moz-quote-pre">On Wed, 3 Apr 2019 at 22:01, Boris Hollas
<a class="m_-5860468016156986646m_-9147584446002501114moz-txt-link-rfc2396E" href="mailto:hollas@informatik.htw-dresden.de" rel="noreferrer noreferrer" target="_blank" moz-do-not-send="true"><hollas@informatik.htw-dresden.de></a> wrote:
</pre>
<blockquote type="cite" style="color:#000000">
<pre class="m_-5860468016156986646m_-9147584446002501114moz-quote-pre">I use
sum((cross_val_predict(model, X, y) - y)**2) / len(y) (*)
to evaluate the performance of a model. This conforms with Murphy: Machine Learning, section 6.5.3, and Hastie et al: The Elements of Statistical Learning, eq. 7.48. However, according to the documentation of cross_val_predict, "it is not appropriate to pass these predictions into an evaluation metric". While it is obvious that cross_val_predict is different from cross_val_score, I don't see what should be wrong with (*).
Also, the explanation that "cross_val_predict simply returns the labels (or probabilities)" is unclear, if not wrong. As I understand it, this function returns estimates and no labels or probabilities.
Regards, Boris
</pre>
</blockquote>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" rel="noreferrer
noreferrer" target="_blank" moz-do-not-send="true">scikit-learn@python.org</a><br>
<a
href="https://mail.python.org/mailman/listinfo/scikit-learn"
rel="noreferrer noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
</blockquote>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<div class="moz-text-plain" wrap="true" graphical-quote="true"
style="font-family: -moz-fixed; font-size: 12px;"
lang="x-western">
<pre class="moz-quote-pre" wrap="">_______________________________________________
scikit-learn mailing list
<a class="moz-txt-link-abbreviated" href="mailto:scikit-learn@python.org" moz-do-not-send="true">scikit-learn@python.org</a>
<a class="moz-txt-link-freetext" href="https://mail.python.org/mailman/listinfo/scikit-learn" moz-do-not-send="true">https://mail.python.org/mailman/listinfo/scikit-learn</a>
</pre>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>