[scikit-learn] Why is cross_val_predict discouraged?

[Boris Hollas]

Am 03.04.19 um 23:46 schrieb Joel Nothman:
> Pull requests improving the documentation are always welcome. At a 
> minimum, users need to know that these compute different things.
>
> Accuracy is not precision. Precision is the number of true positives 
> divided by the number of true positives plus false positives. It 
> therefore cannot be decomposed as a sample-wise measure without 
> knowing the rate of positive predictions. This rate is dependent on 
> the training data and algorithm.

In my last post, I referred to your remark that "for precision ... you 
can't say the same". Since precision can't be computed with formula (*), 
even with a different loss function, I pointed out that (*) can be used 
to compute the accuracy if the loss function is an indicator function.

It is still not clear to me what your point is with your remark that 
"for precision ... you can't say the same". I assume that you want to 
tell that it is not wise to compute TP, FP, FN and then precision and 
recall using cross_val_predict. If this is what you mean, I'd like you 
to explain why.

> I'm not a statistician and cannot speak to issues of computing a mean 
> of means, but if what we are trying to estimate is the performance on 
> a sample of size approximately n_t of a model trained on a sample of 
> size approximately N - n_t, then I wouldn't have thought taking a mean 
> over such measures (with whatever score function) to be unreasonable.
>
In general, a mean of means is not the mean of the original data. The 
pooled mean is the correct metric in this case. However, the pooled mean 
equals the mean of means if all folds are exactly the same size.

> On Thu., 4 Apr. 2019, 3:51 am Boris Hollas, 
> <hollas at informatik.htw-dresden.de 
> <mailto:hollas at informatik.htw-dresden.de>> wrote:
>
>     Am 03.04.19 um 13:59 schrieb Joel Nothman:
>>     The equations in Murphy and Hastie very clearly assume a metric
>>     decomposable over samples (a loss function). Several popular metrics
>>     are not.
>>
>>     For a metric like MSE it will be almost identical assuming the test
>>     sets have almost the same size.
>     What will be almost identical to what? I suppose you mean that (*)
>     is consistent with the scores of the models in the fold (ie, the
>     result of cross_val_score) if the loss function is (x-y)².
>>     For something like Recall
>>     (sensitivity) it will be almost identical assuming similar test set
>>     sizes**and**  stratification. For something like precision whose
>>     denominator is determined by the biases of the learnt classifier on
>>     the test dataset, you can't say the same.
>     I can't follow here. If the loss function is L(x,y) = 1_{x = y},
>     then (*) gives the accuracy.
>>       For something like ROC AUC
>>     score, relying on some decision function that may not be equivalently
>>     calibrated across splits, evaluating in this way is almost
>>     meaningless.
>
>     In any case, I still don't see what may be wrong with (*).
>     Otherwise, the warning in the documentation about the use of
>     cross_val_predict should be removed or revised.
>
>     On the other hand, an example in the documentation uses
>     cross_val_scores.mean(). This is debatable since this computes a
>     mean of means.
>
>>     On Wed, 3 Apr 2019 at 22:01, Boris Hollas
>>     <hollas at informatik.htw-dresden.de>  <mailto:hollas at informatik.htw-dresden.de>  wrote:
>>>     I use
>>>
>>>     sum((cross_val_predict(model, X, y) - y)**2) / len(y)        (*)
>>>
>>>     to evaluate the performance of a model. This conforms with Murphy: Machine Learning, section 6.5.3, and Hastie et al: The Elements of Statistical Learning,  eq. 7.48. However, according to the documentation of cross_val_predict, "it is not appropriate to pass these predictions into an evaluation metric". While it is obvious that cross_val_predict is different from cross_val_score, I don't see what should be wrong with (*).
>>>
>>>     Also, the explanation that "cross_val_predict simply returns the labels (or probabilities)" is unclear, if not wrong. As I understand it, this function returns estimates and no labels or probabilities.
>>>
>>>     Regards, Boris
>
>     _______________________________________________
>     scikit-learn mailing list
>     scikit-learn at python.org <mailto:scikit-learn at python.org>
>     https://mail.python.org/mailman/listinfo/scikit-learn
>
>
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20190404/9b092b55/attachment-0001.html>