Re: [scikit-learn] Plot Cross-validated ROCs for multi-class classification problem
Please stay on the mailing list. There is no single roc curve for a 3 class problem. So what do you want to plot? On 07/20/2018 10:40 AM, serafim loukas wrote:
Hello Andy,
Thank you for your response.
What I want to do is to plot the average(mean) ROC across Folds for a 3-class case. I have managed to do so for the binary case and I am trying to make it work for the mutlti-class case but with no luck.
There is an example in the documentation (http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossv...) for the binary case. I want to do the same ( plot the mean ROC and the confidence interval for my 3-class problem).
Here is also my SO question about this: https://stackoverflow.com/questions/51442818/average-roc-curve-across-folds-... with some code included.
Best, Makis
On 20 Jul 2018, at 16:34, Andreas Mueller <t3kcit@gmail.com <mailto:t3kcit@gmail.com>> wrote:
Hi Makis. What do you mean by a roc curve for multi-class? You can have one curve per class using OVR or one curve per pair of classes. That doesn't need the OneVsRestClassifier, it's more a matter of evaluation.
Cheers, Andy
On 07/20/2018 05:35 AM, serafim loukas wrote:
Dear Scikit-learn community,
I have a 3 class classification problem and I would like to plot the average ROC across Folds. There is an example in scikit-learn website but only for binary classification problems (http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossv...).
I want to do the same but in the case of 3 classes. I have tried to use `clf = OneVsRestClassifier(LinearDiscriminantAnalysis())`but I am having a hard time to make it work.
Any help would be appreciated, Makis
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Hello Makis, 2018-07-20 23:44 GMT+09:00 Andreas Mueller <t3kcit@gmail.com>:
There is no single roc curve for a 3 class problem. So what do you want to plot?
On 07/20/2018 10:40 AM, serafim loukas wrote:
What I want to do is to plot the average(mean) ROC across Folds for a 3-class case.
The prototypical ROC curve uses True Positive Rate and False Positive Rate for its axes, so it is for 2-class problems, and not for 3+-class problems, as Andy mentioned. Perhaps you are wanting the mean and confidence intervals of the n-class Cohen Kappa metric as estimated by either many folds of cross validation, or you want to evaluate your classifier by repeated subsampling experiments and Kappa value distribution/histogram? Hope this helps, J.B.
Hello J.B, I could simply create some ROC curves as shown in the scikit-learn documentation by selecting only 2 classes and then repeating by selecting other pair of classes (in total I have 3 classes so this would result in 3 different ROC figures). An alternative would be I would like to plot the mean and confidence intervals of the 3-class Cohen Kappa metric as estimated by KFolds (k=5) cross-validation. Any tips about this ? Cheers, Makis On 21 Jul 2018, at 16:02, Brown J.B. via scikit-learn <scikit-learn@python.org<mailto:scikit-learn@python.org>> wrote: Hello Makis, 2018-07-20 23:44 GMT+09:00 Andreas Mueller <t3kcit@gmail.com<mailto:t3kcit@gmail.com>>: There is no single roc curve for a 3 class problem. So what do you want to plot? On 07/20/2018 10:40 AM, serafim loukas wrote: What I want to do is to plot the average(mean) ROC across Folds for a 3-class case. The prototypical ROC curve uses True Positive Rate and False Positive Rate for its axes, so it is for 2-class problems, and not for 3+-class problems, as Andy mentioned. Perhaps you are wanting the mean and confidence intervals of the n-class Cohen Kappa metric as estimated by either many folds of cross validation, or you want to evaluate your classifier by repeated subsampling experiments and Kappa value distribution/histogram? Hope this helps, J.B. _______________________________________________ scikit-learn mailing list scikit-learn@python.org<mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn
participants (3)
-
Andreas Mueller -
Brown J.B. -
serafim loukas