Roc curve from multilabel classification has slope
Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set. Everything seems fine, except that some elements have a roc curve that have a slope as follows: I don't know how to interpret the slope in such cases. Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows. The predict method returns probabilities, as I'm using the functional api of keras. Does anyone knows why my roc curves looks like this? Ismael Sent from my iPhone
predict method should not return probabilities in scikit-learn classifiers. predict_proba should. On 8 January 2017 at 07:52, José Ismael Fernández Martínez < ismaelfm_@ciencias.unam.mx> wrote:
Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
[image: image1.PNG]
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
[image: enter image description here] <https://i.stack.imgur.com/XCNCA.png>I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
[image: image3.PNG]
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why my roc curves looks like this?
Ismael
Sent from my iPhone
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
But is not a scikit-learn classifier, is a keras classifier which, in the functional API, predict returns probabilities. What I don't understand is why my plot of the roc curve has a slope, since I call roc_curve passing the actual label as y_true and the output of the classifier (score probabilities) as y_score for every element tested. Sent from my iPhone
On Jan 7, 2017, at 4:04 PM, Joel Nothman <joel.nothman@gmail.com> wrote:
predict method should not return probabilities in scikit-learn classifiers. predict_proba should.
On 8 January 2017 at 07:52, José Ismael Fernández Martínez <ismaelfm_@ciencias.unam.mx> wrote: Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
<image1.PNG>
Everything seems fine, except that some elements have a roc curve that have a slope as follows: I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why my roc curves looks like this?
Ismael
Sent from my iPhone
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Slope usually means there are ties in your predictions. Check your dataset to see if you have repeated predicted values (possibly 1 or 0). On Sat, Jan 7, 2017 at 4:32 PM, José Ismael Fernández Martínez < ismaelfm_@ciencias.unam.mx> wrote:
But is not a scikit-learn classifier, is a keras classifier which, in the functional API, predict returns probabilities. What I don't understand is why my plot of the roc curve has a slope, since I call roc_curve passing the actual label as y_true and the output of the classifier (score probabilities) as y_score for every element tested.
Sent from my iPhone On Jan 7, 2017, at 4:04 PM, Joel Nothman <joel.nothman@gmail.com> wrote:
predict method should not return probabilities in scikit-learn classifiers. predict_proba should.
On 8 January 2017 at 07:52, José Ismael Fernández Martínez < ismaelfm_@ciencias.unam.mx> wrote:
Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
<image1.PNG>
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
[image: enter image description here] <https://i.stack.imgur.com/XCNCA.png>I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why my roc curves looks like this?
Ismael
Sent from my iPhone
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
José, I might be misunderstanding something, but wouldn't it make more sens to plot one ROC curve for every class in your result (using all samples at once), as opposed to plotting it for every training sample as you are doing now? Cf the example below, http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html Roman On 08/01/17 01:42, Jacob Schreiber wrote:
Slope usually means there are ties in your predictions. Check your dataset to see if you have repeated predicted values (possibly 1 or 0).
On Sat, Jan 7, 2017 at 4:32 PM, José Ismael Fernández Martínez <ismaelfm_@ciencias.unam.mx <mailto:ismaelfm_@ciencias.unam.mx>> wrote:
But is not a scikit-learn classifier, is a keras classifier which, in the functional API, predict returns probabilities. What I don't understand is why my plot of the roc curve has a slope, since I call roc_curve passing the actual label as y_true and the output of the classifier (score probabilities) as y_score for every element tested.
Sent from my iPhone On Jan 7, 2017, at 4:04 PM, Joel Nothman <joel.nothman@gmail.com <mailto:joel.nothman@gmail.com>> wrote:
predict method should not return probabilities in scikit-learn classifiers. predict_proba should.
On 8 January 2017 at 07:52, José Ismael Fernández Martínez <ismaelfm_@ciencias.unam.mx <mailto:ismaelfm_@ciencias.unam.mx>> wrote:
Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
<image1.PNG>
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
enter image description here <https://i.stack.imgur.com/XCNCA.png>I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained |model|, instance of Keras, and I have the features |X| and the binarized labels |y|, every element in |y| is an array of length 1000, as it is a multilabel classification problem each element in |y| might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of |binary_crossentropy| and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why my roc curves looks like this?
Ismael
Sent from my iPhone
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
That's indeed the case, there are ties in my predictions. In response to "plotting one ROC curve for every class in your result", it's also part of my analysis. Thank you very much. Ismael 2017-01-08 3:27 GMT-06:00 Roman Yurchak <rth.yurchak@gmail.com>:
José, I might be misunderstanding something, but wouldn't it make more sens to plot one ROC curve for every class in your result (using all samples at once), as opposed to plotting it for every training sample as you are doing now? Cf the example below,
http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
Roman
On 08/01/17 01:42, Jacob Schreiber wrote:
Slope usually means there are ties in your predictions. Check your dataset to see if you have repeated predicted values (possibly 1 or 0).
On Sat, Jan 7, 2017 at 4:32 PM, José Ismael Fernández Martínez <ismaelfm_@ciencias.unam.mx <mailto:ismaelfm_@ciencias.unam.mx>> wrote:
But is not a scikit-learn classifier, is a keras classifier which, in the functional API, predict returns probabilities. What I don't understand is why my plot of the roc curve has a slope, since I call roc_curve passing the actual label as y_true and the output of the classifier (score probabilities) as y_score for every element tested.
Sent from my iPhone On Jan 7, 2017, at 4:04 PM, Joel Nothman <joel.nothman@gmail.com <mailto:joel.nothman@gmail.com>> wrote:
predict method should not return probabilities in scikit-learn classifiers. predict_proba should.
On 8 January 2017 at 07:52, José Ismael Fernández Martínez <ismaelfm_@ciencias.unam.mx <mailto:ismaelfm_@ciencias.unam.mx>> wrote:
Hi, I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
<image1.PNG>
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
enter image description here <https://i.stack.imgur.com/XCNCA.png>I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained |model|, instance of Keras, and I have the features |X| and the binarized labels |y|, every element in |y| is an array of length 1000, as it is a multilabel classification problem each element in |y| might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of |binary_crossentropy| and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why my roc curves looks like this?
Ismael
Sent from my iPhone
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org <mailto:scikit-learn@python.org> https://mail.python.org/mailman/listinfo/scikit-learn <https://mail.python.org/mailman/listinfo/scikit-learn>
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
participants (4)
-
Jacob Schreiber -
Joel Nothman -
José Ismael Fernández Martínez -
Roman Yurchak