<div dir="ltr"><div><div>Thanks Naoya ! This has worked and I am able to generate the .dot files.<br><br></div>Cheers,<br><br></div>Debu<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 29, 2016 at 10:20 AM, Naoya Kanai <span dir="ltr"><<a href="mailto:naopon@gmail.com" target="_blank">naopon@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="m_2097134325457042430markdown-here-wrapper"><p style="margin:0px 0px 1.2em!important">The ‘tree’ name is clashing between the sklearn.tree module and the DecisionTreeClassifier objects in the loop.</p>
<p style="margin:0px 0px 1.2em!important">You can change the import to</p>
<p style="margin:0px 0px 1.2em!important"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline">from sklearn.tree import export_graphviz</code></p><p style="margin:0px 0px 1.2em!important">and modify the method call accordingly.</p>
<div title="MDH:VGhlICd0cmVlJyBuYW1lIGlzIGNsYXNoaW5nIGJldHdlZW4gdGhlIHNrbGVhcm4udHJlZSBtb2R1
bGUgYW5kIHRoZSBEZWNpc2lvblRyZWVDbGFzc2lmaWVyIG9iamVjdHMgaW4gdGhlIGxvb3AuPGRp
dj48YnI+PC9kaXY+PGRpdj5Zb3UgY2FuIGNoYW5nZSB0aGUgaW1wb3J0IHRvPC9kaXY+PGRpdj48
YnI+PC9kaXY+PGRpdj5gZnJvbSBza2xlYXJuLnRyZWUgaW1wb3J0IGV4cG9ydF9ncmFwaHZpemA8
L2Rpdj4=" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 28, 2016 at 8:38 PM, Debabrata Ghosh <span dir="ltr"><<a href="mailto:mailfordebu@gmail.com" target="_blank">mailfordebu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Guillaume,</div><div> <wbr> Thanks for your feedback ! I am still getting an error, while attempting to print the trees. Here is a snapshot of my code. I know I may be missing something very silly, but still wanted to check and see how this works.</div><div><br></div><div><span>>>> clf = RandomForestClassifier(n_estim<wbr>ators=5000, n_jobs=-1)<br>>>> clf.fit(p_features_train,p_lab<wbr>els_train)<br></span>RandomForestClassifier(bootstr<wbr>ap=True, class_weight=None, criterion='gini',<br> max_depth=None, max_features='auto', max_leaf_nodes=None,<br> min_samples_leaf=1, min_samples_split=2,<br> min_weight_fraction_leaf=0.0, n_estimators=5000, n_jobs=-1,<br> oob_score=False, random_state=None, verbose=0,<br> warm_start=False)<span><br>>>> for idx_tree, tree in enumerate(clf.estimators_):<br></span>... export_graphviz(tree, out_file='{}.dot'.format(idx_t<wbr>ree))<br>...<br>Traceback (most recent call last):<br> File "<stdin>", line 2, in <module><br>NameError: name 'export_graphviz' is not defined<span><br>>>> for idx_tree, tree in enumerate(clf.estimators_):<br></span>... tree.export_graphviz(tree, out_file='{}.dot'.format(idx_t<wbr>ree))<br>...<br>Traceback (most recent call last):<br> File "<stdin>", line 2, in <module><br>AttributeError: 'DecisionTreeClassifier' object has no attribute 'export_graphviz'</div><div><br></div><div>Just to give you a background about the libraries, I have imported the following libraries:</div><div><br></div><div>from sklearn.ensemble import RandomForestClassifier<br></div><div>from sklearn import tree</div><div><br></div><div>Thanks again as always !</div><div><br></div><div>Cheers,</div></div><div class="m_2097134325457042430HOEnZb"><div class="m_2097134325457042430h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 29, 2016 at 1:04 AM, Guillaume Lemaître <span dir="ltr"><<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">after the fit you need this call:<br>for idx_tree, tree in enumerate(clf.estimators_):<br> export_graphviz(tree, out_file='{}.dot'.format(idx_t<wbr>ree))<div><div class="m_2097134325457042430m_-6511458374732369339h5"><br><br><div><div class="gmail_extra"><br><div class="gmail_quote">On 28 December 2016 at 20:25, Debabrata Ghosh <span dir="ltr"><<a href="mailto:mailfordebu@gmail.com" target="_blank">mailfordebu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div>Hi Guillaume,<br></div> With respect to the following point you mentioned:<span class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-"><br><div>You can visualize the trees with sklearn.tree.export_graphviz:<br><a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html" target="_blank">http://scikit-learn.org/stable<wbr>/modules/generated/sklearn.tre<wbr>e.export_graphviz.html</a><br><br></div></span><div>I couldn't find a direct method for exporting the RandomForestClassifier trees. Accordingly, I attempted for a workaround using the following code but still no success:<br><br>clf = RandomForestClassifier(n_estim<wbr>ators=5000, n_jobs=-1)<br>clf.fit(p_features_train,p_lab<wbr>els_train)<span class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852gmail-im"><br>for i, tree in enumerate(clf.estimators_):<br> with open('tree_' + str(i) + '.dot', 'w') as dotfile:<br></span> tree.export_graphviz(clf, dotfile)<br><br></div><div>Would you please be able to help me with the piece of code which I need to execute for exporting the RandomForestClassifier trees.<br><br></div><div>Cheers,<br><br></div><div>Debu<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-">On Tue, Dec 27, 2016 at 11:18 PM, Guillaume Lemaître <span dir="ltr"><<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>></span> wrote:<br></span><div><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><span>On 27 December 2016 at 18:17, Debabrata Ghosh <span dir="ltr"><<a href="mailto:mailfordebu@gmail.com" target="_blank">mailfordebu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Dear Joel, Andrew and Roman,<br></div> <wbr> Thank you very much for your individual feedback ! It's very helpful indeed ! A few more points related to my model execution:<br><br></div>1. By the term "scoring" I meant the process of executing the model once again without retraining it. So , for training the model I used RandomForestClassifer library and for my scoring (execution without retraining) I have used joblib.dump and joblib.load<br></div></div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>Go probably with the terms: training, validating, and testing.<br></div><div>This is pretty much standard. Scoring is just the value of a<br></div><div>metric given some data (training data, validation data, or<br></div><div>testing data).<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><div><div><div><div><br></div>2. I have used the parameter n_estimator = 5000 while training my model. Besides it , I have used n_jobs = -1 and haven't used any other parameter<br></div></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>You should probably check those other parameters and understand<br></div><div> what are their effects. You should really check the link of Roman<br></div><div>since GridSearchCV can help you to decide how to fix the parameters.<br><a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV" target="_blank">http://scikit-learn.org/stable<wbr>/modules/generated/sklearn.mod<wbr>el_selection.GridSearchCV.html<wbr>#sklearn.model_selection.GridS<wbr>earchCV</a><br></div><div>Additionally, 5000 trees seems a lot to me.<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><div><div><div><br></div>3. For my "scoring" activity (executing the model without retraining it) is there an alternate approach to joblib library ?<br></div></div></div></div></div></div></div></blockquote><div><br></div></span><div>Joblib only store data. There is not link with scoring (Check Roman answer)<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><div><div><br></div>4. When I execute my scoring job (joblib method) on a dataset , which is completely different to my training dataset then I get similar True Positive Rate and False Positive Rate as of training<br></div></div></div></div></div></div></blockquote><div><br></div></span><div>It is what you should get. <br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><div><br></div>5. However, when I execute my scoring job on the same dataset used for training my model then I get very high TPR and FPR.<br></div></div></div></div></div></blockquote><div><br></div></span><div>You are testing on some data which you used while training. Probably,<br></div><div>one of the first rule is to not do that. If you want to evaluate in some<br></div><div>way your classifier, have a separate set (test set) and only test on that<br></div><div>one. As previously mentioned by Roman, 80% of your data are already<br></div><div>known by the RandomForestClassifier and will be perfectly classified.<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><div><br></div> <wbr> Is there mechanism through which I can visualise the trees created by my RandomForestClassifer algorithm ? While I dumped the model using joblib.dump , there are a bunch of .npy files created. Will those contain the trees ?<br></div></div></div></div></blockquote></span><div><br>You can visualize the trees with sklearn.tree.export_graphviz:<br><a href="http://scikit-learn.org/stable/modules/generated/sklearn.tree.export_graphviz.html" target="_blank">http://scikit-learn.org/stable<wbr>/modules/generated/sklearn.tre<wbr>e.export_graphviz.html</a><br><div class="gmail_extra"><br></div><div class="gmail_extra">The bunch of npy are the data needed to load the RandomForestClassifier which<br></div>you previously dumped.<br> <br></div><div><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div><div><div><br></div>Thanks in advance !<br><br></div>Cheers,<br><br></div>Debu<br></div><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-HOEnZb"><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 27, 2016 at 4:22 PM, Joel Nothman <span dir="ltr"><<a href="mailto:joel.nothman@gmail.com" target="_blank">joel.nothman@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr">Your model is overfit to the training data. Not to say that it's necessarily possible to get a better fit. The default settings for trees lean towards a tight fit, so you might modify their parameters to increase regularisation. Still, you should not expect that evaluating a model's performance on its training data will be indicative of its general performance. This is why we use held-out test sets and cross-validation.</div><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-m_-6289671805834530366HOEnZb"><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-m_-6289671805834530366h5"><div class="gmail_extra"><br><div class="gmail_quote">On 27 December 2016 at 20:51, Roman Yurchak <span dir="ltr"><<a href="mailto:rth.yurchak@gmail.com" target="_blank">rth.yurchak@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">Hi Debu,<br>
<br>
On 27/12/16 08:18, Andrew Howe wrote:<br>
> 5. I got a prediction result with True Positive Rate (TPR) as 10-12<br>
<span>> % on probability thresholds above 0.5<br>
<br>
</span>Getting a high True Positive Rate (recall) is not a sufficient condition<br>
for a well behaved model. Though 0.1 recall is still pretty bad. You<br>
could look at the precision at the same time (or consider, for instance,<br>
the F1 score).<br>
<br>
> 7. I reloaded the model in a different python instance from the<br>
<span>> pickle file mentioned above and did my scoring , i.e., used<br>
> joblib library load method and then instantiated prediction<br>
> (predict_proba method) on the entire set of my original 600 K<br>
> records<br>
</span><span>> Another question – is there an alternate model scoring<br>
> library (apart from joblib, the one I am using) ?<br>
<br>
</span>Joblib is not a scoring library; once you load a model from disk with<br>
joblib you should get ~ the same RandomForestClassifier estimator object<br>
as before saving it.<br>
<br>
> 8. Now when I am running (scoring) my model using<br>
<span>> joblib.predict_proba on the entire set of original data (600 K),<br>
> I am getting a True Positive rate of around 80%.<br>
<br>
</span>That sounds normal, considering what you are doing. Your entire set<br>
consists of 80% of training set (for which the recall, I imagine, would<br>
be close to 1.0) and 20 % test set (with a recall of 0.1), so on<br>
average you would get a recall close to 0.8 for the complete set. Unless<br>
I missed something.<br>
<br>
<br>
> 9. I did some further analysis and figured out that during the<br>
<span>> training process, when the model was predicting on the test<br>
> sample of 120K it could only predict 10-12% of 120K data beyond<br>
> a probability threshold of 0.5. When I am now trying to score my<br>
> model on the entire set of 600 K records, it appears that the<br>
> model is remembering some of it’s past behavior and data and<br>
> accordingly throwing 80% True positive rate<br>
<br>
</span>It feels like your RandomForestClassifier is not properly tuned. A<br>
recall of 0.1 on the test set is quite low. It could be worth trying to<br>
tune it better (cf. <a href="https://stackoverflow.com/a/36109706" rel="noreferrer" target="_blank">https://stackoverflow.com/a/36<wbr>109706</a> ), using some<br>
other metric than the recall to evaluate the performance.<br>
<span class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-m_-6289671805834530366m_-1961140380961122981HOEnZb"><font color="#888888"><br>
<br>
Roman<br>
</font></span><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-m_-6289671805834530366m_-1961140380961122981HOEnZb"><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail-m_-6289671805834530366m_-1961140380961122981h5">______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div></div></div><span class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail-m_-2959403579090563852m_-7842543485078779819gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><div>Guillaume Lemaitre<br></div></div><div>INRIA Saclay - Ile-de-France<br>Equipe PARIETAL<br><a href="mailto:guillaume.lemaitre@inria.fr" target="_blank">guillaume.lemaitre@inria.f</a><a>r</a> --- <a href="https://glemaitre.github.io/" target="_blank">https://glemaitre.github.io/</a><br></div></div></div></div></div></div>
</font></span></div></div></div>
<br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div></div></div><br></div>
<br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="m_2097134325457042430m_-6511458374732369339m_-6226917441227560210gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div><div>Guillaume Lemaitre<br></div></div><div>INRIA Saclay - Ile-de-France<br>Equipe PARIETAL<br><a href="mailto:guillaume.lemaitre@inria.fr" target="_blank">guillaume.lemaitre@inria.f</a><a>r</a> --- <a href="https://glemaitre.github.io/" target="_blank">https://glemaitre.github.io/</a><br></div></div></div></div></div></div>
</div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/<wbr>mailman/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>