Mailman 3 October 2016 - scikit-learn

Github project management tools
by Joel Nothman Dec. 3, 2016

Dec. 3, 2016

One of the biggest issues with scikit-learn as a project is managing its backlog of issues; another is release scheduling. Some of this cannot be fixed as long as our model of voluntary contribution (with a couple of important exceptions) does not change. However, it may be worth considering the new project management features in Github. At the moment we have the following management: * labels corresponding to type (bug, enhancement, new feat, question), scope (API, Build/CI, ?Large Scale, Documentation), difficulty (easy, moderate), status/scheduling (needs contributor, needs review, sprint). * PR status management with title prefixes [WIP], [MRG], [MRG+1], [MRG+2] Firstly, we might benefit from prefixing labels by category, i.e. difficulty:easy so that complementary labels appear together. In truth, PRs have roughly these statuses: * WIP (not ready for review) * waiting for review * waiting for changes (with or without one of the following) * in dispute (i.e. fundamental doubts about the PR) * the above together with 1 or 2 "official" approvals * ready for merge (pending minor changes such as what's new documentation) New github features: * reviews with "approved" or "request changes". A list of approvers can be found in the merge/CI panel. We could replace the MRG+1 annotation with this and use it to track disputation too. I'm not sure how it works with changes that are added after approval. I think it would have avoided one improper merge by me... One downside is that there does not yet seem to be a way to search for PRs with a specified level of approval (while searching for "MRG+1" sort-of works). * Milestone prioritising: issues in a milestone, such as https://github.com/scikit-learn/scikit-learn/milestone/21, can be ranked with drag-and-drop. I think this could help with release scheduling as it would allow us to identify the top priorities for a release and see when enough of them are completed. * The Kanban-style workflow management of the new Projects tool https://github.com/scikit-learn/scikit-learn/projects is another way of managing status and, I think, priority, for a small set of related issues. This might be an alternative way of managing milestone scope, or of working towards big changes like the one just completed for model selection; like proposed expansions to get_feature_names expansion; like estimator tags; making utilities public/private... So with the goal of making it easier to track where attention is most needed, and when to move to release: What's worth trying?

13 37

creating a custom scoring function for cross-validation of classification
by Sumeet Sandhu Nov. 1, 2016

Nov. 1, 2016

Hi, I've been staring at various doc pages for a while to create a custom scorer that uses predict_proba output of a multi-class SGDClassifier : http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cr… http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parame… http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_score… I got the impression I could customize the "scoring'' parameter in cross_val_score directly, but that didn't work. Then I tried customizing the "score_func" parameter in make_scorer, but that didn't work either. Both errors are ValuErrors : Traceback (most recent call last): File "<pyshell#96>", line 3, in <module> accuracy = mean(cross_val_score(LRclassifier, trainPatentVecs, trainLabelVecs, cv=10, scoring = 'topNscorer')) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/cross_validation.py", line 1425, in cross_val_score scorer = check_scoring(estimator, scoring=scoring) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/metrics/scorer.py", line 238, in check_scoring return get_scorer(scoring) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/metrics/scorer.py", line 197, in get_scorer % (scoring, sorted(SCORERS.keys()))) ValueError: 'topNscorer' is not a valid scoring value. Valid options are ['accuracy', 'adjusted_rand_score', 'average_precision', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'log_loss', 'mean_absolute_error', 'mean_squared_error', 'median_absolute_error', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc'] At a high level, I want to find out if the true label was found in the top N multi-class labels coming out of an SGD classifier. Built-in scores like "accuracy" only look at N=1. Here is the code using make_scorer : LRclassifier = SGDClassifier(loss='log') topNscorer = make_scorer(topNscoring, greater_is_better=True, needs_proba=True) accuracyN = mean(cross_val_score(LRclassifier, Data, Labels, scoring = 'topNscorer')) Here is the code for the custom scoring function : def topNscoring(y, yp): ## Inputs y = true label per sample, yp = predict_proba probabilities of all labels per sample N = 5 foundN = [] for ii in xrange(0,shape(yp)[0]): indN = [ w[0] for w in sorted(enumerate(list(yp[ii,:])),key=lambda w:w[1],reverse=True)[0:N] ] if y[ii] in indN: foundN.append(1) else: foundN.append(0) return mean(foundN) Any help will be greatly appreciated. best regards, Sumeet

2 2

Module Level Labels?
by Raghav R V Oct. 31, 2016

Oct. 31, 2016

Hi all, Should we have module level labels? "mod: tree" "mod: model_selection" "mod: linear_models" "mod: ..." I know it will blow up our label count, but I think it will help filter issues / PRs to review. Sometimes I like to look into issues / PRs that concern my two fav. modules "model_selection" and "tree" and I have to resort to some complex searches to cover all possible key words. (Sometimes the OP does not use the right keywords in their issue / PR) (From time to time I may keep popping some crazy suggestions, please feel free to shoot them down) Have a good weekend!! Raghav RV https://github.com/raghavrv

3 3

The implementation of `gradient_boost.py:BinomialDeviance`?
by 颜发才(Yan Facai) Oct. 31, 2016

Oct. 31, 2016

Hi, which paper or book is the foundation of the implementation of `gradient_boost.py:BinomialDeviance`? I recently read the paper: Friedman: greedy function approximation - a gradient boosting machine. I believe that L2_TreeBoost in the paper should be equivalent to BinomialDeviance in scikit-learn, while their implementation are different, for example: + negative_gradient: - in scikit: \tilde{y} = y - expit(pred.ravel()) = y - \frac{1}{1 + exp(- F)} - in paper: \tilde{y} = \frac{2 y}{1 + exp(2yF)} Does anyone can help me? Thanks.

1 1

Problem using boxplots to compare significance of model performance
by Suranga Kasthurirathne Oct. 30, 2016

Oct. 30, 2016

Hi Sebastian! Thank you, you might be onto something here ;) So, I may have to go over 2 models, so McNamara's may not be an option :( In regard to your second comment, in building my boxplots, this is how I input results. plt.boxplot(results) So what does "results" look like? [0.85433808345719897, 0.8976733724549345] These are the two precision values calculated for each neural network. Exactly what should 1Darray_of_model1_results look like? is it one value per model or.... -- Best Regards, Suranga

2 1

Problem using boxplots to compare significance of model performance
by Suranga Kasthurirathne Oct. 30, 2016

Oct. 30, 2016

Hi folks! I'm using scikit-learn to build two neural networks using 10% holdout, and compare their performance using precision. To compare statistical significance in the variance of precision, i'm using scikit's boxplots. My problem is twofold - 1) The standard deviation in the precision of the two models (obtained using precision.std()) is always 0.0. I'm assuming that's a problem. 2) My boxplot is meant to display bars for the two models, but always displays only the first model (nn01) My outcomes for this dataset is binary (0 or 1) since the models assume average=binary by default, is that a problem? For those who'd like to look, my source code can be seen at http://pastebin.com/yvE2T1Sw The code produces the following plot - which is of course only ONE of the bars that I need :( -- Best Regards, Suranga

2 1

Calculating prediction probability per each predicted outcome
by Suranga Kasthurirathne Oct. 28, 2016

Oct. 28, 2016

Hi everyone, I'm currently using Scikit learn to train and test multiple neural networks. My issue - I'm breaking my dataset into 90/10, training on the 90%, and testing on the 10%. For the 10% trained data, I get outcomes as follows: predicted = neural_network.predict(test_data) Here, the predicted variable is basically either 1 or 0, which is what i'm feeding in as the outcome. But how can I get the prediction probability per each predicted outcome? back in the day when I used weka it produced a single prediction, followed by a prediction probability between 1 and 0 for each outcome. -- Best Regards, Suranga

2 2

tree visualization with class names in leaves
by greg g Oct. 25, 2016

Oct. 25, 2016

Hi, I just begin with scikit-learn and would like to visualize a classification tree with class names displayed in the leaves as shown in the SCIKITLEARN.TREE documentation http://scikit-learn.org/stable/modules/tree.html where we find class=’virginica’ etc… I made a tree providing a 2D array X (n1 samples , n2 features) and 1D array Y (n1 corresponding classes ) such that Y(i) is the class of the sample X(i, …) After that I have correct predictions using predict() Then I use the function export_graphviz(clf, out_file=dot_data,feature_names=FEATURES) with FEATURES being the array of my n2 features names in the same order as in X I obtain the tree .png but can’t find a way to have the correct class names in the leaves… In export_graphviz() should I use the class_names optional parameter and how ? Thanks for any help Gregory, Toulouse FRANCE

5 6

NER Tagged Data
by Aakash Agarwal Oct. 24, 2016

Oct. 24, 2016

Hi All. I am trying to implement NER Algo using CRF data. Can anyone point me to some tagged data which i used. I am not able to use coNLL data (2003), i got the tagged data but the words are missing. I downloaded rcv1 dataset but still could not generate training and testing data. I would be grateful if anybody can help me. Thanks in advance! Aakash

1 0

Random Forest with Mean Absolute Error
by Robert Slater Oct. 23, 2016

Oct. 23, 2016

I searched the archives to see if this was a known issue, but could not seem to find anyone else having the problem. Nevertheless, in the latest version (0.18) Random Forest Regressor has the new option of 'mae' for criterion. However it appears to run disporportinally longer than the 'mse' critera, For example: from sklearn.ensemble import RandomForestRegressor rf_tree=50 rf_depth=5 rf=RandomForestRegressor(n_estimators=rf_tree, criterion='mae', max_depth=rf_depth, min_samples_split=4, min_samples_leaf=2, max_features=0.5, max_leaf_nodes=5, oob_score=True, n_jobs=1, random_state=0, verbose=1) from sklearn.ensemble import ExtraTreesRegressor et_tree=100 et=ExtraTreesRegressor(n_estimators=et_tree,max_depth=5,min_samples_split=4, min_samples_leaf=2,max_features=0.5,verbose=1,n_jobs=4) from sklearn.model_selection import train_test_split from sklearn.metrics import mean_absolute_error X_train, X_test, y_train, y_test = train_test_split(train, loss, test_size=0.2, random_state=42) et.fit(X_train,y_train) rf.fit(X_train,y_train) rf_pred=rf.predict(X_test) et_pred=et.predict(X_test) print(mean_absolute_error(y_test,rf_pred)) print(mean_absolute_error(y_test,et_pred)) I was using these two for a recent Kaggle competition. If I use "criterion='mse'" in the Random forest it takes around 1 min to build. Switching to 'mae' causes 100% CPU usage and 30 minutes (at least) if wait time before I kill my kernel. Not sure if the problem is on my end or if there is a real issue so I wanted to reach out and see if there or others.

2 1