Mailman 3 November 2016 - scikit-learn

Announcement: Scikit-learn 0.18.1 released!
by Andreas Mueller Nov. 15, 2016

Nov. 15, 2016

Hey all. I just published the 0.18.1 wheels and source tarball to pypi. The 0.18.1 release is a bugfix release, resolving some issues introduced in 0.18 and also some earlier issues. In particular there were some important relating to the new model_selection module. You can find the whole changelog (which I just realized does not contain all the fixes) here: http://scikit-learn.org/stable/whats_new.html#version-0-18-1 Best, Andy

4 4

Automatic ThresholdClassifier based on cost-function - Classifier Interface?
by Anton Suchaneck Nov. 14, 2016

Nov. 14, 2016

Hi! I tried writing a ThresholdClassifier, that wraps any classifier with predict_proba() and based on a cost function adjusts the threshold for predict(). This helps for imbalanced data. My current cost function assigns cost +cost for a true positive and -1 for a false positive. It seems to run, but I'm not sure if I got the API for a classifier right. Can you tell me whether this is how the functions should be implemented to play together with other parts of sklearn? Especially parameter settings for base.clone both in klass.__init__ and .set_params() seemed weird. Here is the code. The class ThresholdClassifier wraps a clf. RandomForest in this case. Anton from sklearn.base import BaseEstimator, ClassifierMixin from functools import partial def find_threshold_cost_factor(clf, X, y, cost_factor): y_pred = clf.predict_proba(X) top_score = 0 top_threshold = None cur_score=0 for y_pred_el, y_el in sorted(zip(y_pred[:, 1], y), reverse=True): # FIXME: assumes 2 classes if y_el == 0: cur_score -= 1 if y_el == 1: cur_score += cost_factor if cur_score > top_score: top_score = cur_score top_threshold = y_pred_el return top_threshold, top_score class ThresholdClassifier(BaseEstimator, ClassifierMixin): def __init__(self, clf, find_threshold, **params): self.clf = clf self.find_threshold = find_threshold self.threshold = None self.set_params(**params) def score(self, X, y, sample_weight=None): _threshold, score = self.find_threshold(self.clf, X, y) return score def fit(self, X, y): self.clf.fit(X, y) self.threshold, _score=self.find_threshold(self.clf, X, y) self.classes_ = self.clf.classes_ def predict(self, X): y_score=self.clf.predict_proba(X) return np.array(y_score[:,1]>=self.threshold) # FIXME assumes 2 classes def predict_proba(self, X): return self.clf.predict_proba(X) def set_params(self, **params): for param_name in ["clf", "find_threshold", "threshold"]: if param_name in params: setattr(self, param_name, params[param_name]) del params[param_name] self.clf.set_params(**params) return self def get_params(self, deep=True): params={"clf":self.clf, "find_threshold": self.find_threshold, "threshold":self.threshold} params.update(self.clf.get_params(deep)) return params if __name__ == '__main__': import numpy as np import random from sklearn.grid_search import RandomizedSearchCV from sklearn.ensemble import RandomForestClassifier from sklearn.datasets import make_classification from sklearn.cross_validation import train_test_split from sklearn.metrics import make_scorer, classification_report, confusion_matrix np.random.seed(111) random.seed(111) X, y = make_classification(1000, n_features=20, n_informative=4, n_redundant=0, n_repeated=0, n_clusters_per_class=4, # class_sep=0.5, weights=[0.90] ) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y) for cost in [10]: find_threshold=partial(find_threshold_cost_factor, cost_factor=10) def scorer(clf, X, y): return find_threshold(clf, X, y)[1] clfs = [RandomizedSearchCV( ThresholdClassifier(RandomForestClassifier(), find_threshold), {"n_estimators": [100, 200], "criterion": ["entropy"], "min_samples_leaf": [1, 5], "class_weight": ["balanced", None], }, cv=3, scoring=scorer, # Get rid of this, by letting classifier tell it's cost-bsed score? n_iter=8, n_jobs=4), ] for clf in clfs: clf.fit(X_train, y_train) clf_best = clf.best_estimator_ print(clf_best, cost, clf_best.score(X_test, y_test)) print(confusion_matrix(y_test, clf_best.predict(X_test))) #print(find_threshold(clf_best, X_train, y_train)) #print(clf_best.threshold, sorted(zip(clf_best.predict_proba(X_train)[:,1], y_train), reverse=True)[:20])

4 5

GPR intervals and MCMC
by Quaglino Alessio Nov. 8, 2016

Nov. 8, 2016

Hello, I am using scikit-learn 0.18 for doing GP regressions. I really like it and all works great, but I am having doubts concerning the confidence intervals computed by predict(X,return_std=True): - Are they true confidence intervals (i.e. of the mean / latent function) or they are in fact prediction intervals? I tried computing the prediction intervals using sample_y(X) and I get the same answer as that returned by predict(X,return_std=True). - My understanding is therefore that scikit-learn is not fully Bayesian, i.e. it does not compute probability distributions for the parameters, but rather the values that maximize the likelihood? - If I want the confidence interval, is my best option to use an external MCMC optimizer such as PyMC? Thank you in advance! Regards, ------------------------------------------------- Dr. Alessio Quaglino Postdoctoral Researcher Institute of Computational Science Università della Svizzera Italiana

3 2

Re: [scikit-learn] Recurrent Decision Tree
by KevNo Nov. 7, 2016

Nov. 7, 2016

This is nothing to do with Scikit guidelines criteria.... This is about scientific/mathematic view Recurrent Decision Tree which is a specific tree by nature (you cannot apply standard algos on this). Suppose very little number of people has experience with recurrence in Decision Tree... scikit-learn-request(a)python.org wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: Recurrent Decision Tree (Raghav R V) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 7 Nov 2016 15:51:11 +0100 > From: Raghav R V<ragvrv(a)gmail.com> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] Recurrent Decision Tree > Message-ID: > <CACmxyDGsizwT3dW3h=nKuWSUQ1uJseLdraMPSP=4WrMvtHDhaA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > The reference paper seems pretty new with very few citations. Check our FAQ > on inclusion criterion - > http://scikit-learn.org/stable/faq.html#what-are-the-inclusion-criteria-for… > > > On Mon, Nov 7, 2016 at 2:10 PM, Dale T Smith<Dale.T.Smith(a)macys.com> wrote: > >> Searching the mailing list would be the best way to find out this >> information. >> >> >> >> It may be in the contrib packages on github ? have you checked? >> >> >> >> >> >> ____________________________________________________________ >> ____________________________________________________________ >> __________________ >> *Dale T. Smith* *|* Macy's Systems and Technology *|* IFS eCom CSE Data >> Science >> 5985 State Bridge Road, Johns Creek, GA 30097 *|* dale.t.smith(a)macys.com >> >> >> >> *From:* scikit-learn [mailto:scikit-learn-bounces+dale.t.smith= >> macys.com(a)python.org] *On Behalf Of *KevNo >> *Sent:* Friday, November 4, 2016 4:44 PM >> *To:* scikit-learn(a)python.org >> *Subject:* [scikit-learn] Recurrent Decision Tree >> >> >> >> ? EXT MSG: >> >> Just wondering if Recurrent Decision Tree has been investigated >> by Scikit previously. >> >> Main interest is in path dependant (time series data) problems, >> the recurrence is often necessary to model the path dependent state. >> In other words, wrong prediction will affect the subsequent predictions. >> >> Here, a research paper on Recurrent Decision Tree, >> from Walt Disney Research (!) >> >> https://goo.gl/APGpvM >> >> >> Any thought is welcome. >> Thanks >> Brookm >> >> >> >> >> >> scikit-learn-request(a)python.org wrote: >> >> Send scikit-learn mailing list submissions to >> >> scikit-learn(a)python.org >> >> >> >> To subscribe or unsubscribe via the World Wide Web, visit >> >> https://mail.python.org/mailman/listinfo/scikit-learn >> >> or, via email, send a message with subject or body 'help' to >> >> scikit-learn-request(a)python.org >> >> >> >> You can reach the person managing the list at >> >> scikit-learn-owner(a)python.org >> >> >> >> When replying, please edit your Subject line so it is more specific >> >> than "Re: Contents of scikit-learn digest..." >> >> >> >> >> >> Today's Topics: >> >> >> >> 1. Re: hierarchical clustering (Gael Varoquaux) >> >> 2. Naive Bayes - Multinomial Naive Bayes tf-idf (Marcin Miro?czuk) >> >> 3. Re: hierarchical clustering (Jaime Lopez Carvajal) >> >> 4. Re: Naive Bayes - Multinomial Naive Bayes tf-idf (Andy) >> >> >> >> >> >> ---------------------------------------------------------------------- >> >> >> >> Message: 1 >> >> Date: Fri, 4 Nov 2016 10:36:49 +0100 >> >> From: Gael Varoquaux<gael.varoquaux(a)normalesup.org> <gael.varoquaux(a)normalesup.org> >> >> To: Scikit-learn user and developer mailing list >> >> <scikit-learn(a)python.org> <scikit-learn(a)python.org> >> >> Subject: Re: [scikit-learn] hierarchical clustering >> >> Message-ID:<20161104093649.GA137008(a)phare.normalesup.org> <20161104093649.GA137008(a)phare.normalesup.org> >> >> Content-Type: text/plain; charset=us-ascii >> >> >> >> AgglomerativeClustering internally calls scikit learn's version of >> >> cut_tree. I would be curious to know whether this is equivalent to >> >> scipy's fcluster. >> >> >> >> It differs in that it enable to add connectivity contraints. >> >> >> >> >> >> ------------------------------ >> >> >> >> Message: 2 >> >> Date: Fri, 4 Nov 2016 11:45:39 +0100 >> >> From: Marcin Miro?czuk<m.marcinmichal(a)gmail.com> <m.marcinmichal(a)gmail.com> >> >> To: scikit-learn(a)python.org >> >> Subject: [scikit-learn] Naive Bayes - Multinomial Naive Bayes tf-idf >> >> Message-ID: >> >> <CAH6=PuCebYLz32-YqpEUtRrYQvn7EQUiymWCy38Vi9_9Jr+-Fg(a)mail.gmail.com> <CAH6=PuCebYLz32-YqpEUtRrYQvn7EQUiymWCy38Vi9_9Jr+-Fg(a)mail.gmail.com> >> >> Content-Type: text/plain; charset="utf-8" >> >> >> >> Hi, >> >> In our experiments, we use a Multinomial Naive Bayes (MNB). The traditional >> >> MNB implies the TF weight of the words. We read in documentation >> >> http://scikit-learn.org/stable/modules/naive_bayes.html which describes >> >> Multinomial Naive Bayes that "... where the data are typically represented >> >> as word vector counts, although tf-idf vectors are also known to work well >> >> in practice". The "word vector counts" is a TF and it is well known. We >> >> have a problem which the "tf-idf vectors". In this case, i.e. tf-idf it >> >> was implemented the approach of the D. M. Rennie et all Tackling the Poor >> >> Assumptions of Naive Bayes Text Classification? In the documentation, there >> >> are not any citation of this solution. >> >> >> >> Best, >> >> >> >> * This is an EXTERNAL EMAIL. Stop and think before clicking a link or >> opening attachments. >> >> _______________________________________________ >> scikit-learn mailing list >> scikit-learn(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-learn >> >> > >

2 1

Recurrent Decision Tree
by KevNo Nov. 7, 2016

Nov. 7, 2016

Just wondering if Recurrent Decision Tree has been investigated by Scikit previously. Main interest is in path dependant (time series data) problems, the recurrence is often necessary to model the path dependent state. In other words, wrong prediction will affect the subsequent predictions. Here, a research paper on Recurrent Decision Tree, from Walt Disney Research (!) https://goo.gl/APGpvM Any thought is welcome. Thanks Brookm scikit-learn-request(a)python.org wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: hierarchical clustering (Gael Varoquaux) > 2. Naive Bayes - Multinomial Naive Bayes tf-idf (Marcin Miro?czuk) > 3. Re: hierarchical clustering (Jaime Lopez Carvajal) > 4. Re: Naive Bayes - Multinomial Naive Bayes tf-idf (Andy) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 4 Nov 2016 10:36:49 +0100 > From: Gael Varoquaux<gael.varoquaux(a)normalesup.org> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] hierarchical clustering > Message-ID:<20161104093649.GA137008(a)phare.normalesup.org> > Content-Type: text/plain; charset=us-ascii > >> AgglomerativeClustering internally calls scikit learn's version of >> cut_tree. I would be curious to know whether this is equivalent to >> scipy's fcluster. > > It differs in that it enable to add connectivity contraints. > > > ------------------------------ > > Message: 2 > Date: Fri, 4 Nov 2016 11:45:39 +0100 > From: Marcin Miro?czuk<m.marcinmichal(a)gmail.com> > To: scikit-learn(a)python.org > Subject: [scikit-learn] Naive Bayes - Multinomial Naive Bayes tf-idf > Message-ID: > <CAH6=PuCebYLz32-YqpEUtRrYQvn7EQUiymWCy38Vi9_9Jr+-Fg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > In our experiments, we use a Multinomial Naive Bayes (MNB). The traditional > MNB implies the TF weight of the words. We read in documentation > http://scikit-learn.org/stable/modules/naive_bayes.html which describes > Multinomial Naive Bayes that "... where the data are typically represented > as word vector counts, although tf-idf vectors are also known to work well > in practice". The "word vector counts" is a TF and it is well known. We > have a problem which the "tf-idf vectors". In this case, i.e. tf-idf it > was implemented the approach of the D. M. Rennie et all Tackling the Poor > Assumptions of Naive Bayes Text Classification? In the documentation, there > are not any citation of this solution. > > Best, >

3 2

Naive Bayes - Multinomial Naive Bayes tf-idf
by Marcin Mirończuk Nov. 4, 2016

Nov. 4, 2016

Hi, In our experiments, we use a Multinomial Naive Bayes (MNB). The traditional MNB implies the TF weight of the words. We read in documentation http://scikit-learn.org/stable/modules/naive_bayes.html which describes Multinomial Naive Bayes that "... where the data are typically represented as word vector counts, although tf-idf vectors are also known to work well in practice". The "word vector counts" is a TF and it is well known. We have a problem which the "tf-idf vectors". In this case, i.e. tf-idf it was implemented the approach of the D. M. Rennie et all Tackling the Poor Assumptions of Naive Bayes Text Classification? In the documentation, there are not any citation of this solution. Best, -- Marcin M.

2 1

hierarchical clustering
by Jaime Lopez Carvajal Nov. 4, 2016

Nov. 4, 2016

Hi there, I am trying to do image classification using hierarchical clustering. So, I have my data, and apply this steps: from scipy.cluster.hierarchy import dendrogram, linkage data1 = np.array(data) Z = linkage(data, 'ward') dendrogram(Z, truncate_mode='lastp', p=12, show_leaf_counts=False, leaf_rotation=90., leaf_font_size=12.,show_contracted=True) plt.show() So, I can see the dendrogram with 12 clusters as I want, but I dont know how to use this to classify the image. Also, I understand that funtion cluster.hierarchy.cut_tree(Z, n_clusters), that cut the tree at that number of clusters, but again I dont know how to procedd from there. I would like to have something like: cluster = predict(Z, instance) Any advice or direction would be really appreciate, Thanks in advance, Jaime -- *Jaime Lopez Carvajal*

4 5

Fwd: libmf bindings
by Andy Nov. 2, 2016

Nov. 2, 2016

-------- Forwarded Message -------- Subject: libmf bindings Date: Wed, 2 Nov 2016 11:38:00 -0400 From: sam royston <sfoxroyston(a)gmail.com> To: scikit-learn-owner(a)python.org Hi, Thanks for all your hard work on this useful tool! I'm hoping to contribute bindings to Chih-Jen Lin's libmf: https://www.csie.ntu.edu.tw/~cjlin/libmf/ <https://www.csie.ntu.edu.tw/%7Ecjlin/libmf/>. It looks like you guys have functionality for NMF, but used only in the decomposition/ dimensionality reduction setting (and obviously only with non-negative values). Id like to add functionality in the form python wrappers for libmf, much like you have for Chih-Jen Lin's other libraries libsvm and liblinear. Libmf is very efficient and offers great functionality for missing data imputation, recommendation systems and more. I have already written bindings using ctypes, but I see that you have you Cython for libsvm and liblinear - is it necessary that I switch to that interface? Let me know what you think of a contribution like this. Thanks, Sam

3 2

creating a custom scoring function for cross-validation of classification
by Sumeet Sandhu Nov. 1, 2016

Nov. 1, 2016

Hi, I've been staring at various doc pages for a while to create a custom scorer that uses predict_proba output of a multi-class SGDClassifier : http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cr… http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parame… http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_score… I got the impression I could customize the "scoring'' parameter in cross_val_score directly, but that didn't work. Then I tried customizing the "score_func" parameter in make_scorer, but that didn't work either. Both errors are ValuErrors : Traceback (most recent call last): File "<pyshell#96>", line 3, in <module> accuracy = mean(cross_val_score(LRclassifier, trainPatentVecs, trainLabelVecs, cv=10, scoring = 'topNscorer')) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/cross_validation.py", line 1425, in cross_val_score scorer = check_scoring(estimator, scoring=scoring) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/metrics/scorer.py", line 238, in check_scoring return get_scorer(scoring) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/metrics/scorer.py", line 197, in get_scorer % (scoring, sorted(SCORERS.keys()))) ValueError: 'topNscorer' is not a valid scoring value. Valid options are ['accuracy', 'adjusted_rand_score', 'average_precision', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'log_loss', 'mean_absolute_error', 'mean_squared_error', 'median_absolute_error', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc'] At a high level, I want to find out if the true label was found in the top N multi-class labels coming out of an SGD classifier. Built-in scores like "accuracy" only look at N=1. Here is the code using make_scorer : LRclassifier = SGDClassifier(loss='log') topNscorer = make_scorer(topNscoring, greater_is_better=True, needs_proba=True) accuracyN = mean(cross_val_score(LRclassifier, Data, Labels, scoring = 'topNscorer')) Here is the code for the custom scoring function : def topNscoring(y, yp): ## Inputs y = true label per sample, yp = predict_proba probabilities of all labels per sample N = 5 foundN = [] for ii in xrange(0,shape(yp)[0]): indN = [ w[0] for w in sorted(enumerate(list(yp[ii,:])),key=lambda w:w[1],reverse=True)[0:N] ] if y[ii] in indN: foundN.append(1) else: foundN.append(0) return mean(foundN) Any help will be greatly appreciated. best regards, Sumeet

2 2

SVM number of support vectors
by muhammad waseem Nov. 1, 2016

Nov. 1, 2016

Hello All, I am trying to replicate the below figure and wanted to confirm that number of support vectors can be calculated by *support_vectors_* attribute in scikitlearn? [image: Inline image 1] Regards Waseem

1 0