Mailman 3 October 2019 - scikit-learn

logistic regression results are not stable between solvers
by Benoît Presles Jan. 9, 2020

Jan. 9, 2020

Dear scikit-learn users, I am using logistic regression to make some predictions. On my own data, I do not get the same results between solvers. I managed to reproduce this issue on synthetic data (see the code below). All solvers seem to converge (n_iter_ < max_iter), so why do I get different results? If results between solvers are not stable, which one to choose? Best regards, Ben ------------------------------------------ Here is the code I used to generate synthetic data: from sklearn.datasets import make_classification from sklearn.model_selection import StratifiedShuffleSplit from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression # RANDOM_SEED = 2 # X_sim, y_sim = make_classification(n_samples=200, n_features=45, n_informative=10, n_redundant=0, n_repeated=0, n_classes=2, n_clusters_per_class=1, random_state=RANDOM_SEED, shuffle=False) # sss = StratifiedShuffleSplit(n_splits=10, test_size=0.2, random_state=RANDOM_SEED) for train_index_split, test_index_split in sss.split(X_sim, y_sim): X_split_train, X_split_test = X_sim[train_index_split], X_sim[test_index_split] y_split_train, y_split_test = y_sim[train_index_split], y_sim[test_index_split] ss = StandardScaler() X_split_train = ss.fit_transform(X_split_train) X_split_test = ss.transform(X_split_test) # classifier_lbfgs = LogisticRegression(fit_intercept=True, max_iter=20000000, verbose=1, random_state=RANDOM_SEED, C=1e9, solver='lbfgs') classifier_lbfgs.fit(X_split_train, y_split_train) print('classifier lbfgs iter:', classifier_lbfgs.n_iter_) classifier_saga = LogisticRegression(fit_intercept=True, max_iter=20000000, verbose=1, random_state=RANDOM_SEED, C=1e9, solver='saga') classifier_saga.fit(X_split_train, y_split_train) print('classifier saga iter:', classifier_saga.n_iter_) # y_pred_lbfgs = classifier_lbfgs.predict(X_split_test) y_pred_saga = classifier_saga.predict(X_split_test) # if (y_pred_lbfgs==y_pred_saga).all() == False: print('lbfgs does not give the same results as saga :-( !') exit()

5 17

Version 0.21! and plot_tree!
by Andrew Howe Dec. 5, 2019

Dec. 5, 2019

I want to say thank you to all the sklearn developers. The breadth and quality of this software is truly breathtaking. Specifically, I want to say thank you very very much for the plot_tree function! I have wasted a lot of effort in the past, on multiple OSes, getting everything to work so I could view the tree.export_graphviz results. Having this new function to plot the trees natively in matplotlib is extremely useful. Thanks again! Andrew <~~~~~~~~~~~~~~~~~~~~~~~~~~~> J. Andrew Howe, PhD LinkedIn Profile <http://www.linkedin.com/in/ahowe42> ResearchGate Profile <http://www.researchgate.net/profile/John_Howe12/> Open Researcher and Contributor ID (ORCID) <http://orcid.org/0000-0002-3553-1990> Github Profile <http://github.com/ahowe42> Personal Website <http://www.andrewhowe.com> I live to learn, so I can learn to live. - me <~~~~~~~~~~~~~~~~~~~~~~~~~~~>

2 4

CutEncoder - simple suggestion for sklearn.preprocessing
by Ayoub Abozer Nov. 1, 2019

Nov. 1, 2019

Hello. please take a loot to my kaggle notebook. i have a simple suggestion for new encoder for sklearn.preprocessing. https://www.kaggle.com/ayoubabozer/cutencoder/notebook?scriptVersionId=2283… Thanks, Ayoub

2 4

Fwd: CutEncoder - simple suggestion for sklearn.preprocessing
by Ayoub Abozer Oct. 31, 2019

Oct. 31, 2019

---------- Forwarded message --------- From: Ayoub Abozer <ayoub.abozer(a)gmail.com> Date: Fri, Nov 1, 2019 at 1:27 AM Subject: CutEncoder - simple suggestion for sklearn.preprocessing To: <scikit-learn(a)python.org> Hello. please take a loot to my kaggle notebook. i have a simple suggestion for new encoder for sklearn.preprocessing. https://www.kaggle.com/ayoubabozer/cutencoder/notebook?scriptVersionId=2283… Thanks, Ayoub

1 0

Is there possible to combine multiple patterns in one regression model?
by lampahome Oct. 31, 2019

Oct. 31, 2019

I have an idea to predict usage of every blocks of one disk, and I found pattern of blocks are related with time. Ex: block index 0~100 have high access times at 00:00, 12:00, and 18:00 and for 10 minutes. other block index 1000~1100 have high access times at 05:00, 14:00, and 20:00 and for 10 minutes. >From above examples, I assume that some blocks have similar pattern, and other blocks have similar pattern...etc But I have 100,000 blocks, and I can get features like access_times, blk_ID, timestamp(e.g. 00:00~23:59). Is that possible I feed them all into one regression model and predict well? thx

1 0

Can we say stochastic gradient descent as an ML model?
by Bulbul Ahmmed Oct. 28, 2019

Oct. 28, 2019

Dear Scikit Learn Community! Scikit learn puts stochastic gradient descent (SGD) as an ML model under the umbrella of linear model. I know SGD is an optimization algorithm. My question is: can we say SGD is an ML model? Thanks, Best Regards,Bulbul

3 3

Reminder: Monday October 28th meeting
by Adrin Oct. 26, 2019

Oct. 26, 2019

Hi Scikit-learn people, This is a reminder that we'll be having our monthly call on Monday. Please put your thoughts and important topics you have in mind on the project board: https://github.com/scikit-learn/scikit-learn/projects/15 We'll be meeting on https://appear.in/amueller As usual, it'd be nice to have them on the board before the weekend :) See you on Monday, Adrin.

2 1

Re: [scikit-learn] scikit-learn Digest, Vol 43, Issue 38
by WONG Wing Mei Oct. 25, 2019

Oct. 25, 2019

Can I ask whether we can use sample weight in gradient boosting? And how to do it? -----Original Message----- From: scikit-learn [mailto:scikit-learn-bounces+wong.wingmei=uobgroup.com@python.org] On Behalf Of scikit-learn-request(a)python.org Sent: Friday, October 25, 2019 12:00 AM To: scikit-learn(a)python.org Subject: scikit-learn Digest, Vol 43, Issue 38 Send scikit-learn mailing list submissions to scikit-learn(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/scikit-learn or, via email, send a message with subject or body 'help' to scikit-learn-request(a)python.org You can reach the person managing the list at scikit-learn-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of scikit-learn digest..." Today's Topics: 1. Re: Decision tree results sometimes different with scaled data (Alexandre Gramfort) 2. Reminder: Monday October 28th meeting (Adrin) ---------------------------------------------------------------------- Message: 1 Date: Thu, 24 Oct 2019 14:09:01 +0200 From: Alexandre Gramfort <alexandre.gramfort(a)inria.fr> To: Scikit-learn mailing list <scikit-learn(a)python.org> Subject: Re: [scikit-learn] Decision tree results sometimes different with scaled data Message-ID: <CADeotZrh_bXHAqV6WDNRoUt4ZXW_+eObj6_vmwMA50AnahkxgA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" another reason is that we take as threshold the mid point between sample values which is not invariant to arbitrary scaling of the features Alex On Tue, Oct 22, 2019 at 11:56 AM Guillaume Lema?tre <g.lemaitre58(a)gmail.com> wrote: > Even with the same random state, it can happen that several features will > lead to a best split and this split is chosen randomly (even with the seed > fixed - this is reported as an issue I think). Therefore, the rest of the > tree could be different leading to different prediction. > > Another possibility is that we compute the difference between the current > threshold and the next to be tried and only check the entropy if it is > larger than a specific value (I would need to check the source code). After > scaling, it could happen that 2 feature values become too closed to be > considered as a potential split which will make a difference between scaled > and scaled features. But this diff should be really small. > > This is the what I can think on the top of the head. > > Sent from my phone - sorry to be brief and potential misspell. > *From:* geoffrey.bolmier(a)gmail.com > *Sent:* 22 October 2019 11:34 > *To:* scikit-learn(a)python.org > *Reply to:* scikit-learn(a)python.org > *Subject:* [scikit-learn] Decision tree results sometimes different with > scaled data > > Hi all, > > First, let me thank you for the great job your guys are doing developing > and maintaining such a popular library! > > As we all know decision trees are not impacted by scaled data because > splits don't take into account distances between two values within a > feature. > > However I experienced a strange behavior using sklearn decision tree > algorithm. Sometimes results of the model are different depending if input > data has been scaled or not. > > To illustrate my point I ran experiments on the iris dataset consisting of: > > - perform a train/test split > - fit the training set and predict the test set > - fit and predict again with standardized inputs (removing the mean > and scaling to unit variance) > - compare both model predictions > > Experiments have been ran 10,000 times with different random seeds (cf. > traceback and code to reproduce it at the end). > Results showed that for a bit more than 10% of the time we find at least > one different prediction. Hopefully when it's the case only a few > predictions differ, 1 or 2 most of the time. I checked the inputs causing > different predictions and they are not the same depending of the run. > > I'm worried if the rate of different predictions could be larger for other > datasets... > Do you have an idea where it come from, maybe due to floating point errors > or am I doing something wrong? > > Cheers, > Geoffrey > > > ------------------------------------------------------------ > Traceback: > ------------------------------------------------------------ > Error rate: 12.22% > > Seed: 241862 > All pred equal: False > Not scale data confusion matrix: > [[16 0 0] > [ 0 17 0] > [ 0 4 13]] > Scale data confusion matrix: > [[16 0 0] > [ 0 15 2] > [ 0 4 13]] > ------------------------------------------------------------ > Code: > ------------------------------------------------------------ > import numpy as np > > from sklearn.datasets import load_iris > from sklearn.metrics import confusion_matrix > from sklearn.model_selection import train_test_split > from sklearn.preprocessing import StandardScaler > from sklearn.tree import DecisionTreeClassifier > > > X, y = load_iris(return_X_y=True) > > def run_experiment(X, y, seed): > X_train, X_test, y_train, y_test = train_test_split( > X, > y, > stratify=y, > test_size=0.33, > random_state=seed > ) > > scaler = StandardScaler() > > X_train_scaled = scaler.fit_transform(X_train) > X_test_scaled = scaler.transform(X_test) > > clf = DecisionTreeClassifier(random_state=seed) > clf_scaled = DecisionTreeClassifier(random_state=seed) > > clf.fit(X_train, y_train) > clf_scaled.fit(X_train_scaled, y_train) > > pred = clf.predict(X_test) > pred_scaled = clf_scaled.predict(X_test_scaled) > > err = 0 if all(pred == pred_scaled) else 1 > > return err, y_test, pred, pred_scaled > > > n_err, n_run, seed_err = 0, 10000, None > > for _ in range(n_run): > seed = np.random.randint(10000000) > err, _, _, _ = run_experiment(X, y, seed) > n_err += err > > # keep aside last seed causing an error > seed_err = seed if err == 1 else seed_err > > > print(f'Error rate: {round(n_err / n_run * 100, 2)}%', end='\n\n') > > _, y_test, pred, pred_scaled = run_experiment(X, y, seed_err) > > print(f'Seed: {seed_err}') > print(f'All pred equal: {all(pred == pred_scaled)}') > print(f'Not scale data confusion matrix:\n{confusion_matrix(y_test, > pred)}') > print(f'Scale data confusion matrix:\n{confusion_matrix(y_test, > pred_scaled)}') > [image: Sent from Mailspring] > _______________________________________________ > scikit-learn mailing list > scikit-learn(a)python.org > https://mail.python.org/mailman/listinfo/scikit-learn >

3 2

Decision tree results sometimes different with scaled data
by Geoffrey Bolmier Oct. 24, 2019

Oct. 24, 2019

Hi all, First, let me thank you for the great job your guys are doing developing and maintaining such a popular library! As we all know decision trees are not impacted by scaled data because splits don't take into account distances between two values within a feature. However I experienced a strange behavior using sklearn decision tree algorithm. Sometimes results of the model are different depending if input data has been scaled or not. To illustrate my point I ran experiments on the iris dataset consisting of: perform a train/test split fit the training set and predict the test set fit and predict again with standardized inputs (removing the mean and scaling to unit variance) compare both model predictions Experiments have been ran 10,000 times with different random seeds (cf. traceback and code to reproduce it at the end). Results showed that for a bit more than 10% of the time we find at least one different prediction. Hopefully when it's the case only a few predictions differ, 1 or 2 most of the time. I checked the inputs causing different predictions and they are not the same depending of the run. I'm worried if the rate of different predictions could be larger for other datasets... Do you have an idea where it come from, maybe due to floating point errors or am I doing something wrong? Cheers, Geoffrey ------------------------------------------------------------ Traceback: ------------------------------------------------------------ Error rate: 12.22% Seed: 241862 All pred equal: False Not scale data confusion matrix: [[16 0 0] [ 0 17 0] [ 0 4 13]] Scale data confusion matrix: [[16 0 0] [ 0 15 2] [ 0 4 13]] ------------------------------------------------------------ Code: ------------------------------------------------------------ import numpy as np from sklearn.datasets import load_iris from sklearn.metrics import confusion_matrix from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.tree import DecisionTreeClassifier X, y = load_iris(return_X_y=True) def run_experiment(X, y, seed): X_train, X_test, y_train, y_test = train_test_split( X, y, stratify=y, test_size=0.33, random_state=seed ) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) clf = DecisionTreeClassifier(random_state=seed) clf_scaled = DecisionTreeClassifier(random_state=seed) clf.fit(X_train, y_train) clf_scaled.fit(X_train_scaled, y_train) pred = clf.predict(X_test) pred_scaled = clf_scaled.predict(X_test_scaled) err = 0 if all(pred == pred_scaled) else 1 return err, y_test, pred, pred_scaled n_err, n_run, seed_err = 0, 10000, None for _ in range(n_run): seed = np.random.randint(10000000) err, _, _, _ = run_experiment(X, y, seed) n_err += err # keep aside last seed causing an error seed_err = seed if err == 1 else seed_err print(f'Error rate: {round(n_err / n_run * 100, 2)}%', end='\n\n') _, y_test, pred, pred_scaled = run_experiment(X, y, seed_err) print(f'Seed: {seed_err}') print(f'All pred equal: {all(pred == pred_scaled)}') print(f'Not scale data confusion matrix:\n{confusion_matrix(y_test, pred)}') print(f'Scale data confusion matrix:\n{confusion_matrix(y_test, pred_scaled)}')

3 2

New contribution
by Bartosz Podkanowicz Oct. 23, 2019

Oct. 23, 2019

Hi all, I am not sure if it is right place for asking this question. In the next 3 months I would like to contribute to scikit-learn (some code, bug fixs, tests). Unfortunately I have found that many issues labelled as "good first issue" and "help wanted" has people already working on it or some pull requests like #15076, #14781, #14934. I am not sure if starting contibuting with reviewing pull requests is good idea. Can someone guide me how to find issues/enhancements/fixes that I can start working on? or maybe I should start with reviewing some pull requests? I suppose that I missed something when reading contributing guide. Thank you for answers. Kind Regards, Bartosz Podkanowicz

2 1