Mailman 3 April 2020 - scikit-learn

[GridSearchCV] Reduction of elapsed time at the second interation
by Pedro Cardoso May 30, 2020

May 30, 2020

Hello fellows, i am knew at slkearn and I have a question about GridSearchCV: I am running the following code at a jupyter notebook : ----------------------*code*------------------------------- opt_models = dict() for feature in [features1, features2, features3, features4]: cmb = CMB(x_train, y_train, x_test, y_test, feature) cmb.fit() cmb.predict() opt_models[str(feature)]=cmb.get_best_model() ------------------------------------------------------- The CMB class is just a class that contains different classification models (SVC, decision tree, etc...). When cmb.fit() is running, a gridSearchCV is performed at the SVC model (which is within the cmb instance) in order to tune the hyperparameters C, gamma, and kernel. The SCV model is implemented using the sklearn.svm.SVC class. Here is the output of the first and second iteration of the for loop: ---------------------*output*------------------------------------- -> 1st iteration Fitting 5 folds for each of 12 candidates, totalling 60 fits [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers. [Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 6.1s [Parallel(n_jobs=-1)]: Done 2 tasks | elapsed: 6.1s [Parallel(n_jobs=-1)]: Done 3 tasks | elapsed: 6.1s [Parallel(n_jobs=-1)]: Done 4 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 5 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 6 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 7 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 8 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 9 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 10 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 11 tasks | elapsed: 6.2s [Parallel(n_jobs=-1)]: Done 12 tasks | elapsed: 6.3s [Parallel(n_jobs=-1)]: Done 13 tasks | elapsed: 6.3s [Parallel(n_jobs=-1)]: Done 14 tasks | elapsed: 6.3s [Parallel(n_jobs=-1)]: Done 15 tasks | elapsed: 6.4s [Parallel(n_jobs=-1)]: Done 16 tasks | elapsed: 6.4s [Parallel(n_jobs=-1)]: Done 17 tasks | elapsed: 6.4s [Parallel(n_jobs=-1)]: Done 18 tasks | elapsed: 6.4s [Parallel(n_jobs=-1)]: Done 19 tasks | elapsed: 6.5s [Parallel(n_jobs=-1)]: Done 20 tasks | elapsed: 6.5s [Parallel(n_jobs=-1)]: Done 21 tasks | elapsed: 6.5s [Parallel(n_jobs=-1)]: Done 22 tasks | elapsed: 6.6s [Parallel(n_jobs=-1)]: Done 23 tasks | elapsed: 6.7s [Parallel(n_jobs=-1)]: Done 24 tasks | elapsed: 6.7s [Parallel(n_jobs=-1)]: Done 25 tasks | elapsed: 6.7s [Parallel(n_jobs=-1)]: Done 26 tasks | elapsed: 6.8s [Parallel(n_jobs=-1)]: Done 27 tasks | elapsed: 6.8s [Parallel(n_jobs=-1)]: Done 28 tasks | elapsed: 6.9s [Parallel(n_jobs=-1)]: Done 29 tasks | elapsed: 6.9s [Parallel(n_jobs=-1)]: Done 30 tasks | elapsed: 6.9s [Parallel(n_jobs=-1)]: Done 31 tasks | elapsed: 7.0s [Parallel(n_jobs=-1)]: Done 32 tasks | elapsed: 7.0s [Parallel(n_jobs=-1)]: Done 33 tasks | elapsed: 7.0s [Parallel(n_jobs=-1)]: Done 34 tasks | elapsed: 7.0s [Parallel(n_jobs=-1)]: Done 35 tasks | elapsed: 7.1s [Parallel(n_jobs=-1)]: Done 36 tasks | elapsed: 7.1s [Parallel(n_jobs=-1)]: Done 37 tasks | elapsed: 7.2s [Parallel(n_jobs=-1)]: Done 38 tasks | elapsed: 7.2s [Parallel(n_jobs=-1)]: Done 39 tasks | elapsed: 7.2s [Parallel(n_jobs=-1)]: Done 40 tasks | elapsed: 7.2s [Parallel(n_jobs=-1)]: Done 41 tasks | elapsed: 7.3s [Parallel(n_jobs=-1)]: Done 42 tasks | elapsed: 7.3s [Parallel(n_jobs=-1)]: Done 43 tasks | elapsed: 7.3s [Parallel(n_jobs=-1)]: Done 44 tasks | elapsed: 7.4s [Parallel(n_jobs=-1)]: Done 45 tasks | elapsed: 7.4s [Parallel(n_jobs=-1)]: Done 46 tasks | elapsed: 7.5s -> 2nd iteration Fitting 5 folds for each of 12 candidates, totalling 60 fits [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers. [Parallel(n_jobs=-1)]: Done 1 tasks | elapsed: 0.0s [Parallel(n_jobs=-1)]: Batch computation too fast (0.0260s.) Setting batch_size=14. [Parallel(n_jobs=-1)]: Done 2 tasks | elapsed: 0.0s [Parallel(n_jobs=-1)]: Done 3 tasks | elapsed: 0.0s [Parallel(n_jobs=-1)]: Done 4 tasks | elapsed: 0.0s [Parallel(n_jobs=-1)]: Done 5 tasks | elapsed: 0.0s [Parallel(n_jobs=-1)]: Done 60 out of 60 | elapsed: 0.7s finished --------------------------------------------------------------------------------------------------------------------- As you can see, the first iteration gets a elapsed time much larger than the 2nd iteration. Does it make sense? I am afraid that the model is doing some kind of cache or shortcut from the 1st iteration, and consequently could decrease the model training/performance? I already read the sklearn documentation and I didn't saw any warning/note about this kind of behaviour. Thank you very much for your time :)

2 2

Class weight SVC
by Francesco basciani May 27, 2020

May 27, 2020

Hi, i have a question regarding the class weights in SVC. I have an imbalanced binary classification problem. In my case the ratio between the positive class and the negative class is 4:1. I just want to know if setting class weight to: class_weight = {1: 0.25, 0: 1} is the same to setting it to: class_weight = {1: 1, 0: 4}. Because in my case i obtain differents results using the two definition of the class weight Inviato da Posta<https://go.microsoft.com/fwlink/?LinkId=550986> per Windows 10

2 1

Why does sklearn require one-hot-encoding for categorical features? Can we have a "factor" data type?
by C W May 6, 2020

May 6, 2020

Hello everyone, I am frustrated with the one-hot-encoding requirement for categorical feature. Why? I've used R and Stata software, none needs such transformation. They have a data type called "factors", which is different from "numeric". My problem with OHE: One-hot-encoding results in large number of features. This really blows up quickly. And I have to fight curse of dimensionality with PCA reduction. That's not cool! Can sklearn have a "factor" data type in the future? It would make life so much easier. Thanks a lot!

7 8

StackingClassifier
by Andrew Howe May 5, 2020

May 5, 2020

Hi All Quick question about the stacking classifier <https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.Stacking…>. How do I know the order of the features that the final estimator uses? I've got an example which I've created like this (the LGRG and KSVM objects were previously defined, but as they seem they would be): passThrough = True finalEstim = DecisionTreeClassifier(random_state=42) stkClas = StackingClassifier(estimators=[('Logistic Regression', LGRG), ('Kernel SVM', KSVM)], cv=crossValInput, passthrough=passThrough, final_estimator=finalEstim, n_jobs=-1) Given this setup, I *think* the features input to the final estimator are - Logistic regression prediction probabilities for all classes - Kernel SVM prediction probabilities for all classes - original features of data passed into the stacking classifier I can find no documentation on this, though, and don't know of any relevant attribute on the final estimator. I need this to help interpret the final estimator tree - and specifically to provide feature labels for plot_tree. Thanks! Andrew <~~~~~~~~~~~~~~~~~~~~~~~~~~~> J. Andrew Howe, PhD LinkedIn Profile <http://www.linkedin.com/in/ahowe42> ResearchGate Profile <http://www.researchgate.net/profile/John_Howe12/> Open Researcher and Contributor ID (ORCID) <http://orcid.org/0000-0002-3553-1990> Github Profile <http://github.com/ahowe42> Personal Website <http://www.andrewhowe.com> I live to learn, so I can learn to live. - me <~~~~~~~~~~~~~~~~~~~~~~~~~~~>

2 3

Vote: Add Adrin Jalali to the scikit-learn technical committee
by Andreas Mueller May 2, 2020

May 2, 2020

Hi All. Given all his recent contributions, I want to nominate Adrin Jalali to the Technical Committee: https://scikit-learn.org/stable/governance.html#technical-committee According to the governance document, this will require a discussion and vote. I think we can move to the vote immediately unless someone objects. Thanks for all your work Adrin! Cheers, Andy

13 14

MLPClassifier/Regressor and Kernel Processes when Multiprocessing
by Taylor J Keding April 28, 2020

April 28, 2020

Hi SciKit-Learn folks, I am building a stacked generalization classifier using the multilayer perceptron classifier as one of it's submodels. All data have been preprocessed appropriately and I am tuning each submodel's hyperparameters with a customized randomized search protocol (very similar to sklearn's RandomizedSearchCV). Importantly, I am using Python's Multiprocessing.Pool() to parallelize this search. When I start the hyperparameter search, jobs/threads do indeed spawn appropriately. Tuning other submodels (RandomForestClassifier, SVC, GradientBoostingClassifier, SDGClassifier) works perfectly, which each job (model with particular randomized parameters) being scored with cross_val_score and returning when the Pool of workers is complete. All is well until I reach the MLPClassifier model. Jobs spawn as with the other models, however, System CPU (Linux Kernel) processes surge and overwhelm my server. Approximately 20% of the CPUs are running User processes, while the other 80% of CPUS are running System/Kernel processes, causing immense slow-down. Again, this only happens with the MLPClassifier - all other models run appropriately with ~98% User processes and ~2% System/Kernel processes. Is there something unique in the MLPClassifier/Regressor models that causes increased System/Kernel processes compared to other models? In an attempt to troubleshoot, I used sklearn's RandomizedSearchCV instead of my custom implementation and the same problems happen (with n_jobs specified in the same way). Any help with why the MLP models are behaving this way during multiprocessing is much appreciated. Best, Taylor Keding

2 1

precision_recall_curve giving incorrect results on very small example
by David R April 28, 2020

April 28, 2020

Here is a very small example using precision_recall_curve(): from sklearn.metrics import precision_recall_curve, precision_score, recall_score y_true = [0, 1] y_predict_proba = [0.25,0.75] precision, recall, thresholds = precision_recall_curve(y_true, y_predict_proba) precision, recall which results in: (array([1., 1.]), array([1., 0.])) Now let's calculate manually to see whether that's correct. There are three possible class vectors depending on threshold: [0,0], [0,1], and [1,1]. We have to discard [0,0] because it gives an undefined precision (divide by zero). So, applying precision_score() and recall_score() to the other two: y_predict_class=[0,1] precision_score(y_true, y_predict_class), recall_score(y_true, y_predict_class) which gives: (1.0, 1.0) and y_predict_class=[1,1] precision_score(y_true, y_predict_class), recall_score(y_true, y_predict_class) which gives (0.5, 1.0) This seems not to match the output of precision_recall_curve() (which for example did not produce a 0.5 precision value). Am I missing something?

1 0

Re: [scikit-learn] Vote: Add Adrin Jalali to the scikit-learn technical committee
by Hanmin Qin April 27, 2020

April 27, 2020

+1 Hanmin Qin ----- Original Message ----- From: Jeremie du Boisberranger <jeremie.du-boisberranger(a)inria.fr> To: scikit-learn(a)python.org Subject: Re: [scikit-learn] Vote: Add Adrin Jalali to the scikit-learn technical committee Date: 2020-04-27 21:23 +1 On 27/04/2020 15:18, Nicolas Hug wrote: > +1 > > On 4/27/20 9:16 AM, Gael Varoquaux wrote: >> +1 >> >> And thank you very much Adrin! >> >> On Mon, Apr 27, 2020 at 09:12:02AM -0400, Andreas Mueller wrote: >>> Hi All. >>> Given all his recent contributions, I want to nominate Adrin Jalali >>> to the >>> Technical Committee: >>> https://scikit-learn.org/stable/governance.html#technical-committee >>> According to the governance document, this will require a discussion >>> and >>> vote. >>> I think we can move to the vote immediately unless someone objects. >>> Thanks for all your work Adrin! >>> Cheers, >>> Andy >>> _______________________________________________ >>> scikit-learn mailing list >>> scikit-learn(a)python.org >>> https://mail.python.org/mailman/listinfo/scikit-learn > _______________________________________________ > scikit-learn mailing list > scikit-learn(a)python.org > https://mail.python.org/mailman/listinfo/scikit-learn _______________________________________________ scikit-learn mailing list scikit-learn(a)python.org https://mail.python.org/mailman/listinfo/scikit-learn

1 0

April 27th scikit-learn monthly meeting
by Chiara Marmo April 27, 2020

April 27, 2020

Hi all, The next scikit-learn monthly meeting will take place on Monday April 27th at the usual time: https://www.timeanddate.com/worldclock/meetingdetails.html?year=2020&month=… While these meetings are mainly for core-devs to discuss the current topics, we're also happy to welcome non-core devs and other projects maintainers! Feel free to join, using the following link: https://anaconda.zoom.us/j/94399382811?pwd=cXBtQ2lTVEtVbFpVTkE3TVFxdEhqZz09 Meeting ID: 943 9938 2811 Password: 68473658 If you plan to attend and you would like to discuss something specific about your contribution please add your name (or github pseudo) in the "Issue and comments from contributors <https://hackmd.io/5c6LxpnWSzeaBwJfuX5gPA?both#Issue-and-comments-from-contr…>", of the public pad: https://hackmd.io/5c6LxpnWSzeaBwJfuX5gPA *@core devs, please make sure to update your notes on Friday.* Best, Chiara

4 4

Number of informative features vs total number of features
by Benoît Presles April 3, 2020

April 3, 2020

Dear sklearn users, I did some supervised classification simulations with the make_classification function from sklearn increasing the number of informative features from 1 out of 40 to 40 out of 40 (100%). I did not generate any repeated or redundant features. I fixed the number of classes to two and the number of clusters per class to one. I split the dataset 100 times using the StratifiedShuffleSplit function into two subsets: a training set and a test set (80% - 20%). I performed a logistic regression and calculated training and testing accuracies and averaged the results over the 100 splits leading to a mean training accuracy and a mean testing accuracy. I was expecting to get an increasing accuracy score as a function of informative features for both the training and the test sets. On the contrary, I have got the best training and test scores for one informative feature. Why do I get these results ? Thanks for your help, Best regards, Ben Below the simulation code I have written: import numpy as np from sklearn.datasets import make_classification from sklearn.model_selection import StratifiedShuffleSplit from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score import matplotlib.pyplot as plt RANDOM_SEED = 4 n_inf = np.array([1, 5, 10, 15, 20, 25, 30, 35, 40]) mean_training_score_array = np.array([]) mean_testing_score_array = np.array([]) for n_inf_value in n_inf: X, y = make_classification(n_samples=2500, n_features=40, n_informative=n_inf_value, n_redundant=0, n_repeated=0, n_classes=2, n_clusters_per_class=1, random_state=RANDOM_SEED, shuffle=False) # print('Simulated data - number of informative features = ' + str(n_inf_value)) # sss = StratifiedShuffleSplit(n_splits=100, test_size=0.2, random_state=RANDOM_SEED) training_score_array = np.array([]) testing_score_array = np.array([]) for train_index_split, test_index_split in sss.split(X, y): X_split_train, X_split_test = X[train_index_split], X[test_index_split] y_split_train, y_split_test = y[train_index_split], y[test_index_split] scaler = StandardScaler() X_split_train = scaler.fit_transform(X_split_train) X_split_test = scaler.transform(X_split_test) lr = LogisticRegression(fit_intercept=True, max_iter=1e9, verbose=0, random_state=RANDOM_SEED, solver='lbfgs', tol=1e-6, C=10) lr.fit(X_split_train, y_split_train) y_pred_train = lr.predict(X_split_train) y_pred_test = lr.predict(X_split_test) accuracy_train_score = accuracy_score(y_split_train, y_pred_train) accuracy_test_score = accuracy_score(y_split_test, y_pred_test) training_score_array = np.append(training_score_array, accuracy_train_score) testing_score_array = np.append(testing_score_array, accuracy_test_score) mean_training_score_array = np.append(mean_training_score_array, np.average(training_score_array)) mean_testing_score_array = np.append(mean_testing_score_array, np.average(testing_score_array)) # print('mean_training_score_array=' + str(mean_training_score_array)) print('mean_testing_score_array=' + str(mean_testing_score_array)) # plt.plot(n_inf, mean_training_score_array, 'r', label='mean training score') plt.plot(n_inf, mean_testing_score_array, 'g', label='mean testing score') plt.xlabel('number of informative features out of 40') plt.ylabel('accuracy') plt.legend() plt.show()

2 2