Mailman 3 March 2017 - scikit-learn

Differences between scikit-learn and Spark.ml for regression toy problem
by Frank Astier March 15, 2017

March 15, 2017

(this was also posted to stackoverflow on 03/10) I am setting up a very simple logistic regression problem in scikit-learn and in spark.ml, and the results diverge: the models they learn are different, but I can't figure out why (data is the same, model type is the same, regularization is the same...). No doubt I am missing some setting on one side or the other. Which setting? How should I set up either scikit or spark.ml to find the same model as its counterpart? I give the sklearn code and spark.ml code below. Both should be ready to cut-and-paste and run. scikit-learn code: ---------------------- import numpy as np from sklearn.linear_model import LogisticRegression, Ridge X = np.array([ [-0.7306653538519616, 0.0], [0.6750417712898752, -0.4232874171873786], [0.1863463229359709, -0.8163423997075965], [-0.6719842051493347, 0.0], [0.9699938346531928, 0.0], [0.22759406190283604, 0.0], [0.9688721028330911, 0.0], [0.5993795346650845, 0.0], [0.9219423508390701, -0.8972778242305388], [0.7006904841584055, -0.5607635619919824] ]) y = np.array([ 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0 ]) m, n = X.shape # Add intercept term to simulate inputs to GameEstimator X_with_intercept = np.hstack((X, np.ones(m)[:,np.newaxis])) l = 0.3 e = LogisticRegression( fit_intercept=False, penalty='l2', C=1/l, max_iter=100, tol=1e-11) e.fit(X_with_intercept, y) print e.coef_ # => [[ 0.98662189 0.45571052 -0.23467255]] # Linear regression is called Ridge in sklearn e = Ridge( fit_intercept=False, alpha=l, max_iter=100, tol=1e-11) e.fit(X_with_intercept, y) print e.coef_ # =>[ 0.32155545 0.17904355 0.41222418] spark.ml code: ------------------- import org.apache.spark.{SparkConf, SparkContext} import org.apache.spark.ml.classification.LogisticRegression import org.apache.spark.ml.regression.LinearRegression import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.mllib.regression.LabeledPoint import org.apache.spark.sql.SQLContext object TestSparkRegression { def main(args: Array[String]): Unit = { import org.apache.log4j.{Level, Logger} Logger.getLogger("org").setLevel(Level.OFF) Logger.getLogger("akka").setLevel(Level.OFF) val conf = new SparkConf().setAppName("test").setMaster("local") val sc = new SparkContext(conf) val sparkTrainingData = new SQLContext(sc) .createDataFrame(Seq( LabeledPoint(0.0, Vectors.dense(-0.7306653538519616, 0.0)), LabeledPoint(1.0, Vectors.dense(0.6750417712898752, -0.4232874171873786)), LabeledPoint(1.0, Vectors.dense(0.1863463229359709, -0.8163423997075965)), LabeledPoint(0.0, Vectors.dense(-0.6719842051493347, 0.0)), LabeledPoint(1.0, Vectors.dense(0.9699938346531928, 0.0)), LabeledPoint(1.0, Vectors.dense(0.22759406190283604, 0.0)), LabeledPoint(1.0, Vectors.dense(0.9688721028330911, 0.0)), LabeledPoint(0.0, Vectors.dense(0.5993795346650845, 0.0)), LabeledPoint(0.0, Vectors.dense(0.9219423508390701, -0.8972778242305388)), LabeledPoint(0.0, Vectors.dense(0.7006904841584055, -0.5607635619919824)))) .toDF("label", "features") val logisticModel = new LogisticRegression() .setRegParam(0.3) .setLabelCol("label") .setFeaturesCol("features") .fit(sparkTrainingData) println(s"Spark logistic model coefficients: ${logisticModel.coefficients} Intercept: ${logisticModel.intercept}") // Spark logistic model coefficients: [0.5451588538376263,0.26740606573584713] Intercept: -0.13897955358689987 val linearModel = new LinearRegression() .setRegParam(0.3) .setLabelCol("label") .setFeaturesCol("features") .setSolver("l-bfgs") .fit(sparkTrainingData) println(s"Spark linear model coefficients: ${linearModel.coefficients} Intercept: ${linearModel.intercept}") // Spark linear model coefficients: [0.19852664861346023,0.11501200541407802] Intercept: 0.45464906876832323 sc.stop() } } Thanks, Frank

4 3

Label encoding for classifiers and soft targets
by Javier López Peña March 15, 2017

March 15, 2017

Hi there! I have been recently experimenting with model regularization through the use of soft targets, and I’d like to be able to play with that from sklearn. The main idea is as follows: imagine I want to fit a (probabilisitic) classifier with three possible targets, 0, 1, 2 If I pass my training set (X, y) to a sklearn classifier, the target vector y gets encoded so that each target becomes an array, [1, 0, 0], [0, 1, 0], or [0, 0, 1] What I would like to do is to be able to pass the targets directly in the encoded form, and avoid any further encoding. This allows for instance to pass targets as [0.9, 0.5, 0.5] if I want to prevent my classifier from becoming too opinionated on its predicted probabilities. Ideally I would like to do something like this: ``` clf = SomeClassifier(*parameters, encode_targets=False) ``` and then call ``` elf.fit(X, encoded_y) ``` Would it be simple to modify sklearn code to do this, or would it require a lot of tinkering such as modifying every single classifier under the sun? Cheers, J

4 9

Logistic regression with elastic net regularization
by Stuart Reynolds March 14, 2017

March 14, 2017

Is there an implementation of logistic regression with elastic net regularization in scikit? (or pointers on implementing this - its seems non-convex and so you might expect poor behavior with some optimizers) - Stuart

5 7

Error while using GridSearchCV.
by Shubham Singh Tomar March 12, 2017

March 12, 2017

Hi, I'm trying to use GridSearchCV to tune the parameters for DecisionTreeRegressor. I'm using sklearn 0.18.1 I'm getting the following error: ---------------------------------------------------------------------------TypeError Traceback (most recent call last)<ipython-input-36-192f7c286a58> in <module>() 1 # Fit the training data to the model using grid search----> 2 reg = fit_model(X_train, y_train) 3 4 # Produce the value for 'max_depth' 5 print "Parameter 'max_depth' is {} for the optimal model.".format(reg.get_params()['max_depth']) <ipython-input-35-500141c331d9> in fit_model(X, y) 11 12 # Create cross-validation sets from the training data---> 13 cv_sets = ShuffleSplit(X.shape[0], n_splits = 10, test_size = 0.20, random_state = 0) 14 15 # TODO: Create a decision tree regressor object TypeError: __init__() got multiple values for keyword argument 'n_splits' -- *Thanks,* *Shubham Singh Tomar* *Autodidact24.github.io <http://Autodidact24.github.io>*

3 3

Re: [scikit-learn] scikit-learn Digest, Vol 12, Issue 18
by Gautam Borad March 12, 2017

March 12, 2017

On 11 Mar 2017 22:32, <scikit-learn-request(a)python.org> wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Label encoding for classifiers and soft targets > (Javier L?pez Pe?a) > 2. issue suggestion - decision trees - GSoC (Konstantinos Katrioplas) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 11 Mar 2017 13:04:57 +0000 > From: Javier L?pez Pe?a <jlopez(a)ende.cc> > To: scikit-learn(a)python.org > Subject: [scikit-learn] Label encoding for classifiers and soft > targets > Message-ID: <542B0BDD-F329-4F26-9001-9F535426306C(a)ende.cc> > Content-Type: text/plain; charset=utf-8 > > Hi there! > > I have been recently experimenting with model regularization through the > use of soft targets, > and I?d like to be able to play with that from sklearn. > > The main idea is as follows: imagine I want to fit a (probabilisitic) > classifier with three possible > targets, 0, 1, 2 > > If I pass my training set (X, y) to a sklearn classifier, the target > vector y gets encoded so that > each target becomes an array, [1, 0, 0], [0, 1, 0], or [0, 0, 1] > > What I would like to do is to be able to pass the targets directly in the > encoded form, and avoid > any further encoding. This allows for instance to pass targets as [0.9, > 0.5, 0.5] if I want to prevent > my classifier from becoming too opinionated on its predicted probabilities. > > Ideally I would like to do something like this: > ``` > clf = SomeClassifier(*parameters, encode_targets=False) > ``` > > and then call > ``` > elf.fit(X, encoded_y) > ``` > > Would it be simple to modify sklearn code to do this, or would it require > a lot of tinkering > such as modifying every single classifier under the sun? > > Cheers, > J > > ------------------------------ > > Message: 2 > Date: Sat, 11 Mar 2017 15:29:30 +0200 > From: Konstantinos Katrioplas <konst.katrioplas(a)gmail.com> > To: scikit-learn(a)python.org > Subject: [scikit-learn] issue suggestion - decision trees - GSoC > Message-ID: <33a3a5bf-37dd-1cad-c4ae-ef4b62294a8c(a)gmail.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hello all, > > While I am waiting for the PR that I have submitted to be evaluated > (https://github.com/scikit-learn/scikit-learn/pull/8563), would you > suggest another (easy) issue for me to work on? Ideally something for > which I will write some substantial code, so as to present it in my > application for GSoC? > > Is anyone interested to mentor me in the parallelization of decision > trees? I admit I am not yet really familiar with the current tree code > (although I have been using the method for regression on a research > project) but I am very much intrigued by the idea and willing to learn > all about it until the summer. > > Regards, > Konstantinos > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > scikit-learn mailing list > scikit-learn(a)python.org > https://mail.python.org/mailman/listinfo/scikit-learn > > > ------------------------------ > > End of scikit-learn Digest, Vol 12, Issue 18 > ******************************************** >

1 0

issue suggestion - decision trees - GSoC
by Konstantinos Katrioplas March 11, 2017

March 11, 2017

Hello all, While I am waiting for the PR that I have submitted to be evaluated (https://github.com/scikit-learn/scikit-learn/pull/8563), would you suggest another (easy) issue for me to work on? Ideally something for which I will write some substantial code, so as to present it in my application for GSoC? Is anyone interested to mentor me in the parallelization of decision trees? I admit I am not yet really familiar with the current tree code (although I have been using the method for regression on a research project) but I am very much intrigued by the idea and willing to learn all about it until the summer. Regards, Konstantinos

1 0

MAPE in scikit-learn?
by Raghav R V March 8, 2017

March 8, 2017

Hi all, Do we want Median Absolute Percentage Error in scikit-learn? Ref: KDD2017 - https://tianchi.shuju.aliyun.com/competition/information.htm?spm=5176.10006… Thanks -- Raghav RV https://github.com/raghavrv

5 4

Linear Discriminant Analysis with Cross Validation in Python
by Serafeim Loukas March 8, 2017

March 8, 2017

Dear scikit members, I would like to ask if there is any function that implements Linear Discriminant Analysis with Cross Validation (leave one out). Thank you in advance, S

3 5

contribution to scikit-learn - questions
by Konstantinos Katrioplas March 6, 2017

March 6, 2017

Hello all, My name is Konstantinos and I would like to contribute to scikit-learn. I am relatively new to open source development and I want to work on some easy bug-fixing to get used to the github workflow. Firstly, is this issue open and should I try working on it? https://github.com/scikit-learn/scikit-learn/issues/8425 If not, would you suggest another? Furthermore, when trying to build with make I get this: make: nosetests: Command not found Makefile:32: recipe for target 'test-code' failed make: *** [test-code] Error 127 Is this in any way expected and do you know what I might be missing? Finally, is there an IRC channel particularly for scikit-learn? Thanks in advance, Konstantinos

2 1

Scikit-learn survey results
by Andreas Mueller March 6, 2017

March 6, 2017

Hey all. In case you're interested, here is a summary view of the scikit-learn survey I posted recently: https://www.surveymonkey.com/results/SM-RHGZVZ73/ tldr; Preprocessing takes the most time, people want out-of-core learning, better integration with pandas and easier visualization of models and data. People would use automatic machine learning if it was there, but it's not the highest priority item. There is also a lot of interesting info in the comments, but because I was not able to go through all of them yet, I don't want to publish them publicly in case there is sensitive information included (and if anyone knows if there are legal implications if there wasn't a disclaimer, please let me know). Cheers, Andy

3 3