Mailman 3 December 2017 - scikit-learn

Any plans on generalizing Pipeline and transformers?
by Manuel Castejón Limas Jan. 8, 2018

Jan. 8, 2018

Dear all, Kudos to scikit-learn! Having said that, Pipeline is killing me not being able to transform anything other than X. My current study case would need: - Transformers being able to handle both X and y, e.g. clustering X and y concatenated - Pipeline being able to change other params, e.g. sample_weight Currently, I'm augmenting X through every step with the extra information which seems to work ok for my_pipe.fit_transform(X_train,y_train) but breaks on my_pipe.transform(X_test) for the lack of the y parameter. Ok, I can inherit and modify a descendant from Pipeline class to allow the y parameter which is not ideal but I guess it is an option. The gritty part comes when having to adapt every regressor at the end of the ladder in order to split the extra information from the raw data in X and not being able to generate more than one subproduct from each preprocessing step My current research involves clustering the data and using that classification along with X in order to predict outliers which generates sample_weight info and I would love to use that on the final regressor. Currently there seems not to be another option than pasting that info on X. All in all, I'm stuck with this API limitation and I would love to learn some tricks from you if you could enlighten me. Thanks in advance! Manuel Castejón-Limas

4 6

Re: [scikit-learn] Any plans on generalizing Pipeline and transformers?
by Manuel Castejón Limas Jan. 4, 2018

Jan. 4, 2018

Thank you all for your interest! In order to clarify the case allow me to try to synthesize the spirit of what I'd like to put into the pipeline using this sequence of steps: #%% import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.cluster import DBSCAN from sklearn.mixture import GaussianMixture from sklearn.model_selection import train_test_split np.random.seed(seed=42) """ Data preparation """ URL = "https://raw.githubusercontent.com/mcasl/PAELLA/master/data/sin_60_ percent_noise.csv" data = pd.read_csv(URL, usecols=['V1','V2']) X, y = data[['V1']], data[['V2']] (data_train, data_test, X_train, X_test, y_train, y_test) = train_test_split(data, X, y) """ Parameters setup """ dbscan__eps = 0.06 mclust__n_components = 3 paella__noise_label = -1 paella__max_it = 20, paella__regular_size = 400, paella__minimum_size = 100, paella__width_r = 0.99, paella__n_neighbors = 5, paella__power = 30, paella__random_state = None #%% """ DBSCAN clustering to detect noise suspects (label == -1) """ dbscan_input = data_train dbscan_clustering = DBSCAN(eps = dbscan__eps) dbscan_output = dbscan_clustering.fit_predict(dbscan_input) plt.scatter(data_train['V1'], data_train['V2'], cmap='cool', alpha=0.1, c=np.int64(dbscan_output == -1)) #%% """ GaussianMixture fitted with filtered data_train in order to help locate the ellipsoids but predict is applied to the whole data_train set. """ mclust_input = data_train[ dbscan_output != 1] mclust_clustering = GaussianMixture(n_components = mclust__n_components) mclust_clustering.fit(mclust_input) mclust_output = mclust_clustering.predict(data_train) plt.scatter(data_train['V1'], data_train['V2'], cmap='cool', alpha=0.1, c=mclust_output) #%% """ mclust and dbscan results are combined. """ clustering_output = mclust_output.copy() clustering_output[dbscan_output == -1] = -1 plt.scatter(data_train['V1'], data_train['V2'], cmap='cool', alpha=0.1, c=clustering_output) #%% """ Old-good Paella paper: https://link.springer.com/article/10.1023/B:DAMI. 0000031630.50685.7c The Paella algorithm calculates sample_weight to be used by the final step regressor (Yes, it is an outlier detection algorithm but we are focusing now on this interesting collateral result). I am currently aggressively changing the code in order to make it fit somehow with the pipeline """ from paella import Paella paella_input = pd.concat([data, clustering_output], axis=1, inplace=False) paella_run = Paella(noise_label = paella__noise_label, max_it = paella__max_it, regular_size = paella__regular_size, minimum_size = paella__minimum_size, width_r = paella__width_r, n_neighbors = paella__n_neighbors, power = paella__power, random_state = paella__random_state) paella_output = paella_run.fit_predict(paella_input, y_train) # paella_output is a vector with sample_weight #%% """ Here we fit a regressor using sample_weight=paella_output """ from sklearn.linear_model import LinearRegression regressor_input=X_train lm = LinearRegression() lm.fit(X=regressor_input, y=y_train, sample_weight=paella_output) regressor_output = lm.predict(X_train) #... In this example we can see that: - A particular step might need results produced not necessarily from the immediately previous step. - The X parameter is not sequentially transformed. Sometimes we might need to skip to a previous step - y sometimes is the target, sometimes is not. For the regressor it is indeed, but for the paella algorithm the prediction is expressed as a vector representing sample_weights. All in all the conclusion is that the chain of processes is not as linear as imposed by the current API. I guess that all these difficulties could be solved by: - Passing a dictionary through the different steps containing the partial results that the following steps will need. - As a christmas gift :-) , a reference to the pipeline itself inserted in that dictionary could provide access to the internal status of the previous steps should it be needed. Another interesting study case with similar needs would be a regressor using a previous clustering step in order to fit one model per cluster. In such case, the clustering results would be needed during the fitting. Thanks for your interest! Manolo

2 4

MLPClassifier as a feature selector
by Thomas Evangelidis Jan. 3, 2018

Jan. 3, 2018

Greetings, I want to train a MLPClassifier with one hidden layer and use it as a feature selector for an MLPRegressor. Is it possible to get the values of the neurons from the last hidden layer of the MLPClassifier to pass them as input to the MLPRegressor? If it is not possible with scikit-learn, is anyone aware of any scikit-compatible NN library that offers this functionality? For example this one: http://scikit-neuralnetwork.readthedocs.io/en/latest/index.html I wouldn't like to do this in Tensorflow because the MLP there is much slower than scikit-learn's implementation. Thomas -- ====================================================================== Dr Thomas Evangelidis Post-doctoral Researcher CEITEC - Central European Institute of Technology Masaryk University Kamenice 5/A35/2S049, 62500 Brno, Czech Republic email: tevang(a)pharm.uoa.gr tevang3(a)gmail.com website: https://sites.google.com/site/thomasevangelidishomepage/

6 7

Topic for thesis work on scikit learn
by Gaurav Dhingra Dec. 31, 2017

Dec. 31, 2017

Hi everyone, I am a final year (5th year) undergraduate Applied Mathematics student in India. I am thinking of doing my final year thesis by doing some work (coding part) on scikit learn, so I was thinking if anyone could tell me if there are available topics (not necessarily names of those topics) that I could work on being an undergraduate student? I would want to expand upon this in December when my exams will be over. But in the mean time would want to take a step in that direction by just knowing if there will be available topics that I could work on. It could be the case that available topics are not so easy for an undergraduate, still in that case I would like to do some research on the topics first. -- Best, Gaurav Dhingra (sent from Thunderbird email client)

2 4

Re: [scikit-learn] scikit-learn Digest, Vol 21, Issue 29
by Orges Leka Dec. 29, 2017

Dec. 29, 2017

Hello, You could use the following code: X_weight = [ ] for x in X: for i in range(len(mlp.coefs_)-1): x =np.array([math.tanh(v) for v in (x.dot(mlp.coefs_[i])+mlp.intercepts_[i])]) X_weight.append(x) where it is assumed that mlp is your trained MLP-Classifier, and you have trained with tanh-activation function X is your matrix which you want to compute the features, and x iterates over the vectors of this matrix. X_weight is a list of vectors with the computed weights. Kind regards Orges Leka 2017-12-29 17:46 GMT+01:00 <scikit-learn-request(a)python.org>: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: MLPClassifier as a feature selector (Thomas Evangelidis) > 2. Re: MLPClassifier as a feature selector (Javier L?pez) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 29 Dec 2017 12:09:00 +0100 > From: Thomas Evangelidis <tevang3(a)gmail.com> > To: Scikit-learn mailing list <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] MLPClassifier as a feature selector > Message-ID: > <CAACvdx0gO+5B7L6EyQbQSTWtoGBZKZKKPc0bGsxM > GkbDe6dd9w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Alright, with these attributes I can get the weights and biases, but what > about the values on the nodes of the last hidden layer? Do I have to work > them out myself or there is a straightforward way to get them? > > On 7 December 2017 at 04:25, Manoj Kumar <manojkumarsivaraj334(a)gmail.com> > wrote: > > > Hi, > > > > The weights and intercepts are available in the coefs_ and intercepts_ > > attribute respectively. > > > > See https://github.com/scikit-learn/scikit-learn/blob/ > > a24c8b46/sklearn/neural_network/multilayer_perceptron.py#L835 > > > > On Wed, Dec 6, 2017 at 4:56 PM, Brown J.B. via scikit-learn < > > scikit-learn(a)python.org> wrote: > > > >> I am also very interested in knowing if there is a sklearn cookbook > >> solution for getting the weights of a one-hidde-layer MLPClassifier. > >> J.B. > >> > >> 2017-12-07 8:49 GMT+09:00 Thomas Evangelidis <tevang3(a)gmail.com>: > >> > >>> Greetings, > >>> > >>> I want to train a MLPClassifier with one hidden layer and use it as a > >>> feature selector for an MLPRegressor. > >>> Is it possible to get the values of the neurons from the last hidden > >>> layer of the MLPClassifier to pass them as input to the MLPRegressor? > >>> > >>> If it is not possible with scikit-learn, is anyone aware of any > >>> scikit-compatible NN library that offers this functionality? For > example > >>> this one: > >>> > >>> http://scikit-neuralnetwork.readthedocs.io/en/latest/index.html > >>> > >>> I wouldn't like to do this in Tensorflow because the MLP there is much > >>> slower than scikit-learn's implementation. > >>> > >>> > >>> Thomas > >>> > >>> > >>> -- > >>> > >>> ====================================================================== > >>> > >>> Dr Thomas Evangelidis > >>> > >>> Post-doctoral Researcher > >>> CEITEC - Central European Institute of Technology > >>> Masaryk University > >>> Kamenice 5/A35/2S049, > >>> 62500 Brno, Czech Republic > >>> > >>> email: tevang(a)pharm.uoa.gr > >>> > >>> tevang3(a)gmail.com > >>> > >>> > >>> website: https://sites.google.com/site/thomasevangelidishomepage/ > >>> > >>> > >>> _______________________________________________ > >>> scikit-learn mailing list > >>> scikit-learn(a)python.org > >>> https://mail.python.org/mailman/listinfo/scikit-learn > >>> > >>> > >> > >> _______________________________________________ > >> scikit-learn mailing list > >> scikit-learn(a)python.org > >> https://mail.python.org/mailman/listinfo/scikit-learn > >> > >> > > > > > > -- > > Manoj, > > http://github.com/MechCoder > > > > _______________________________________________ > > scikit-learn mailing list > > scikit-learn(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-learn > > > > > > > -- > > ====================================================================== > > Dr Thomas Evangelidis > > Post-doctoral Researcher > CEITEC - Central European Institute of Technology > Masaryk University > Kamenice 5/A35/2S049, > 62500 Brno, Czech Republic > > email: tevang(a)pharm.uoa.gr > > tevang3(a)gmail.com > > > website: https://sites.google.com/site/thomasevangelidishomepage/ >

1 0

Re: [scikit-learn] Text classification of large dataset
by Ranjana Girish Dec. 27, 2017

Dec. 27, 2017

Hai all, Thank you for your suggestions. But I am still getting *memory error* while doing feature selection *fs = feature_selection.SelectPercentile(feature_selection.chi2, percentile=20)* *documenttermmatrix1 = fs.fit_transform(documenttermmatrix,y1)* *documenttermmatrix* will be of shape *(1594516,232832)* type of *documenttermmatrix * is *scipy csr matrix* Am I doing anything wrong? Is there any better way of doing feature selection?

1 0

Support Vector Machines: Sensitive to Single Datapoints?
by Taylor, Johnmark Dec. 22, 2017

Dec. 22, 2017

Hello, I am a researcher in fMRI and am using SVMs to analyze brain data. I am doing decoding between two classes, each of which has 24 exemplars per class. I am comparing two different methods of cross-validation for my data: in one, I am training on 23 exemplars from each class, and testing on the remaining example from each class, and in the other, I am training on 22 exemplars from each class, and testing on the remaining two from each class (in case it matters, the data is structured into different neuroimaging "runs", with each "run" containing several "blocks"; the first cross-validation method is leaving out one block at a time, the second is leaving out one run at a time). Now, I would've thought that these two CV methods would be very similar, since the vast majority of the training data is the same; the only difference is in adding two additional points. However, they are yielding very different results: training on 23 per class is yielding 60% decoding accuracy (averaged across several subjects, and statistically significantly greater than chance), training on 22 per class is yielding chance (50%) decoding. Leaving aside the particulars of fMRI in this case: is it unusual for single points (amounting to less than 5% of the data) to have such a big influence on SVM decoding? I am using a cost parameter of C=1. I must say it is counterintuitive to me that just a couple points out of two dozen could make such a big difference. Thank you very much, and cheers, JohnMark

6 5

Re: [scikit-learn] Text classification of large dataet
by Joel Nothman Dec. 20, 2017

Dec. 20, 2017

To clarify: You have 2.3M samples How many features? How many active features on average per sample? In 7k classes: multiclass or multilabel? Have you tried limiting the depth of the forest? Have you tried embedding your feature space into a smaller vector (pre-trained embeddings, hashing, lda, PCA or random projection)?

1 0

Text classification of large dataet
by Ranjana Girish Dec. 20, 2017

Dec. 20, 2017

Hai all, I am doing text classification. I have around 10 million data to be classified to around 7k category. Below is the code I am using *# Importing the libraries* *import pandas as pd* *import nltk* *from nltk.corpus import stopwords* *from nltk.tokenize import word_tokenize* *from nltk.stem.wordnet import WordNetLemmatizer* *from nltk.stem.porter import PorterStemmer* *import re* *from sklearn.feature_extraction.text import CountVectorizer* *import random* *from sklearn.naive_bayes import MultinomialNB,GaussianNB* *from sklearn.metrics import accuracy_score* *from sklearn.metrics import precision_recall_curve* *from sklearn.metrics import average_precision_score* *from sklearn import feature_selection* *from scipy.sparse import csr_matrix* *from scipy import sparse* *import sys* *from sklearn import preprocessing* *import numpy as np* *import pickle* *sys.setrecursionlimit(200000000)* *random.seed(20000)* *trainset1=pd.read_csv("trainsetgrt500sample10.csv",encoding = "ISO-8859-1")* *trainset2=pd.read_csv("trainsetlessequal500.csv",encoding = "ISO-8859-1")* *dataset=pd.concat([trainset1,trainset2])* *dataset=dataset.dropna()* *dataset['ProductDescription']=dataset['ProductDescription'].str.replace('[^a-zA-Z]', ' ')* *dataset['ProductDescription']=dataset['ProductDescription'].str.replace('[\d]', ' ')* *dataset['ProductDescription']=dataset['ProductDescription'].str.lower()* *del trainset1* *del trainset2 * *stop = stopwords.words('english')* *lemmatizer = WordNetLemmatizer()* *dataset['ProductDescription']=dataset['ProductDescription'].str.replace(r'\b(' + r'|'.join(stop) + r')\b\s*', ' ')* *dataset['ProductDescription']=dataset['ProductDescription'].str.replace('\s\s+',' ')* *dataset['ProductDescription'] =dataset['ProductDescription'].apply(word_tokenize)* *ADJ, ADJ_SAT, ADV, NOUN, VERB = 'a', 's', 'r', 'n', 'v'* *POS_LIST = [NOUN, VERB, ADJ, ADV]* *for tag in POS_LIST:* * dataset['ProductDescription'] = dataset['ProductDescription'].apply(lambda x: list(set([lemmatizer.lemmatize(item,tag) for item in x])))* *dataset['ProductDescription']=dataset['ProductDescription'].apply(lambda x : " ".join(x))* *countvec = CountVectorizer(min_df=0.00008)* *documenttermmatrix=countvec.fit_transform(dataset['ProductDescription'])* *documenttermmatrix.shape* *column=countvec.get_feature_names()* *filename1 = 'columnnamessample10mastermerge.sav'* *pickle.dump(column, open(filename1, 'wb'))* *y_train=dataset['classpath']* *y_train=dataset['classpath'].tolist()* *labels_train= preprocessing.LabelEncoder()* *labels_train.fit(y_train)* *y1_train=labels_train.transform(y_train)* *del dataset* *del countvec* *del column* *clf = MultinomialNB()* *model=clf.fit(documenttermmatrix,y_train)* *filename2 = 'modelnaivebayessample10withfs.sav'* *pickle.dump(model, open(filename2, 'wb'))* I am using system with *128 GB RAM.* As I was unable to train all 10 million data, I did *stratified sampling* and the trainset reduced to 2.3 million Still I was unable to Train 2.3 million data I got* memory error* when i used *random forest (nestimator=30),**Naive Bayes* and *SVM* *I have stucked* *Can Anyone please tell whether any memory leak in my code and how to use system with 128 GB RAM effectively* Thanks Ranjana

2 1

Parallel MLP version
by Luigi Lomasto Dec. 20, 2017

Dec. 20, 2017

Hi all, I have a computational problem to training my neural network so, can you say me if exists any parallel version about MLP library?

3 2