Mailman 3 July 2017 - scikit-learn

merging the predicted labels with original dataframe
by Ruchika Nayyar July 20, 2017

July 20, 2017

Hi Scikit-learn Users, I am analyzing some proxy logs to use Machine learning to classify the events recorded as either "OBSERVED" or "BLOCKED". This is a little snippet of my code: The input file is a csv with tokenized string fields. ************** # load the file M = pd.read_csv("output100k.csv").fillna('') # define the fields to use min_df = 0.001 max_df = .7 TxtCols = ['request__tokens', 'requestClientApplication__tokens', 'destinationZoneURI__tokens','cs-categories__tokens', 'fileType__tokens', 'requestMethod__tokens','tcp_status1', 'app','tcp_status2','dhost' ] NumCols = ['rt', 'out', 'in', 'time-taken','rt_length', 'dt_length'] # vectorize the fields TfidfModels = [TfidfVectorizer(min_df = min_df, max_df=max_df).fit(M[t]) for t in TxtCols] # define the columns of sparse matrix X = hstack([m.transform(M[n].fillna('')) for m,n in zip(TfidfModels, TxtCols)] + \ [csr_matrix(pd.to_numeric(M[n]).fillna(-1).values).T for n in NumCols]) # target variable Y = M.act.values ## Define train/test parts and scale them X_train, X_test, y_train, y_test = tts(X, Y, test_size=0.2) scaler = StandardScaler(with_mean=False, with_std=True) scaler.fit(X_train) X_train=scaler.transform(X_train) X_test=scaler.transform(X_test) # define the model and train clf = MLPClassifier(activation='logistic', solver='lbfgs').fit(X_train,y_train) # use the model to predict on X_test and convert into a data frame df=pd.DataFrame(clf.predict(X_test)) ** 199845 OBSERVED 199846 OBSERVED [199847 rows x 1 columns]> ** Now at the end I have a DataFrame with 20K entries with just one column "Label", how di I connect it to the main dataframe M, since I want to do some investigations on this outcome ? Any help? Thanks, Ruchika

3 4

Max f1 score for soft classifier?
by Stuart Reynolds July 18, 2017

July 18, 2017

Does scikit have a function to find the maximum f1 score (and decision threshold) for a (soft) classifier? - Stuart

4 5

scikit-learn 0.19b2 is available for testing
by Olivier Grisel July 17, 2017

July 17, 2017

The new release is coming and we are seeking feedback from beta testers! pip install scikit-learn==0.19b2 conda-forge packages should follow in the coming hours / days. Note that many models have changed behaviors and some things have been deprecated, see the full changelog at: http://scikit-learn.org/dev/whats_new.html#version-0-19 As usual please report any regression or other bugs as an issue on github. Thanks to anyone who contributed to the release! -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel

4 3

PR review request
by Ashim Bhattarai July 17, 2017

July 17, 2017

Hi -- I was wondering if somebody could review the pull request at https://github.com/scikit-learn/scikit-learn/pull/9348 in which I have worked on adding euclidean distance calculation in the presence of NaNs. Thanks in advance. Best, Ashim

1 0

Agglomerative clustering problem
by Ariani A July 15, 2017

July 15, 2017

Hi all, I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated! I have to use agglomerative clustering! Thanks, -Ariani

3 3

Inquiry third-party package affiliation
by Sebastian July 14, 2017

July 14, 2017

Hi, First off, thanks for a great package! A while ago I needed a package for building general-purpose ensembles combining any set of Scikit-learn transformers and estimators. I couldn't find any so I set out to develop such an extension and recently released the result as ML-Ensemble, http://mlens.readthedocs.io/en/latest/. I am contacting you to ask if you way consider adding this library to your reference list of related packages? It would be hugely appreciated to have a small mention on your site as an ensemble wrapper around Scikit-learn. A bit about ML-Ensemble: It is written in Python following with a Scikit-learn API; it uses joblib with memmapping to achieve scalable parallelization and any ensemble estimator can pass as a Scikit-learn estimator. The library is unit tested on Linux, Mac and Windows for Python 2.7, 3.5 and 3.6, and has been downloaded about four thousand times a month after launch. All the best, Sebastian Flennerhag

1 0

Line graph of weighted graph
by serafeim loukas July 14, 2017

July 14, 2017

Dear scikit-learn users, I would like to know if there is any function that returns the line graph of a weighted graph. I am aware of the linegraph function (http://igraph.org/python/doc/igraph.GraphBase-class.html#linegraph) but I would like to take the weights into consideration. Thank you, Makis

1 0

Contribution
by Gürhan Ceylan July 14, 2017

July 14, 2017

Hi everyone, I am wondering, How can I use external optimization algorithms with scikit-learn, for instance neural network <http://scikit-learn.org/stable/modules/neural_networks_supervised.html#algo…> , instead of defined algorithms ( Stochastic Gradient Descent, Adam, or L-BFGS). Furthermore, I want to introduce a new unconstrained optimization algorithm to scikit-learn, implementation of the algorithm and related paper can be found here <https://github.com/sibirbil/PMBSolve>. I couldn't find any explanation <http://scikit-learn.org/stable/developers/contributing.html>, about the situation. Do you have defined procedure to make such kind of contributions? If this is not the case, How should I start to make such a proposal/contribution ? Kind regards, Gürhan C.

4 4

Agglomerative Clustering without knowing number of clusters
by Ariani A July 13, 2017

July 13, 2017

I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated!

2 6

Contribution - Markov Clustering
by Uri Goren July 12, 2017

July 12, 2017

Hi, I've been advised to contact you before working on an implementation of a new feature. I am thinking of implementing the Markov clustering and add it to sklearn.cluster module. See: https://micans.org/mcl/ https://gist.github.com/urigoren/1f76567f3af56ed8c33f076537768a60 Do you know if anyone else has started working on it ? Would you advise against it for some reason ? Thank you, Uri

4 5