Mailman 3 July 2016 - scikit-learn

Re: [scikit-learn] [Scikit-learn-general] Estimator serialisability
by Joel Nothman July 14, 2016

July 14, 2016

This has been discussed numerous times. I suppose no one thinks supporting pickle only is great, but a custom dict is unmaintainable. The best we've got AFAIK (and it looks <https://github.com/jpmml/jpmml-sklearn/graphs/contributors> like it's getting better all the time) is a tool to convert one-way to PMML, which is portable to production environments. See https://github.com/jpmml/sklearn2pmml (python interface) and https://github.com/jpmml/jpmml-sklearn(command-line interface and guts of the thing). I hope that helps; and thanks to Villu Ruusmann: that list of supported estimators is awesome! PS: please write to the new list at scikit-learn(a)python.org On 14 July 2016 at 17:24, Miroslav Zoričák <miroslav.zoricak(a)gmail.com> wrote: > Hi everybody, > > I have been using scikit-learn for a while, but I have run into a problem > that does not seem to have any good solutions. > > Basically I would like to: > - build my pipeline in a Jupyter Notebook > - persist it (to json or hdf5) > - load it in production and execute the prediction there > > The problem is that for persisting estimators such as the RobustScaler for > example, the recommended way is to pickle them. Now I don't want to do > this, for three reasons: > > - Security, pickle is potentially dangerous > - Portability, I can't unpickle it in scala for example > - Pickle stores a lot of details and information which is not strictly > necessary to reconstruct the RobustScaler and therefore might prevent it > from being reconstructed correctly if a different version is used. > > Another option I would seem to have is to access the private members of > each serialiser that I want to use and store them on my own, but this is > inconvenient, because: > > - It forces me as a user to understand how the robust scaler works and how > it stores its internal state, which is generally bad for usability > - The internal implementation could change, leaving me to fix my > serialisers (see #1) > - I would need to do this for each new Estimator I decide to use > > Now, to me it seems the solution is quite obvious: > Write a Mixin or update the BaseEstimator class to include two additional > methods: > > to_dict() - will return a dictionary such, that when passed to > from_dict(dictionary) - it will reconstruct the original object > > these dictionaries could be passed to the JSON module or the YAML module > or stored elsewhere. We could provide more convenience methods to do this > for the user. > > In case of the RobustScaler the dict would look something like: > { "center": "0,0", "scale": "1.0"} > > Now the bulk of the work is writing these serialisers and deserialisers > for all of the estimators, but that can be simplified by adding a method > that could do that automatically via reflection and the estimator would > only need to specify which fields to serialise. > > I am happy to start working on this and create a pull request on Github, > but before I do that I wanted to get some initial thoughts and reactions > from the community, so please let me know what you think. > > Best Regards, > Miroslav Zoricak > -- > Best Regards, > Miroslav Zoricak > > > ------------------------------------------------------------------------------ > What NetFlow Analyzer can do for you? Monitors network bandwidth and > traffic > patterns at an interface-level. Reveals which users, apps, and protocols > are > consuming the most bandwidth. Provides multi-vendor support for NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > planning > reports.http://sdm.link/zohodev2dev > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > >

4 5

Bm25 pull request
by Basil Beirouti July 12, 2016

July 12, 2016

Hi, Joel thanks for pointing out the indentation issue. I have fixed it. Can someone explain what the 3 tests that were automatically run on my code are? And why did the Appveyor and Travis ones fail? Sincerely, Basil Beirouti Sent from my iPhone > On Jul 11, 2016, at 11:00 AM, scikit-learn-request(a)python.org wrote: > > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: Scikit learn GridSearchCV fit method ValueError Found > array with 0 sample (Maciek W?jcikowski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 11 Jul 2016 13:33:28 +0200 > From: Maciek W?jcikowski <maciek(a)wojcikowski.pl> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] Scikit learn GridSearchCV fit method > ValueError Found array with 0 sample > Message-ID: > <CAH2JJR1BqHC0PzNv7uaugkQ9GDBUTev4yuJ1qOWuJa=eWZ1wnQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Shouldn't you pass labels (binary) instead of continuous data? If you wish > to stick to logK's and keep the distribution unchanged then you'd better > reduce the number of classes (eg round the values to nearest integer?). > > It might be the case that the counts per class are floored and you get 0 > for some cases. > > ---- > Pozdrawiam, | Best regards, > Maciek W?jcikowski > maciek(a)wojcikowski.pl > > 2016-07-11 13:16 GMT+02:00 Micha? Nowotka <mmmnow(a)gmail.com>: > >> Hi Maciek, >> >> Thanks for suggestion, I think the problem indeed is related to the >> StratifiedKFold because if I use KFold instead the code works fine. >> However, if I print StratifiedKFold object it looks fine to me: >> >> sklearn.cross_validation.StratifiedKFold(labels=[ 5.43 8.74 8.1 >> 6.55 7.66 6.52 8.6 7.1 6.4 8.05 7.89 6.68 >> 8.06 6.17 5.5 7.96 5.78 6. 7.74 5.83 6.51 6.31 6.68 9.22 >> 6.07 7.06 7.12 8.64 5.72 6.4 7.64 5.74 7.41 6.49 6.81 7.1 >> 7.66 6.68 7.05 6.28 5.49 6.35 6.9 6.2 7.51 5.65 9.3 5.84 >> 6.92 5.75 6.92 8.8 7.04 5.81 5.73 5.31 7.13 7.66 6.98 5.93 >> 8.24 6.96 8.22 7.27 7.34 5.91 5.57 6.5 7.28 6.74 4.92 6.88 >> 5.8 9.15 6.63 6.37 8.66 6.4 ], n_folds=5, shuffle=False, >> random_state=None) >> >> >> On Fri, Jul 8, 2016 at 10:42 PM, Maciek W?jcikowski >> <maciek(a)wojcikowski.pl> wrote: >>> Hi Micha?, >>> >>> What are the class counts in that set? Maybe there is a problem with >>> generating stratified subsamples (eg some classes get below 1 sample)? >>> >>> ---- >>> Pozdrawiam, | Best regards, >>> Maciek W?jcikowski >>> maciek(a)wojcikowski.pl >>> >>> 2016-07-08 17:22 GMT+02:00 Micha? Nowotka <mmmnow(a)gmail.com>: >>>> >>>> Hi, >>>> >>>> Sorry for cross posting >>>> >>>> ( >> http://stackoverflow.com/questions/38263933/scikit-learn-gridsearchcv-fit-m… >> ) >>>> but I don't know where is better to get help with my problem. >>>> I'm working on a VM with Jupyter notebook server installed. >>>> From time to time I add new notebooks and reevaluate old ones to see >>>> if they still work. >>>> >>>> This notebook stopped working due to some changes in scikit-learn API >>>> and some parameters become obsolete: >> https://github.com/chembl/mychembl/blob/master/ipython_notebooks/10_myChEMB… >>>> >>>> I've created a corrected version of the notebook here: >>>> >>>> https://gist.github.com/anonymous/676c55cc501ffa48fecfcc1e1252d433 >>>> >>>> But I'm stuck in cell 36 on this code: >>>> >>>> from sklearn.cross_validation import KFold >>>> from sklearn.grid_search import GridSearchCV >>>> >>>> X_traina, X_testa, y_traina, y_testa = >>>> cross_validation.train_test_split(x, y, test_size=0.95, >>>> random_state=23) >>>> >>>> params = {'min_samples_split': [8], 'max_depth': [20], >>>> 'min_samples_leaf': [1],'n_estimators':[200]} >>>> cv = KFold(n=len(X_traina),n_folds=10,shuffle=True) >>>> cv_stratified = StratifiedKFold(y_traina, n_folds=5) >>>> gs = GridSearchCV(custom_forest, params, >>>> cv=cv_stratified,verbose=1,refit=True) >>>> gs.fit(X_traina,y_traina) >>>> >>>> This gives me: >>>> >>>> ValueError: Found array with 0 sample(s) (shape=(0, 491)) while a >>>> minimum of 1 is required. >>>> >>>> Now I don't understand this because when I print shapes of the samples: >>>> >>>> print (X_traina.shape, X_testa.shape, y_traina.shape, y_testa.shape) >>>> >>>> I'm getting: >>>> >>>> ((78, 491), (1489, 491), (78,), (1489,)) >>>> >>>> Interestingly, if I change the test_size parameter to 0.88 (like in >>>> the example corrected notebook) it works and this is the highest value >>>> where it works. For this value, the shapes are: >>>> >>>> ((188, 491), (1379, 491), (188,), (1379,)) >>>> >>>> So the question is - what should I change in my code to make it work >>>> for test_size set to 0.95 as well? >>>> >>>> Kind regards, >>>> >>>> Michal Nowotka >>>> _______________________________________________ >>>> scikit-learn mailing list >>>> scikit-learn(a)python.org >>>> https://mail.python.org/mailman/listinfo/scikit-learn >>> >>> >>> >>> _______________________________________________ >>> scikit-learn mailing list >>> scikit-learn(a)python.org >>> https://mail.python.org/mailman/listinfo/scikit-learn >> _______________________________________________ >> scikit-learn mailing list >> scikit-learn(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-learn >

2 1

Scikit learn GridSearchCV fit method ValueError Found array with 0 sample
by Michał Nowotka July 11, 2016

July 11, 2016

Hi, Sorry for cross posting (http://stackoverflow.com/questions/38263933/scikit-learn-gridsearchcv-fit-m…) but I don't know where is better to get help with my problem. I'm working on a VM with Jupyter notebook server installed. >From time to time I add new notebooks and reevaluate old ones to see if they still work. This notebook stopped working due to some changes in scikit-learn API and some parameters become obsolete: https://github.com/chembl/mychembl/blob/master/ipython_notebooks/10_myChEMB… I've created a corrected version of the notebook here: https://gist.github.com/anonymous/676c55cc501ffa48fecfcc1e1252d433 But I'm stuck in cell 36 on this code: from sklearn.cross_validation import KFold from sklearn.grid_search import GridSearchCV X_traina, X_testa, y_traina, y_testa = cross_validation.train_test_split(x, y, test_size=0.95, random_state=23) params = {'min_samples_split': [8], 'max_depth': [20], 'min_samples_leaf': [1],'n_estimators':[200]} cv = KFold(n=len(X_traina),n_folds=10,shuffle=True) cv_stratified = StratifiedKFold(y_traina, n_folds=5) gs = GridSearchCV(custom_forest, params, cv=cv_stratified,verbose=1,refit=True) gs.fit(X_traina,y_traina) This gives me: ValueError: Found array with 0 sample(s) (shape=(0, 491)) while a minimum of 1 is required. Now I don't understand this because when I print shapes of the samples: print (X_traina.shape, X_testa.shape, y_traina.shape, y_testa.shape) I'm getting: ((78, 491), (1489, 491), (78,), (1489,)) Interestingly, if I change the test_size parameter to 0.88 (like in the example corrected notebook) it works and this is the highest value where it works. For this value, the shapes are: ((188, 491), (1379, 491), (188,), (1379,)) So the question is - what should I change in my code to make it work for test_size set to 0.95 as well? Kind regards, Michal Nowotka

2 3

Added BM25Transformer and BM25Vectorizer to sklearn.feature_extraction.text
by Basil Beirouti July 10, 2016

July 10, 2016

Hi all, I have submitted a pull request to the main branch. I added BM25Transformer and BM25Vectorizer, which are very similar to TFIDFTransformer and TFIDFVectorizer, except they implement the BM25 algorithm instead. Would really appreciate feedback on the quality of my work and how I can improve. Sincerely, Basil Beirouti On Sat, Jul 9, 2016 at 11:00 AM, <scikit-learn-request(a)python.org> wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: Create a "Feature_Weight" Parameter at > RandomForestRegressor (Andreas Mueller) > 2. Re: Scikit learn GridSearchCV fit method ValueError Found > array with 0 sample (Maciek W?jcikowski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 8 Jul 2016 17:00:42 -0400 > From: Andreas Mueller <t3kcit(a)gmail.com> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] Create a "Feature_Weight" Parameter at > RandomForestRegressor > Message-ID: <5780147A.5040901(a)gmail.com> > Content-Type: text/plain; charset=windows-1252; format=flowed > > You would need to implement a custom splitter, I think. > > On 07/04/2016 04:09 PM, luizfgoncalves(a)dcc.ufmg.br wrote: > > I would like to give different weights to the features in the feature set > > for the split task of Random Forest. Right now, only the MSE metric is > > used to select the best split, and I want to do something like feature[i] > > = MSI[i] * feature_weight[i]. This way, I'll be able to give more > > importance to the features I already know that are better. > > > > In my mind, this change would be called on the fit function, something > > like this: def fit(self, X, y, sample_weight, feature_weight): > > And the feature_weight would be a vector with customized weights for all > > features present in the dataset. > > > > What is the best way to do that? I'm having a really hard time figuring > > out how to do this changes on the code. > > Thanks a lot for your attention. > > > > Luiz Felipe > > > > _______________________________________________ > > scikit-learn mailing list > > scikit-learn(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-learn > > > > ------------------------------ > > Message: 2 > Date: Fri, 8 Jul 2016 23:42:06 +0200 > From: Maciek W?jcikowski <maciek(a)wojcikowski.pl> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] Scikit learn GridSearchCV fit method > ValueError Found array with 0 sample > Message-ID: > < > CAH2JJR35CFDJPqTNFn7+uSCVKUVJEPM9mjYDwLTgkipLeWcVCw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Micha?, > > What are the class counts in that set? Maybe there is a problem with > generating stratified subsamples (eg some classes get below 1 sample)? > > ---- > Pozdrawiam, | Best regards, > Maciek W?jcikowski > maciek(a)wojcikowski.pl > > 2016-07-08 17:22 GMT+02:00 Micha? Nowotka <mmmnow(a)gmail.com>: > > > Hi, > > > > Sorry for cross posting > > ( > > > http://stackoverflow.com/questions/38263933/scikit-learn-gridsearchcv-fit-m… > > ) > > but I don't know where is better to get help with my problem. > > I'm working on a VM with Jupyter notebook server installed. > > From time to time I add new notebooks and reevaluate old ones to see > > if they still work. > > > > This notebook stopped working due to some changes in scikit-learn API > > and some parameters become obsolete: > > > > > > > https://github.com/chembl/mychembl/blob/master/ipython_notebooks/10_myChEMB… > > > > I've created a corrected version of the notebook here: > > > > https://gist.github.com/anonymous/676c55cc501ffa48fecfcc1e1252d433 > > > > But I'm stuck in cell 36 on this code: > > > > from sklearn.cross_validation import KFold > > from sklearn.grid_search import GridSearchCV > > > > X_traina, X_testa, y_traina, y_testa = > > cross_validation.train_test_split(x, y, test_size=0.95, > > random_state=23) > > > > params = {'min_samples_split': [8], 'max_depth': [20], > > 'min_samples_leaf': [1],'n_estimators':[200]} > > cv = KFold(n=len(X_traina),n_folds=10,shuffle=True) > > cv_stratified = StratifiedKFold(y_traina, n_folds=5) > > gs = GridSearchCV(custom_forest, params, > > cv=cv_stratified,verbose=1,refit=True) > > gs.fit(X_traina,y_traina) > > > > This gives me: > > > > ValueError: Found array with 0 sample(s) (shape=(0, 491)) while a > > minimum of 1 is required. > > > > Now I don't understand this because when I print shapes of the samples: > > > > print (X_traina.shape, X_testa.shape, y_traina.shape, y_testa.shape) > > > > I'm getting: > > > > ((78, 491), (1489, 491), (78,), (1489,)) > > > > Interestingly, if I change the test_size parameter to 0.88 (like in > > the example corrected notebook) it works and this is the highest value > > where it works. For this value, the shapes are: > > > > ((188, 491), (1379, 491), (188,), (1379,)) > > > > So the question is - what should I change in my code to make it work > > for test_size set to 0.95 as well? > > > > Kind regards, > > > > Michal Nowotka > > _______________________________________________ > > scikit-learn mailing list > > scikit-learn(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-learn > > >

1 0

Create a "Feature_Weight" Parameter at RandomForestRegressor
by luizfgoncalves＠dcc.ufmg.br July 8, 2016

July 8, 2016

I would like to give different weights to the features in the feature set for the split task of Random Forest. Right now, only the MSE metric is used to select the best split, and I want to do something like feature[i] = MSI[i] * feature_weight[i]. This way, I'll be able to give more importance to the features I already know that are better. In my mind, this change would be called on the fit function, something like this: def fit(self, X, y, sample_weight, feature_weight): And the feature_weight would be a vector with customized weights for all features present in the dataset. What is the best way to do that? I'm having a really hard time figuring out how to do this changes on the code. Thanks a lot for your attention. Luiz Felipe

2 1

GradientBoostingRegressor with training, validation, and test set.
by Olga Lyashevska July 8, 2016

July 8, 2016

Hi, I implement GradientBoostingRegressor algorithm. I randomly divide the dataset into three parts: a training set (50%), a validation set (25%), and a test set (25%). I understand that the training set is used for model fitting (1); the validation set is used for estimation of prediction error for model selection (2); and, finally, the test set is used for assessment of the final chosen model (3). However, I am not sure how to implement this. Can anyone give any examples? Many thanks, Olga X_train, X_test, y_train, y_test = cv.train_test_split(X, y, test_size=0.5) X_test, X_val, y_test, y_val = cv.train_test_split(X_test, y_test, test_size=0.5) params = {'n_estimators': 2000, 'max_depth': 4, 'min_samples_leaf': 4, 'learning_rate': 0.01, 'min_samples_split': 1, 'subsample': 0.75, 'random_state': 42, 'loss': 'ls'} est = ensemble.GradientBoostingRegressor(**params) est.fit(X_train, y_train) # 1 mean_squared_error(y_test, est.predict(X_test)) # 3

2 4

Re: [scikit-learn] Using fit_intercept with sparse matrices
by Tom DLT July 5, 2016

July 5, 2016

note2: The LogisticRegression and Ridge(solver='sag') code do fit the intercept without breaking sparsity. For other solvers in Ridge, in the case of a sparse X input, the solver will automatically be changed to 'sag' and raise a warning. Tom 2016-07-04 7:24 GMT+02:00 Tom Dupré la Tour <tom.duprelatour.10(a)gmail.com>: > note2: > > The LogisticRegression and Ridge(solver='sag') code do fit the intercept > without breaking sparsity. > > For other solvers in Ridge, in the case of a sparse X input, the solver > will automatically be changed to 'sag' and raise a warning. > Le 2 juil. 2016 15:48, "Alexandre Gramfort" < > alexandre.gramfort(a)telecom-paristech.fr> a écrit : > >> note: >> >> the Lasso and ElasticNet code do fit the intercept without breaking >> sparsity. >> >> Alex >> _______________________________________________ >> scikit-learn mailing list >> scikit-learn(a)python.org >> https://mail.python.org/mailman/listinfo/scikit-learn >> >

7 7

Using fit_intercept with sparse matrices
by Jaidev Deshpande July 2, 2016

July 2, 2016

Hi, I usually encounter many cases when I've forgotten that my input to the `AnyEstimator.fit` method is a sparse matrix, and I've set `fit_intercept=False`. To avoid this, I could of course make a habit of not tampering with the default `fit_intercept=True`, but I think it would be better and more idiot-proof if the method raises a warning saying something like, "fit_intercept=False works best when the data is centered, and sparse matrices can't be centered. Please set fit_intercept=True for better results." What do you think? If doing this is worthwhile, I'll send a PR. Thanks!

2 1

Adding BM25 to sklearn.feature_extraction.text
by Basil Beirouti July 2, 2016

July 2, 2016

Hi Joel, I'm not by my dev computer right now so I can't show you the code, but the problem is that the term frequency - f(q,D) in that wiki article - appears in both the numerator and the denominator. Also, in the denominator, you must add a scalar quantity to f(q,D), which is unsupported if f(q,D) is coming from a sparse matrix. You can factor the equation in different ways but you can't get around the main issue that the sparse matrix must appear in the numerator and denominator. So instead of doing any actual matrix multiplication I just loop on the non-zero elements in the sparse matrix (term-frequency matrix) and fill in the new BM25matrix, element by element. Any suggestions that use actual sparse matrix operations would be appreciated. I know I can also loop on the non-zero elements and construct a sparse csr_matrix from that using the.indptr attribute etc. but I'm hoping there's a way to use matrix operations. Sincerely, Basil Beirouti On Fri, Jul 1, 2016 at 11:00 AM, <scikit-learn-request(a)python.org> wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Adding BM25 to sklearn.feature_extraction.text (Update) > (Basil Beirouti) > 2. Re: Adding BM25 to sklearn.feature_extraction.text (Update) > (Joel Nothman) > 3. Re: Adding BM25 to sklearn.feature_extraction.text (Update) > (Sebastian Raschka) > 4. Re: partial_fit implementation for IsolationForest > (donkey-hotei(a)cryptolab.net) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 30 Jun 2016 17:23:18 -0500 > From: Basil Beirouti <basilbeirouti(a)gmail.com> > To: scikit-learn(a)python.org > Subject: [scikit-learn] Adding BM25 to sklearn.feature_extraction.text > (Update) > Message-ID: > < > CAB4mTg8tMwoA0NwsfXmVtYWqS547F2NOmP5vj3LTCaNqXjqeWQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello everyone, > > I have successfully created a few versions of the BM25Transformer. I looked > at TFIDFTransformer for guidance and I noticed that it outputs a sparse > matrix when given a sparse termcount matrix as an input. > > Unfortunately, the fastest implementation of BM25Transformer that I have > been able to come up with does NOT output a sparse matrix, it will return a > regular numpy matrix. > > Benchmarked against the entire 20newsgroups corpus, here is how they > perform (assuming input is csr_matrix for all): > > 1.) finishes in 4 seconds, outputs a regular numpy matrix > 2.) finishes in 30 seconds, outputs a dok_matrix > 3.) finishes in 130 seconds, outputs a regular numpy matrix > > It's worth noting that using algorithm 1 and converting the output to a > sparse matrix still takes less time than 3, and takes about as long as 2. > > So my question is, how important is it that my BM25Transformer outputs a > sparse matrix? > > I'm going to try another implementation which looks directly at the data, > indices, and indptr attributes of the inputted csr_matrix. I just wanted to > check in and see what people thought. > > Sincerely, > Basil Beirouti >

2 1

Bm25
by Basil Beirouti July 1, 2016

July 1, 2016

Hi Vlad, Thanks for the quick reply. Unfortunately there's still the question of adding a scalar to every element in sparse matrix, which is not allowed for sparse matrices, and which is not possible to avoid in the equation. Sincerely, Basil Beirouti > On Jul 1, 2016, at 4:36 PM, scikit-learn-request(a)python.org wrote: > > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Adding BM25 to scikit-learn.feature_extraction.text > (Basil Beirouti) > 2. Re: Adding BM25 to scikit-learn.feature_extraction.text > (Vlad Niculae) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 1 Jul 2016 16:17:43 -0500 > From: Basil Beirouti <basilbeirouti(a)gmail.com> > To: scikit-learn(a)python.org > Subject: [scikit-learn] Adding BM25 to > scikit-learn.feature_extraction.text > Message-ID: > <CAB4mTg8805nNdAja5cscf+pHrJyq0btC-AGzegd8Cqb95sVdHw(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi everyone, > > to put it succinctly, here's the BM25 equation: > > f(w,D) * (k+1) / (k*B + f(w,D)) > > where w is the word, and D is the document (corresponding to rows and > columns, respectively). f is a sparse matrix because only a fraction of the > whole vocabulary of words appears in any given single document. > > B is a function of only the document, but it doesn't matter, you can think > of it as a constant if you want. > > The problem is since f(w,D) is almost always zero, I only need to do the > calculation (ie. multiply by (k+1) then divide by (k*B + f(w,D))) when > f(w,D) is not zero. Is there a clever way to do this with masks? > > You can refactor the above equation to get this: > > (k+1)/(k*B/f(w,D) + 1) but alas we still have f(w,D) appearing in a > denominator, which is bad (because of dividing by zero). > > So anyway, currently I am converting to a coo_matrix and iterator through > the non-zero values like this: > > cx = x.tocoo() > for i,j,v in itertools.izip(cx.row, cx.col, cx.data): > (i,j,v) > > > That iterator is incredibly fast, but unfortunately coo_matrix does > not support assignment. So I create a new copy of either a dok sparse > matrix or a regular numpy array and assign to that. > > I could also deal directly with the .data, .indptr, and indices > attributes of csr_matrix, and see if it's possible to create a copy of > .data attribute and update the values accordingly. I was hoping > somebody had encountered this type of issue before. > > Sincerely, > > Basil Beirouti >

2 4