Mailman 3 June 2016 - scikit-learn

isotonic regression weird behavior?
by Jonathan Taylor June 22, 2016

June 22, 2016

Was trying to fit isotonic regression with non-trivial y_min and y_max: In [*17*]: X Out[*17*]: array([ 1.26336413, 1.31853693, -0.57200917, 0.3072928 , -0.70686507, -0.17614937, -1.59943059, 1.05908504, 1.3958263 , 1.90580318, 0.20992272, 0.02836316, -0.08092235, 0.44438247, 0.01791253, -0.3771914 , -0.89577538, -0.37726249, -1.32687569, 0.18013201]) In [*18*]: iso.isotonic_regression(X, y_min=0, y_max=0.1) Out[*18*]: array([-0.00826919, -0.00826919, -0.00826919, -0.00826919, -0.00826919, -0.00826919, -0.00826919, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344, 0.10449344]) The solution does not satisfy the bounds that each entry should be in [0,0.1] -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137 390 Serra Mall Stanford, CA 94305 Tel: 650.723.9230 Fax: 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo

4 5

Decoding Differences Between SKL SVM and Matlab Libsvm Even When Parameters the Same
by Taylor, Johnmark June 22, 2016

June 22, 2016

Hello, I am moving much of my neuroimaging coding over to Python from Matlab and so I am switching from using libsvm in Matlab to using Scikit-learn SVM in Python. Just to make sure I am not changing anything substantive about my analyses, I am experimenting with the two implementations and trying to see whether I can get them to yield identical results. In Python I am using: clf = svm.SVC(kernel='linear',C=1,probability=True) In Matlab (libsvm) I am using: clf = libsvmtrain(svm_training_labels,svm_training_vectors,['-t 0 -b 1 -c 1']) When I run the SVM using these two different ways using simulated data, I get subtly different results, even though I have fixed all of the parameters of the SVMs to be the same using input arguments (linear classifier, C=1, use probability estimates), and even though all the other default parameters seem to be the same across these functions (tolerance = .001, both using shrinking heuristics by default). To give more details regarding the simulations: One simulation I ran was designed to be absurdly difficult--it yielded 40% accuracy for Matlab libsvm, and 44% accuracy for scikit-learn svm (binary classification, chance = 50%). In this simulation, the two SVMs agreed in their predictions only 18% of the time (in other words, they were both not only guessing below chance, but they nearly always gave opposite guesses compared to each other). The other simulation was easier, yielding 68% accuracy for Matlab libsvm, and 67% accuracy for scikit-learn SVM. In this simulation, the two SVMs agreed in their predictions 97% of the time. So even though they often got it wrong, they tended to make the same wrong guesses. Any idea of what could possibly be leading to differences in the results? My understanding is that SKL uses libsvm under the hood, so it's a been confusing why the decoders are behaving differently. Both analyses are being run on the same computer (Linux OS). Thank you very much, JohnMark Taylor PhD Student, Harvard Vision Sciences Lab

3 2

The sum of feature importances !=1
by Enhui HUANG June 21, 2016

June 21, 2016

Hi, When I ran the following code: X, y = make_classification(n_samples=100) clf = GradientBoostingClassifier(random_state=0).fit(X, y) imp=clf.feature_importances_ print "The sum of feature importances:", sum(imp) The sum of feature importances is not always equal to 1. So do you have a nice explanation for this situation? Besides, if a tree only contains a root, could we say all its feature importances are 0? I guess the root trees will influence sum of feature importances. Is it right? Best, Enhui

1 0

interpretation of manifold.MDS results
by Bill Ross June 19, 2016

June 19, 2016

I have a set of photos and a set of metrics for assigning distances between them. I am out to find the dimensionality of (space needed to embed) each set of distances, out of curiosity and to inform my use of the metrics in forming the character of an AI. It was suggested that manifold.MDS could help, and it looked perfect for a toy case I wrote. However, with actual data I found that the stress_ term doesn't go to 0 when the number of components reaches the number of points: http://stackoverflow.com/questions/37855596/calculate-the-spatial-dimension… I'm not sure if this question has achieved visibility, since I added the scikit-learn tag after it was created. I'd appreciate any discussion of the meaning of the value in the limit for stress_, and whether a meaningful dimensionality measure can be derived. Thanks, Bill

1 0

The culture of commit squashing
by Joel Nothman June 18, 2016

June 18, 2016

For the last few years, there's been a notion that we should squash PRs down to a single commit before merging. Squashing can give a cleaner commit history, and avoid overrepresentation of minor work given silly commit count metrics used by Github and others. I'm not sure if there are other motivations. Recently I've seen several contributors amending commits and force-pushing changes. I find this disruptive to the reviewing process in a number of ways (links are broken; what's changed is hard to discern, when it could have been indicated in a commit message and diff; etc.). I have had to ask these several users to cease and desist. I also find that performing the squash can be unnecessary overhead either for the merger or the PR developer. I think squashing is more trouble than it's worth, except where: * there are embarrassingly many minor commits in a PR * squashing first makes a rebase easier because of concurrent changes to the codebase * otherwise for cosmetic reasons only when there is low reviewing activity on the PR While squashing is far from the slowest part of our review process, being able to hit the merge button and move on would be great. Do others agree that a culture of amending commits in the ordinary case is counterproductive? (And apologies for wasting your time on such a silly issue, but I'm sick of clicking links in emails to find the commit's disappeared.)

10 15

Estimator.predict() thread safety
by Philip Tully June 17, 2016

June 17, 2016

Hi all, I notice when I train a model and expose the predict function through a web API, predict takes longer to run in a multi-threaded environment than a single-threaded one. I'm guessing the root cause has something to do with thread collisions but I must be doing something incorrectly within the code (I set n_jobs=-1 for both FeatureUnion and estimators/gridsearchers there) has someone else ran into a similar issue? I can provide more details if this Q is rather opaque still best, Philip

3 7

Question about error of LLE and backtransformation of coordinates
by Matthieu Brucher June 16, 2016

June 16, 2016

Hi! The errors are quite small compared to the machine precision. As the reduction is also an approximation of the underlying manifold, not an "isotropic" one as well (you can see int he example that red points are less squashed together than blue ones), you won't have a perfect reconstruction either. In a way, if you can reproduce in the reduced space the same distances (or barycenters for LLE) compared the original space, then you can have a perfect reconstruction (but it will still be subject to floating point precision). For the sphere, you can't: take 4 points, can you make the fourth as a barycenter of the other 3? No. That's the error you are seeing. Cheers, Matthieu 2016-06-16 10:12 GMT+01:00 Unger, Jörg <joerg.unger(a)bam.de>: > > I’ve tried the example that is available here > > > > http://scikit-learn.org/stable/auto_examples/manifold/plot_manifold_sphere.… > > > > These are essentially points on a 3D sphere, so the dimension of the embedded manifold is two. > > I’ve changed the example a little bit to extract the error as well. So instead of > > > > trans_data = manifold\ > > .LocallyLinearEmbedding(n_neighbors, 2, > > method=method).fit_transform(sphere_data).T > > > > I’ve done something like > > solver = manifold.LocallyLinearEmbedding(n_neighbors, dim_y, method=method) > > trans_data = solver.fit_transform(sphere_data).T > > error = solver.reconstruction_error_ > > > > I would have expected the error to be significant for dim_y=1, since I can’t reproduce with just a single coordinate the results. For dim_y=2, I expected a significant decrease, and for dim_y=3, I expected to exactly recover the original result. > > > > What I get is (for standard LLE) > > dim_y = 1 : error = 1.62031573333e-07 > > dim_y = 2 : error = 1.79465538543e-06 > > dim_y = 3 : error = 7.00280676182e-06 > > > > Could anyone explain, why I do not get the expected results? > > > > Furthermore, is there an option to retransform the coordinates from the local dimension to the global dimension? I’m interested in transforming the original global samples to local coordinates (this is done via the transform method), but then I would like to transform samples from coordinates in the embedded space back into the global space. > > > > Best regards, > > Jörg F. Unger > > > ------------------------------------------------------------------------------ > What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic > patterns at an interface-level. Reveals which users, apps, and protocols are > consuming the most bandwidth. Provides multi-vendor support for NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity planning > reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381 > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > -- Information System Engineer, Ph.D. Blog: http://blog.audio-tk.com/ LinkedIn: http://www.linkedin.com/in/matthieubrucher -- Information System Engineer, Ph.D. Blog: http://blog.audio-tk.com/ LinkedIn: http://www.linkedin.com/in/matthieubrucher

1 0

Re: [scikit-learn] adding BM25 relevance function
by Basil Beirouti June 16, 2016

June 16, 2016

Hello Pavel and Joel, I forked the repository and cloned it on my machine. I'm using pycharm on a Mac, and while looking at text.py, I'm getting an unresolved reference for "xrange" at line 28: from ..externals.six.moves import range Pycharm says Function 'six.py' is too large to analyze, so I'm not sure if this error is somehow related to that. I decided to try to build the code as a sanity check but I can't find any reliable instructions as to how to do that. Naively, I opened terminal and cd to the directory above "scikit-learn" folder (where I had cloned my fork) and tried to run: $ python3 setup.py install Which didn't work. I got this error: ImportError: No module named 'sklearn' Can someone point me in the right direction? And how can the code try to import sklearn if it doesn't exist yet? Note I haven't installed the release version of scikit-learn using pip or any other tool, but I should be able to bootstrap it from the source code, right? Here's the full error message if it helps. Forgive me if it's a silly mistake, but I haven't found any reliable guidelines online. File "setup.py", line 84, in <module> from numpy.distutils.core import setup File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/distutils/core.py", line 26, in <module> from numpy.distutils.command import config, config_compiler, \ File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/distutils/command/build_ext.py", line 18, in <module> from numpy.distutils.system_info import combine_paths File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/distutils/system_info.py", line 232, in <module> triplet = str(p.communicate()[0].decode().strip()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 791, in communicate stdout = _eintr_retry_call(self.stdout.read) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) KeyboardInterrupt Basils-MacBook-Pro:sklearn basilbeirouti$ python3 setup.py install non-existing path in '__check_build': '_check_build.c' Appending sklearn.__check_build configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.__check_build') Appending sklearn._build_utils configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn._build_utils') Appending sklearn.covariance configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.covariance') Appending sklearn.covariance/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.covariance/tests') Appending sklearn.cross_decomposition configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.cross_decomposition') Appending sklearn.cross_decomposition/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.cross_decomposition/tests') Appending sklearn.feature_selection configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.feature_selection') Appending sklearn.feature_selection/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.feature_selection/tests') Appending sklearn.gaussian_process configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.gaussian_process') Appending sklearn.gaussian_process/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.gaussian_process/tests') Appending sklearn.mixture configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.mixture') Appending sklearn.mixture/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.mixture/tests') Appending sklearn.model_selection configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.model_selection') Appending sklearn.model_selection/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.model_selection/tests') Appending sklearn.neural_network configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.neural_network') Appending sklearn.neural_network/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.neural_network/tests') Appending sklearn.preprocessing configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.preprocessing') Appending sklearn.preprocessing/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.preprocessing/tests') Appending sklearn.semi_supervised configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.semi_supervised') Appending sklearn.semi_supervised/tests configuration to sklearn Ignoring attempt to set 'name' (from 'sklearn' to 'sklearn.semi_supervised/tests') Warning: Assuming default configuration (./_build_utils/{setup__build_utils,setup}.py was not found)Warning: Assuming default configuration (./covariance/{setup_covariance,setup}.py was not found)Warning: Assuming default configuration (./covariance/tests/setup_covariance/{setup_covariance/tests,setup}.py was not found)Warning: Assuming default configuration (./cross_decomposition/{setup_cross_decomposition,setup}.py was not found)Warning: Assuming default configuration (./cross_decomposition/tests/setup_cross_decomposition/{setup_cross_decomposition/tests,setup}.py was not found)Warning: Assuming default configuration (./feature_selection/{setup_feature_selection,setup}.py was not found)Warning: Assuming default configuration (./feature_selection/tests/setup_feature_selection/{setup_feature_selection/tests,setup}.py was not found)Warning: Assuming default configuration (./gaussian_process/{setup_gaussian_process,setup}.py was not found)Warning: Assuming default configuration (./gaussian_process/tests/setup_gaussian_process/{setup_gaussian_process/tests,setup}.py was not found)Warning: Assuming default configuration (./mixture/{setup_mixture,setup}.py was not found)Warning: Assuming default configuration (./mixture/tests/setup_mixture/{setup_mixture/tests,setup}.py was not found)Warning: Assuming default configuration (./model_selection/{setup_model_selection,setup}.py was not found)Warning: Assuming default configuration (./model_selection/tests/setup_model_selection/{setup_model_selection/tests,setup}.py was not found)Warning: Assuming default configuration (./neural_network/{setup_neural_network,setup}.py was not found)Warning: Assuming default configuration (./neural_network/tests/setup_neural_network/{setup_neural_network/tests,setup}.py was not found)Warning: Assuming default configuration (./preprocessing/{setup_preprocessing,setup}.py was not found)Warning: Assuming default configuration (./preprocessing/tests/setup_preprocessing/{setup_preprocessing/tests,setup}.py was not found)Warning: Assuming default configuration (./semi_supervised/{setup_semi_supervised,setup}.py was not found)Warning: Assuming default configuration (./semi_supervised/tests/setup_semi_supervised/{setup_semi_supervised/tests,setup}.py was not found)Traceback (most recent call last): File "setup.py", line 85, in <module> setup(**configuration(top_path='').todict()) File "setup.py", line 44, in configuration config.add_subpackage('cluster') File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy/distutils/misc_util.py", line 1003, in add_subpackage caller_level = 2) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy/distutils/misc_util.py", line 972, in get_subpackage caller_level = caller_level + 1) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy/distutils/misc_util.py", line 884, in _get_configuration_from_setup_py ('.py', 'U', 1)) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imp.py", line 234, in load_module return load_source(name, filename, file) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/imp.py", line 172, in load_source module = _load(spec) File "<frozen importlib._bootstrap>", line 693, in _load File "<frozen importlib._bootstrap>", line 673, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 662, in exec_module File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed File "./cluster/setup.py", line 8, in <module> from sklearn._build_utils import get_blas_info ImportError: No module named 'sklearn' On Tue, Jun 14, 2016 at 11:41 AM, <scikit-learn-request(a)python.org> wrote: > Send scikit-learn mailing list submissions to > scikit-learn(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scikit-learn > or, via email, send a message with subject or body 'help' to > scikit-learn-request(a)python.org > > You can reach the person managing the list at > scikit-learn-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of scikit-learn digest..." > > > Today's Topics: > > 1. Re: Adding BM25 relevance function (Pavel Soriano) > 2. Re: The culture of commit squashing (Andreas Mueller) > 3. Re: The culture of commit squashing (Tom DLT) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 14 Jun 2016 16:11:10 +0000 > From: Pavel Soriano <sorianopavel(a)gmail.com> > To: Scikit-learn user and developer mailing list > <scikit-learn(a)python.org> > Subject: Re: [scikit-learn] Adding BM25 relevance function > Message-ID: > < > CAN0wWk93r2aw9No65CGiCW5hQG7-oFYVZaMJQpXpegTXMSqPLg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hey, > > Good thing that you are trying to finish this. > > Well, I looked into my old notes, and the Delta tf-idf comes from the > "Delta > TFIDF: An Improved Feature Space for Sentiment Analysis" > <http://ebiquity.umbc.edu/_file_directory_/papers/446.pdf> paper. I guess > it is not very popular and apparently it has a drawback: it does not take > into account the number of times a word occurs in each document while > calculating the distribution amongst classes. At least that is what I wrote > on my notes... > > As for the delta idf... If it helps, I can look into my old code cause I do > not know what I was talking about. I guess it has to do somehow with the > paper cited before. > > Cheers, > > Pavel Soriano > > > > > On Tue, Jun 14, 2016 at 5:49 PM Basil Beirouti <basilbeirouti(a)gmail.com> > wrote: > > > Hi Joel, > > > > Thanks for your response and for digging up that archived thread, it > gives > > me a lot of clarity. > > > > I see your point about BM25, but I think in most cases where TFIDF makes > > sense, BM25 makes sense as well, but it could be "overkill". > > > > Consider that TFIDF does not produce normalized results either > > < > http://scikit-learn.org/stable/auto_examples/text/document_clustering.html#… > >, > > If BM25 requires dimensionality reduction (eg. using LSA) , so too would > > TFIDF. The term-document matrix is the same size no matter which > weighting > > scheme is used. The only difference is that BM25 produces better results > > when the corpus is large enough that the term frequency in a document, > and > > the document frequency in the corpus, can vary considerably across a > broad > > range of values.Maybe you could even say TFIDF and BM25 are the same > > equation except BM25 has a few additional hyperparameters (b and k). > > > > So is the advantage that BM25 provides for large diverse corpora with it? > > or is it marginal? Perhaps you can point me to some more examples where > > TFIDF is used (in supervised setting preferably) and I can plug in BM25 > in > > place of TFIDF and see how it compares. Here are some I found: > > > > > > > http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_da… > > *(supervised)* > > > > > http://scikit-learn.org/stable/auto_examples/text/document_clustering.html#… > > (*unsupervised)* > > > > Thank you! > > Basil > > > > PS: By the way, I'm not familiar with the delta-idf transform that Pavel > > mentions in the archive you linked, I'll have to delve deeper into that. > I > > agree with the response to Pavel that he should be putting it in a > separate > > class, not adding on to the TFIDF. I think it would take me about 6-8 > weeks > > to adapt my code to the fit transform model and submit a pull request. > > > > > > > > > > > > > > _______________________________________________ > > scikit-learn mailing list > > scikit-learn(a)python.org > > https://mail.python.org/mailman/listinfo/scikit-learn > > > -- > Pavel SORIANO > > PhD Student > ERIC Laboratory > Universit? de Lyon >

3 2

Re: [scikit-learn] Adding BM25 relevance function
by Basil Beirouti June 14, 2016

June 14, 2016

Hi Joel, Thanks for your response and for digging up that archived thread, it gives me a lot of clarity. I see your point about BM25, but I think in most cases where TFIDF makes sense, BM25 makes sense as well, but it could be "overkill". Consider that TFIDF does not produce normalized results either <http://scikit-learn.org/stable/auto_examples/text/document_clustering.html#…>, If BM25 requires dimensionality reduction (eg. using LSA) , so too would TFIDF. The term-document matrix is the same size no matter which weighting scheme is used. The only difference is that BM25 produces better results when the corpus is large enough that the term frequency in a document, and the document frequency in the corpus, can vary considerably across a broad range of values.Maybe you could even say TFIDF and BM25 are the same equation except BM25 has a few additional hyperparameters (b and k). So is the advantage that BM25 provides for large diverse corpora with it? or is it marginal? Perhaps you can point me to some more examples where TFIDF is used (in supervised setting preferably) and I can plug in BM25 in place of TFIDF and see how it compares. Here are some I found: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_da… *(supervised)* http://scikit-learn.org/stable/auto_examples/text/document_clustering.html#… (*unsupervised)* Thank you! Basil PS: By the way, I'm not familiar with the delta-idf transform that Pavel mentions in the archive you linked, I'll have to delve deeper into that. I agree with the response to Pavel that he should be putting it in a separate class, not adding on to the TFIDF. I think it would take me about 6-8 weeks to adapt my code to the fit transform model and submit a pull request.

2 1

Re: [scikit-learn] Adding BM25 relevance function to sklearn.feature_extraction.text
by Basil Beirouti June 14, 2016

June 14, 2016

Hello all, You can use sklearn.feature_extraction.text.TfidfVectorizer to learn a corpus of documents and rank them in order of relevance to a new previously unseen query. BM25 works in a similar manner to TfidfVectorizer, but is more complex and considered one of the most successful information retrieval algorithms. I currently have code that implements BM25 quite efficiently to learn a corpus of documents and I want to modify/port it to align with the fit-transform framework of sklearn. I think it could fit neatly into the current codebase. Questions: 1.) Would this be a desirable feature? 2.) Any advice for how to proceed with this? Things to watch out for? Any and all advice is welcome. Thanks! Basil

2 1