[scikit-learn] Decoding Differences Between SKL SVM and Matlab Libsvm Even When Parameters the Same

[Taylor, Johnmark]

Hello,

I am moving much of my neuroimaging coding over to Python from Matlab and
so I am switching from using libsvm in Matlab to using Scikit-learn SVM in
Python. Just to make sure I am not changing anything substantive about my
analyses, I am experimenting with the two implementations and trying to see
whether I can get them to yield identical results.

In Python I am using:

clf = svm.SVC(kernel='linear',C=1,probability=True)

In Matlab (libsvm) I am using:

clf = libsvmtrain(svm_training_labels,svm_training_vectors,['-t 0 -b 1 -c 1'])

When I run the SVM using these two different ways using simulated data, I
get subtly different results, even though I have fixed all of the
parameters of the SVMs to be the same using input arguments (linear
classifier, C=1, use probability estimates), and even though all the other
default parameters seem to be the same across these functions (tolerance =
.001, both using shrinking heuristics by default).

To give more details regarding the simulations:

One simulation I ran was designed to be absurdly difficult--it yielded 40%
accuracy for Matlab libsvm, and 44% accuracy for scikit-learn svm (binary
classification, chance = 50%). In this simulation, the two SVMs agreed in
their predictions only 18% of the time (in other words, they were both not
only guessing below chance, but they nearly always gave opposite guesses
compared to each other).

The other simulation was easier, yielding 68% accuracy for Matlab libsvm,
and 67% accuracy for scikit-learn SVM. In this simulation, the two SVMs
agreed in their predictions 97% of the time. So even though they often got
it wrong, they tended to make the same wrong guesses.

Any idea of what could possibly be leading to differences in the results?
My understanding is that SKL uses libsvm under the hood, so it's a been
confusing why the decoders are behaving differently. Both analyses are
being run on the same computer (Linux OS).

Thank you very much,

JohnMark Taylor

PhD Student, Harvard Vision Sciences Lab
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20160622/e6f04237/attachment.html>