<div dir="ltr"><div>In other words, according to that arrangement, is scikit-learn implying that section 1.17 is the best regressor out of the listed, 1.1 to 1.17?</div><div><br></div><div>If yes, I'd like to know if i can run the model on standard pc cpu ram and should still expect good results or do I need cloud hardware?</div><div><br></div><div>If I should expect good results on a pc, scikit says that needing gpu power is obsolete, since certain scikit models perform better (than ml designed for gpu) that are not designed for gpu, for that reason. Is this true? How much hardware is a practical expectation for running the best scikit models and getting the best results?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 11, 2019 at 9:02 AM <<a href="mailto:scikit-learn-request@python.org">scikit-learn-request@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send scikit-learn mailing list submissions to<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:scikit-learn-owner@python.org" target="_blank">scikit-learn-owner@python.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of scikit-learn digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: logistic regression results are not stable between<br>
solvers (Andreas Mueller)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 11 Oct 2019 15:42:58 +0200<br>
From: Andreas Mueller <<a href="mailto:t3kcit@gmail.com" target="_blank">t3kcit@gmail.com</a>><br>
To: <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
Subject: Re: [scikit-learn] logistic regression results are not stable<br>
between solvers<br>
Message-ID: <<a href="mailto:d55949d6-3355-f892-f6b3-030edf1c7947@gmail.com" target="_blank">d55949d6-3355-f892-f6b3-030edf1c7947@gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"; Format="flowed"<br>
<br>
<br>
<br>
On 10/10/19 1:14 PM, Beno?t Presles wrote:<br>
><br>
> Thanks for your answers.<br>
><br>
> On my real data, I do not have so many samples. I have a bit more than <br>
> 200 samples in total and I also would like to get some results with <br>
> unpenalized logisitic regression.<br>
> What do you suggest? Should I switch to the lbfgs solver?<br>
Yes.<br>
> Am I sure that with this solver I will not have any convergence issue <br>
> and always get the good result? Indeed, I did not get any convergence <br>
> warning with saga, so I thought everything was fine. I noticed some <br>
> issues only when I decided to test several solvers. Without comparing <br>
> the results across solvers, how to be sure that the optimisation goes <br>
> well? Shouldn't scikit-learn warn the user somehow if it is not the case?<br>
We should attempt to warn in the SAGA solver if it doesn't converge. <br>
That it doesn't raise a convergence warning should probably be <br>
considered a bug.<br>
It uses the maximum weight change as a stopping criterion right now.<br>
We could probably compute the dual objective once in the end to see if <br>
we converged, right? Or is that not possible with SAGA? If not, we might <br>
want to caution that no convergence warning will be raised.<br>
<br>
><br>
> At last, I was using saga because I also wanted to do some feature <br>
> selection by using l1 penalty which is not supported by lbfgs...<br>
You can use liblinear then.<br>
<br>
<br>
><br>
> Best regards,<br>
> Ben<br>
><br>
><br>
> Le 09/10/2019 ? 23:39, Guillaume Lema?tre a ?crit?:<br>
>> Ups I did not see the answer of Roman. Sorry about that. It is coming <br>
>> back to the same conclusion :)<br>
>><br>
>> On Wed, 9 Oct 2019 at 23:37, Guillaume Lema?tre <br>
>> <<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a> <mailto:<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>>> wrote:<br>
>><br>
>> Uhm actually increasing to 10000 samples solve the convergence issue.<br>
>> SAGA is not designed to work with a so small sample size most<br>
>> probably.<br>
>><br>
>> On Wed, 9 Oct 2019 at 23:36, Guillaume Lema?tre<br>
>> <<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a> <mailto:<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>>> wrote:<br>
>><br>
>> I slightly change the bench such that it uses pipeline and<br>
>> plotted the coefficient:<br>
>><br>
>> <a href="https://gist.github.com/glemaitre/8fcc24bdfc7dc38ca0c09c56e26b9386" rel="noreferrer" target="_blank">https://gist.github.com/glemaitre/8fcc24bdfc7dc38ca0c09c56e26b9386</a><br>
>><br>
>> I only see one of the 10 splits where SAGA is not converging,<br>
>> otherwise the coefficients<br>
>> look very close (I don't attach the figure here but they can<br>
>> be plotted using the snippet).<br>
>> So apart from this second split, the other differences seems<br>
>> to be numerical instability.<br>
>><br>
>> Where I have some concern is regarding the convergence rate<br>
>> of SAGA but I have no<br>
>> intuition to know if this is normal or not.<br>
>><br>
>> On Wed, 9 Oct 2019 at 23:22, Roman Yurchak<br>
>> <<a href="mailto:rth.yurchak@gmail.com" target="_blank">rth.yurchak@gmail.com</a> <mailto:<a href="mailto:rth.yurchak@gmail.com" target="_blank">rth.yurchak@gmail.com</a>>> wrote:<br>
>><br>
>> Ben,<br>
>><br>
>> I can confirm your results with penalty='none' and C=1e9.<br>
>> In both cases,<br>
>> you are running a mostly unpenalized logisitic<br>
>> regression. Usually<br>
>> that's less numerically stable than with a small<br>
>> regularization,<br>
>> depending on the data collinearity.<br>
>><br>
>> Running that same code with<br>
>> ? - larger penalty ( smaller C values)<br>
>> ? - or larger number of samples<br>
>> ? yields for me the same coefficients (up to some tolerance).<br>
>><br>
>> You can also see that SAGA convergence is not good by the<br>
>> fact that it<br>
>> needs 196000 epochs/iterations to converge.<br>
>><br>
>> Actually, I have often seen convergence issues with SAG<br>
>> on small<br>
>> datasets (in unit tests), not fully sure why.<br>
>><br>
>> -- <br>
>> Roman<br>
>><br>
>> On 09/10/2019 22:10, serafim loukas wrote:<br>
>> > The predictions across solver are exactly the same when<br>
>> I run the code.<br>
>> > I am using 0.21.3 version. What is yours?<br>
>> ><br>
>> ><br>
>> > In [13]: import sklearn<br>
>> ><br>
>> > In [14]: sklearn.__version__<br>
>> > Out[14]: '0.21.3'<br>
>> ><br>
>> ><br>
>> > Serafeim<br>
>> ><br>
>> ><br>
>> ><br>
>> >> On 9 Oct 2019, at 21:44, Beno?t Presles<br>
>> <<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a><br>
>> <mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a>><br>
>> >> <mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a><br>
>> <mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a>>>> wrote:<br>
>> >><br>
>> >> (y_pred_lbfgs==y_pred_saga).all() == False<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > scikit-learn mailing list<br>
>> > <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a> <mailto:<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
>> > <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
>> ><br>
>><br>
>> _______________________________________________<br>
>> scikit-learn mailing list<br>
>> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a> <mailto:<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
>> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
>><br>
>><br>
>><br>
>> -- <br>
>> Guillaume Lemaitre<br>
>> Scikit-learn @ Inria Foundation<br>
>> <a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
>><br>
>><br>
>><br>
>> -- <br>
>> Guillaume Lemaitre<br>
>> Scikit-learn @ Inria Foundation<br>
>> <a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
>><br>
>><br>
>><br>
>> -- <br>
>> Guillaume Lemaitre<br>
>> Scikit-learn @ Inria Foundation<br>
>> <a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
>><br>
>> _______________________________________________<br>
>> scikit-learn mailing list<br>
>> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
>> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
> _______________________________________________<br>
> scikit-learn mailing list<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mail.python.org/pipermail/scikit-learn/attachments/20191011/a7052cd9/attachment-0001.html" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/scikit-learn/attachments/20191011/a7052cd9/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br>
<br>
------------------------------<br>
<br>
End of scikit-learn Digest, Vol 43, Issue 21<br>
********************************************<br>
</blockquote></div></div>