<div dir="ltr"><div dir="ltr">Please, respect and refinement when addressing the contributors and users of scikit-learn.</div><div dir="ltr"><br></div><div>Gael's statement is perfect -- complexity does not imply better prediction.</div><div>The choice of estimator (and algorithm) depends on the structure of the model desired for the data presented.</div><div>Estimator superiority cannot be proven in a context- and/or data-agnostic fashion.</div><div><br></div><div>J.B.<br></div><div dir="ltr"><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">2019年10月13日(日) 6:13 Mike Smith <<a href="mailto:javaeurusd@gmail.com">javaeurusd@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">"Second complexity does not<br>> imply better prediction. " <br><div><br></div><div>Complexity doesn't imply prediction? Perhaps you're having a translation error.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 12, 2019 at 2:04 PM <<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send scikit-learn mailing list submissions to<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:scikit-learn-owner@python.org" target="_blank">scikit-learn-owner@python.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of scikit-learn digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: scikit-learn Digest, Vol 43, Issue 24 (Mike Smith)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 12 Oct 2019 14:04:12 -0700<br>
From: Mike Smith <<a href="mailto:javaeurusd@gmail.com" target="_blank">javaeurusd@gmail.com</a>><br>
To: <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
Subject: Re: [scikit-learn] scikit-learn Digest, Vol 43, Issue 24<br>
Message-ID:<br>
<CAEWZffD-hNviFkyxuM8CgDR3XSWOyn=<a href="mailto:4LRy2NJvjwvVr4RgobQ@mail.gmail.com" target="_blank">4LRy2NJvjwvVr4RgobQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
"... > If I should expect good results on a pc, scikit says that needing<br>
gpu power is<br>
> obsolete, since certain scikit models perform better (than ml designed<br>
for gpu)<br>
> that are not designed for gpu, for that reason. Is this true?"<br>
<br>
Where do you see this written? I think that you are looking for overly<br>
simple stories that you are not true."<br>
<br>
Gael, see the below from the scikit-learn FAQ. You can also find this<br>
yourself at the main FAQ:<br>
<br>
[image: 2019-10-12 14_00_05-Frequently Asked Questions ? scikit-learn<br>
0.21.3 documentation.png]<br>
<br>
<br>
On Sat, Oct 12, 2019 at 9:03 AM <<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a>> wrote:<br>
<br>
> Send scikit-learn mailing list submissions to<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
><br>
> To subscribe or unsubscribe via the World Wide Web, visit<br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
> or, via email, send a message with subject or body 'help' to<br>
> <a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a><br>
><br>
> You can reach the person managing the list at<br>
> <a href="mailto:scikit-learn-owner@python.org" target="_blank">scikit-learn-owner@python.org</a><br>
><br>
> When replying, please edit your Subject line so it is more specific<br>
> than "Re: Contents of scikit-learn digest..."<br>
><br>
><br>
> Today's Topics:<br>
><br>
> 1. Re: Is scikit-learn implying neural nets are the best<br>
> regressor? (Gael Varoquaux)<br>
><br>
><br>
> ----------------------------------------------------------------------<br>
><br>
> Message: 1<br>
> Date: Fri, 11 Oct 2019 13:34:33 -0400<br>
> From: Gael Varoquaux <<a href="mailto:gael.varoquaux@normalesup.org" target="_blank">gael.varoquaux@normalesup.org</a>><br>
> To: Scikit-learn mailing list <<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
> Subject: Re: [scikit-learn] Is scikit-learn implying neural nets are<br>
> the best regressor?<br>
> Message-ID: <<a href="mailto:20191011173433.bbywiqnwjjpvsi4r@phare.normalesup.org" target="_blank">20191011173433.bbywiqnwjjpvsi4r@phare.normalesup.org</a>><br>
> Content-Type: text/plain; charset=iso-8859-1<br>
><br>
> On Fri, Oct 11, 2019 at 10:10:32AM -0700, Mike Smith wrote:<br>
> > In other words, according to that arrangement, is scikit-learn implying<br>
> that<br>
> > section 1.17 is the best regressor out of the listed, 1.1 to 1.17?<br>
><br>
> No.<br>
><br>
> First they are not ordered in order of complexity (Naive Bayes is<br>
> arguably simpler than Gaussian Processes). Second complexity does not<br>
> imply better prediction.<br>
><br>
> > If I should expect good results on a pc, scikit says that needing gpu<br>
> power is<br>
> > obsolete, since certain scikit models perform better (than ml designed<br>
> for gpu)<br>
> > that are not designed for gpu, for that reason. Is this true?<br>
><br>
> Where do you see this written? I think that you are looking for overly<br>
> simple stories that you are not true.<br>
><br>
> > How much hardware is a practical expectation for running the best<br>
> > scikit models and getting the best results?<br>
><br>
> This is too vague a question for which there is no answer.<br>
><br>
> Ga?l<br>
><br>
> > On Fri, Oct 11, 2019 at 9:02 AM <<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a>> wrote:<br>
><br>
> > Send scikit-learn mailing list submissions to<br>
> > ? ? ? ? <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
><br>
> > To subscribe or unsubscribe via the World Wide Web, visit<br>
> > ? ? ? ? <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
> > or, via email, send a message with subject or body 'help' to<br>
> > ? ? ? ? <a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a><br>
><br>
> > You can reach the person managing the list at<br>
> > ? ? ? ? <a href="mailto:scikit-learn-owner@python.org" target="_blank">scikit-learn-owner@python.org</a><br>
><br>
> > When replying, please edit your Subject line so it is more specific<br>
> > than "Re: Contents of scikit-learn digest..."<br>
><br>
><br>
> > Today's Topics:<br>
><br>
> > ? ?1. Re: logistic regression results are not stable between<br>
> > ? ? ? solvers (Andreas Mueller)<br>
><br>
><br>
> ><br>
> ----------------------------------------------------------------------<br>
><br>
> > Message: 1<br>
> > Date: Fri, 11 Oct 2019 15:42:58 +0200<br>
> > From: Andreas Mueller <<a href="mailto:t3kcit@gmail.com" target="_blank">t3kcit@gmail.com</a>><br>
> > To: <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> > Subject: Re: [scikit-learn] logistic regression results are not<br>
> stable<br>
> > ? ? ? ? between solvers<br>
> > Message-ID: <<a href="mailto:d55949d6-3355-f892-f6b3-030edf1c7947@gmail.com" target="_blank">d55949d6-3355-f892-f6b3-030edf1c7947@gmail.com</a>><br>
> > Content-Type: text/plain; charset="utf-8"; Format="flowed"<br>
><br>
><br>
><br>
> > On 10/10/19 1:14 PM, Beno?t Presles wrote:<br>
><br>
> > > Thanks for your answers.<br>
><br>
> > > On my real data, I do not have so many samples. I have a bit more<br>
> than<br>
> > > 200 samples in total and I also would like to get some results with<br>
> > > unpenalized logisitic regression.<br>
> > > What do you suggest? Should I switch to the lbfgs solver?<br>
> > Yes.<br>
> > > Am I sure that with this solver I will not have any convergence<br>
> issue<br>
> > > and always get the good result? Indeed, I did not get any<br>
> convergence<br>
> > > warning with saga, so I thought everything was fine. I noticed some<br>
> > > issues only when I decided to test several solvers. Without<br>
> comparing<br>
> > > the results across solvers, how to be sure that the optimisation<br>
> goes<br>
> > > well? Shouldn't scikit-learn warn the user somehow if it is not<br>
> the case?<br>
> > We should attempt to warn in the SAGA solver if it doesn't converge.<br>
> > That it doesn't raise a convergence warning should probably be<br>
> > considered a bug.<br>
> > It uses the maximum weight change as a stopping criterion right now.<br>
> > We could probably compute the dual objective once in the end to see<br>
> if<br>
> > we converged, right? Or is that not possible with SAGA? If not, we<br>
> might<br>
> > want to caution that no convergence warning will be raised.<br>
><br>
><br>
> > > At last, I was using saga because I also wanted to do some feature<br>
> > > selection by using l1 penalty which is not supported by lbfgs...<br>
> > You can use liblinear then.<br>
><br>
><br>
><br>
> > > Best regards,<br>
> > > Ben<br>
><br>
><br>
> > > Le 09/10/2019 ? 23:39, Guillaume Lema?tre a ?crit?:<br>
> > >> Ups I did not see the answer of Roman. Sorry about that. It is<br>
> coming<br>
> > >> back to the same conclusion :)<br>
><br>
> > >> On Wed, 9 Oct 2019 at 23:37, Guillaume Lema?tre<br>
> > >> <<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a> <mailto:<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>>> wrote:<br>
><br>
> > >>? ? ?Uhm actually increasing to 10000 samples solve the convergence<br>
> > issue.<br>
> > >>? ? ?SAGA is not designed to work with a so small sample size most<br>
> > >>? ? ?probably.<br>
><br>
> > >>? ? ?On Wed, 9 Oct 2019 at 23:36, Guillaume Lema?tre<br>
> > >>? ? ?<<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a> <mailto:<a href="mailto:g.lemaitre58@gmail.com" target="_blank">g.lemaitre58@gmail.com</a>>><br>
> wrote:<br>
><br>
> > >>? ? ? ? ?I slightly change the bench such that it uses pipeline and<br>
> > >>? ? ? ? ?plotted the coefficient:<br>
><br>
> > >>? ? ? ? ?<a href="https://gist.github.com/glemaitre/" rel="noreferrer" target="_blank">https://gist.github.com/glemaitre/</a><br>
> > 8fcc24bdfc7dc38ca0c09c56e26b9386<br>
><br>
> > >>? ? ? ? ?I only see one of the 10 splits where SAGA is not<br>
> converging,<br>
> > >>? ? ? ? ?otherwise the coefficients<br>
> > >>? ? ? ? ?look very close (I don't attach the figure here but they<br>
> can<br>
> > >>? ? ? ? ?be plotted using the snippet).<br>
> > >>? ? ? ? ?So apart from this second split, the other differences<br>
> seems<br>
> > >>? ? ? ? ?to be numerical instability.<br>
><br>
> > >>? ? ? ? ?Where I have some concern is regarding the convergence<br>
> rate<br>
> > >>? ? ? ? ?of SAGA but I have no<br>
> > >>? ? ? ? ?intuition to know if this is normal or not.<br>
><br>
> > >>? ? ? ? ?On Wed, 9 Oct 2019 at 23:22, Roman Yurchak<br>
> > >>? ? ? ? ?<<a href="mailto:rth.yurchak@gmail.com" target="_blank">rth.yurchak@gmail.com</a> <mailto:<a href="mailto:rth.yurchak@gmail.com" target="_blank">rth.yurchak@gmail.com</a>>><br>
> wrote:<br>
><br>
> > >>? ? ? ? ? ? ?Ben,<br>
><br>
> > >>? ? ? ? ? ? ?I can confirm your results with penalty='none' and<br>
> C=1e9.<br>
> > >>? ? ? ? ? ? ?In both cases,<br>
> > >>? ? ? ? ? ? ?you are running a mostly unpenalized logisitic<br>
> > >>? ? ? ? ? ? ?regression. Usually<br>
> > >>? ? ? ? ? ? ?that's less numerically stable than with a small<br>
> > >>? ? ? ? ? ? ?regularization,<br>
> > >>? ? ? ? ? ? ?depending on the data collinearity.<br>
><br>
> > >>? ? ? ? ? ? ?Running that same code with<br>
> > >>? ? ? ? ? ? ?? - larger penalty ( smaller C values)<br>
> > >>? ? ? ? ? ? ?? - or larger number of samples<br>
> > >>? ? ? ? ? ? ?? yields for me the same coefficients (up to some<br>
> > tolerance).<br>
><br>
> > >>? ? ? ? ? ? ?You can also see that SAGA convergence is not good by<br>
> the<br>
> > >>? ? ? ? ? ? ?fact that it<br>
> > >>? ? ? ? ? ? ?needs 196000 epochs/iterations to converge.<br>
><br>
> > >>? ? ? ? ? ? ?Actually, I have often seen convergence issues with<br>
> SAG<br>
> > >>? ? ? ? ? ? ?on small<br>
> > >>? ? ? ? ? ? ?datasets (in unit tests), not fully sure why.<br>
><br>
> > >>? ? ? ? ? ? ?--<br>
> > >>? ? ? ? ? ? ?Roman<br>
><br>
> > >>? ? ? ? ? ? ?On 09/10/2019 22:10, serafim loukas wrote:<br>
> > >>? ? ? ? ? ? ?> The predictions across solver are exactly the same<br>
> when<br>
> > >>? ? ? ? ? ? ?I run the code.<br>
> > >>? ? ? ? ? ? ?> I am using 0.21.3 version. What is yours?<br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?> In [13]: import sklearn<br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?> In [14]: sklearn.__version__<br>
> > >>? ? ? ? ? ? ?> Out[14]: '0.21.3'<br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?> Serafeim<br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?>> On 9 Oct 2019, at 21:44, Beno?t Presles<br>
> > >>? ? ? ? ? ? ?<<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a><br>
> > >>? ? ? ? ? ? ?<mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a>><br>
> > >>? ? ? ? ? ? ?>> <mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a><br>
> > >>? ? ? ? ? ? ?<mailto:<a href="mailto:benoit.presles@u-bourgogne.fr" target="_blank">benoit.presles@u-bourgogne.fr</a>>>> wrote:<br>
> > >>? ? ? ? ? ? ?>><br>
> > >>? ? ? ? ? ? ?>> (y_pred_lbfgs==y_pred_saga).all() == False<br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?><br>
> > >>? ? ? ? ? ? ?> _______________________________________________<br>
> > >>? ? ? ? ? ? ?> scikit-learn mailing list<br>
> > >>? ? ? ? ? ? ?> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a> <mailto:<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
> > >>? ? ? ? ? ? ?><br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
> > >>? ? ? ? ? ? ?><br>
><br>
> > >>? ? ? ? ? ? ?_______________________________________________<br>
> > >>? ? ? ? ? ? ?scikit-learn mailing list<br>
> > >>? ? ? ? ? ? ?<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a> <mailto:<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
> > >>? ? ? ? ? ? ?<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
><br>
><br>
> > >>? ? ? ? ?--<br>
> > >>? ? ? ? ?Guillaume Lemaitre<br>
> > >>? ? ? ? ?Scikit-learn @ Inria Foundation<br>
> > >>? ? ? ? ?<a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
><br>
><br>
><br>
> > >>? ? ?--<br>
> > >>? ? ?Guillaume Lemaitre<br>
> > >>? ? ?Scikit-learn @ Inria Foundation<br>
> > >>? ? ?<a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
><br>
><br>
><br>
> > >> --<br>
> > >> Guillaume Lemaitre<br>
> > >> Scikit-learn @ Inria Foundation<br>
> > >> <a href="https://glemaitre.github.io/" rel="noreferrer" target="_blank">https://glemaitre.github.io/</a><br>
><br>
> > >> _______________________________________________<br>
> > >> scikit-learn mailing list<br>
> > >> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> > >> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
> > > _______________________________________________<br>
> > > scikit-learn mailing list<br>
> > > <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> > > <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
> > -------------- next part --------------<br>
> > An HTML attachment was scrubbed...<br>
> > URL: <<br>
> <a href="http://mail.python.org/pipermail/scikit-learn/attachments/20191011/" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/scikit-learn/attachments/20191011/</a><br>
> > a7052cd9/attachment-0001.html><br>
><br>
> > ------------------------------<br>
><br>
> > Subject: Digest Footer<br>
><br>
> > _______________________________________________<br>
> > scikit-learn mailing list<br>
> > <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> > <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
><br>
> > ------------------------------<br>
><br>
> > End of scikit-learn Digest, Vol 43, Issue 21<br>
> > ********************************************<br>
><br>
><br>
> > _______________________________________________<br>
> > scikit-learn mailing list<br>
> > <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> > <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
><br>
> --<br>
> Gael Varoquaux<br>
> Research Director, INRIA Visiting professor, McGill<br>
> <a href="http://gael-varoquaux.info" rel="noreferrer" target="_blank">http://gael-varoquaux.info</a> <a href="http://twitter.com/GaelVaroquaux" rel="noreferrer" target="_blank">http://twitter.com/GaelVaroquaux</a><br>
><br>
><br>
> ------------------------------<br>
><br>
> Subject: Digest Footer<br>
><br>
> _______________________________________________<br>
> scikit-learn mailing list<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
><br>
> ------------------------------<br>
><br>
> End of scikit-learn Digest, Vol 43, Issue 24<br>
> ********************************************<br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mail.python.org/pipermail/scikit-learn/attachments/20191012/6959d075/attachment.html" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/scikit-learn/attachments/20191012/6959d075/attachment.html</a>><br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: 2019-10-12 14_00_05-Frequently Asked Questions ? scikit-learn 0.21.3 documentation.png<br>
Type: image/png<br>
Size: 26245 bytes<br>
Desc: not available<br>
URL: <<a href="http://mail.python.org/pipermail/scikit-learn/attachments/20191012/6959d075/attachment.png" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/scikit-learn/attachments/20191012/6959d075/attachment.png</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br>
<br>
------------------------------<br>
<br>
End of scikit-learn Digest, Vol 43, Issue 25<br>
********************************************<br>
</blockquote></div>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
</blockquote></div></div>