<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>My comments are at the end as some people do not like top posts.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> scikit-learn <scikit-learn-bounces+avigross=verizon.net@python.org> <b>On Behalf Of </b>Matti Viljamaa<br><b>Sent:</b> Friday, January 25, 2019 3:31 PM<br><b>To:</b> Scikit-learn mailing list <scikit-learn@python.org><br><b>Subject:</b> Re: [scikit-learn] How to determine suitable cluster algo<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=FI>Also,<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Remember that some algos may exhibit “sweet spots” w.r.t. computation time and gained accuracy.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So you might want to keep measuring “explained variance”, while you add complexity to your models. And then do plots of model complexity vs explained variance.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>E.g. in MLPClassifier you’d plot e.g. hidden layers against explained variance to figure out where adding hidden layers starts to exhibit lesser gain in explained variance.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Lähetetty Windows 10:n <a href="https://go.microsoft.com/fwlink/?LinkId=550986">Sähköposti</a>sta<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>Lähettäjä: </b><a href="mailto:matti.v.viljamaa@gmail.com">Matti Viljamaa</a><br><b>Lähetetty: </b>Friday, 25 January 2019 13.43<br><b>Vastaanottaja: </b><a href="mailto:scikit-learn@python.org">Scikit-learn mailing list</a><br><b>Aihe: </b>VS: [scikit-learn] How to determine suitable cluster algo<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=FI>For determining what one can afford computaionally, see e.g.:<o:p></o:p></span></p><p class=MsoNormal><span lang=FI>https://stackoverflow.com/questions/22443041/predicting-how-long-an-scikit-learn-classification-will-take-to-run<o:p></o:p></span></p><p class=MsoNormal><span lang=FI><a href="https://www.redditcom/r/scikit_learn/comments/a746h0/is_there_any_way_to_estimate_how_long_a_given/">https://www.reddit.com/r/scikit_learn/comments/a746h0/is_there_any_way_to_estimate_how_long_a_given/</a><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Lähetetty Windows 10:n <a href="https://go.microsoft.com/fwlink/?LinkId=550986">Sähköposti</a>sta<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>Lähettäjä: </b><a href="mailto:pahome.chen@mirlab.org">lampahome</a><br><b>Lähetetty: </b>Friday, 25 January 2019 3.42<br><b>Vastaanottaja: </b><a href="mailto:scikit-learn@python.org">Scikit-learn mailing list</a><br><b>Aihe: </b>Re: [scikit-learn] How to determine suitable cluster algo<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Maybe the suitable way is try-and-error?<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>What I'm interesting is that my datasets is very huge and I can't try number of cluster from 1 to N if I have N samples<o:p></o:p></p></div><div><p class=MsoNormal>That cost too much time for me.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Maybe I should define the initial number of cluster based on execution time?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Then analyze the next step is increase/decrease the number of cluster?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal>thx<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>__COMMENT__<o:p></o:p></p><p class=MsoNormal>This is a question, not a suggestion.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The poster suggested they have such a large amount of data that looking for larger numbers of clusters to find a ‘sweet’ spot may take too much time.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Is there any value in taking a much smaller random sample of data that remains big enough and trying that on a reasonable range of clusters? The results would not be definitive but might supply a clue as to what range to try again with the full data.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>As I see mentioned, the run time may not be going up if the data is constant and the number of clusters varies. I am not sure what clustering algorithms you want to use but for something like K-means with reasonable data, generally the number of clusters that show meaningful results are usually much smaller than the number of items in the data. The algorithms often terminate when successive runs show little change. This may likely be a tunable parameter. So if you ask it to make N+1 clusters it may even terminate sooner than for N if it is because that number of clusters more closely resembles the variation in the data. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>And, again, if you are using a K-means variant, it may be better to use some human intervention to see if a particular level of clustering fits some model you can make that explains what each cluster has in common. If you overfit, the number of clusters can effectively be the number of unique items in your data and probably has no meaningful purpose.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Again, just a question. There are algorithms out there that deal better with large data than others.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Avi<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>