<div><div><div dir="auto">Hi Jake,</div><div dir="auto"><br></div><div dir="auto">Thanks for the quick reply.</div><div dir="auto"><br></div><div dir="auto">What I meant is different from the TfIdfVectorizer. Let me clarify:</div><div dir="auto"><br></div><div dir="auto">In the TfIdfVectorizer, the raw counts are multiplied by IDF, which badically means normalizing the counts by document frequencies, tf * idf.</div><div dir="auto">But still, tf is deined here as the raw count of a term in the dicument.</div><div dir="auto"><br></div><div dir="auto">What I am suggesting, is to add the possibility to use another definition of tf, tf= relative frequency of a term in a document = raw counts / document length.</div><div dir="auto">On top of this, one could further normalize by IDF to get the TF-IDF (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency_2">https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency_2</a>).</div></div></div><div dir="auto"><br></div><div dir="auto">When can this be useful? Here is an example:</div><div dir="auto">Say term t occurs 5 times in document d1, and also 5 times in document d2.</div><div dir="auto">At first glance, it seems that the term conveys the same information about both documents. But if we also check document lengths, and find that length of d1 is 20, wheras lenght of d2 is 200, then probably the “importance” and information carried by the same term in the two documents is not the same.</div><div dir="auto">If we use relative frequency instead of absolute counts, then tf1=5/20=0.4 whereas tf2=5/200=0.04.</div><div dir="auto"><br></div><div dir="auto">There are many practical cases (document similarity, document classification, etc...) where using relative frequencies yields better results, and it might be worth making the CountVectorizer support this.</div><div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Yacine.</div><div><div><br><div class="gmail_quote"><div>On Sun, Jan 28, 2018 at 15:12 Jacob Vanderplas <<a href="mailto:jakevdp@cs.washington.edu" target="_blank">jakevdp@cs.washington.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi Yacine,<div>If I'm understanding you correctly, I think what you have in mind is already implemented in scikit-learn in the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html" target="_blank">TF-IDF vectorizer</a>.</div><div><br></div><div>Best,</div><div> Jake</div></div><div class="gmail_extra"><br clear="all"><div><div class="m_6891695652684885055m_3723827363691810006gmail_signature" data-smartmail="gmail_signature"><div><div><div><div> <font size="1">Jake VanderPlas</font></div><div><font size="1"> Senior Data Science Fellow</font></div><div><span style="font-size:x-small"> Director of Open Software</span><div><div><font size="1"> </font><span style="font-size:x-small">University of Washington </span><span style="font-size:x-small">eScience Institute<br></span></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote"></div></div><div class="gmail_extra"><div class="gmail_quote">On Sat, Jan 27, 2018 at 9:59 PM, Yacine MAZARI <span><<a href="mailto:y.mazari@gmail.com" target="_blank">y.mazari@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div><div><div><div><div><div><div>Hello,<br><br></div>I would like to work on adding an additional feature to "sklearn.feature_extraction.text.CountVectorizer".<br><br></div>In the current implementation, the definition of term frequency is the number of times a term t occurs in document d.<br></div><br>However, another definition that is very commonly used in practice is the <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency_2" target="_blank">term frequency adjusted for document length</a>, i.e: tf = raw counts / document length.<br><br></div>I intend to implement this by adding an additional boolean parameter "relative_frequency" to the constructor of CountVectorizer.<br></div>If the parameter is true, normalize X by document length (along x=1) in "CountVectorizer.fit_transform()".<br><br></div>What do you think?<br></div>If this sounds reasonable an worth it, I will send a PR.<br><br></div>Thank you,<br></div>Yacine.<br></div>
<br></blockquote></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br></blockquote></div><br></div>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
</blockquote></div></div></div>