[scikit-learn] why the modification in the df-idf formula?
Sebastian Raschka
mail at sebastianraschka.com
Tue May 28 10:52:48 EDT 2024
Hi Sole,
It’s been a long time, but I remember helping with drafting the Tf-idf text in the documentation as part of a scikit-learn sprint at SciPy a looong time ago where I mentioned this difference (since it initially surprised me, because I couldn’t get it to match my from-scratch implementation). As far as I remember, the sklearn version addressed some instability issues for certain edge cases.
I am not sure if that helps, but I have briefly compared the textbook vs the sklearn tf-idf here: https://github.com/rasbt/machine-learning-book/blob/main/ch08/ch08.ipynb
Best,
Sebastian
--
Sebastian Raschka, PhD
Machine learning and AI researcher, https://sebastianraschka.com
Staff Research Engineer at Lightning AI, https://lightning.ai
On May 28, 2024 at 9:43 AM -0500, Sole Galli via scikit-learn <scikit-learn at python.org>, wrote:
> Hi guys,
>
> I'd like to understand why sklearn's implementation of tf-idf is different from the standard textbook notation as described in the docs: https://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting
>
> Do you have any reference that I could take a look at? I didn't manage to find them in the docs, maybe I missed something?
>
> Thank you!
>
> Best wishes
> Sole
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/scikit-learn/attachments/20240528/b91277f4/attachment.html>
More information about the scikit-learn
mailing list