Hello everyone, I am Sole, I started the conversation on feature engine <https://feature-engine.readthedocs.io>, a package I created for feature engineering. Regarding the grouping of *rare / infrequent* categories into an umbrella term like "Rare", "Other", etc, which Federico raised recently, I would like to provide some literature at the end of this email, that quotes the use of this procedure. These are a series of articles by the best solutions to the 2009 KDD annual competition, which were compiled into one "book <http://www.mtome.com/Publications/CiML/CiML-v3-book.pdf>", and I am sure you are aware of it already. I would also like to highlight that this is extremely common practice in the industry, not only to avoid overfitting, but also to handle unseen categories when models are deployed. It would be great to see this functionality added to both the OrdinalEncoder and the OneHotEncoder, with triggers on the representation of the label in the dataset (eg. percentage) Pointing to the main quotes from these articles <http://www.mtome.com/Publications/CiML/CiML-v3-book.pdf>: Page 4 of the summary and introductory article: "For categorical variables, grouping of under-represented categories proved to be useful to avoid overfitting. The winners of the fast and the slow track used similar strategies consisting in retaining the most populated categories and coarsely grouping the others in an unsupervised way" Page 23: "Most of the learning algorithms we were planning to use do not handle categorical variables, so we needed to recode them. This was done in a standard way, by generating indicator vari- ables for the different values a categorical attribute could take. The only slightly non-standard decision was to limit ourselves to encoding only the 10 most common values of each categorical attribute, rather than all the values, in order to avoid an explosion in the number of features from variables with a huge vocabulary" Page 36: "We consolidate the extremely low populated entries (having fewer than 200 examples) with their neighbors to smooth out the outliers. Similarly, we group some categorical variables which have a large number of entries ( > 1000 distinct values) into 100 categories." See bulletpoints in Page 47 I hope you find these useful. Let me know if / how I can help. Regards Sole On Fri, 19 Apr 2019 at 17:54, federico vaggi <vaggi.federico@gmail.com> wrote:
Hi everyone,
I wanted to use the scikit-learn transformer API to clean up some messy data as input to a neural network. One of the steps involves converting categorical variables (of very high cardinality) into integers for use in an embedding layer.
Unfortunately, I cannot quite use LabelEncoder to do solve this. When dealing with categorical variables with very high cardinality, I found it useful in practice to have a threshold value for the frequency under which a variable ends up with the 'unk' or 'rare' label. This same label would also end up applied at test time to entries that were not observed in the train set.
This is relatively straightforward to add to the existing label encoder code, but it breaks the contract slightly: if we encode some variables with a 'rare' label, then the transform operation is no longer a bijection.
Is this feature too niche for the main sklearn? I saw there was a package ( https://feature-engine.readthedocs.io/en/latest/RareLabelCategoricalEncoder....) that implemented a similar feature discussed in the mailing list. _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn