{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/similarity-encoding-for-learning-with-dirty","title":"Similarity encoding for learning with dirty categorical variables","arxiv_id":"1806.00979","date":"2018-06-04","proceeding":null,"authors":["Patricio Cerda","Gaël Varoquaux","Balázs Kégl"],"abstract":"For statistical learning, categorical variables in a table are usually\nconsidered as discrete entities and encoded separately to feature vectors,\ne.g., with one-hot encoding. \"Dirty\" non-curated data gives rise to categorical\nvariables with a very high cardinality but redundancy: several categories\nreflect the same entity. In databases, this issue is typically solved with a\ndeduplication step. We show that a simple approach that exposes the redundancy\nto the learning algorithm brings significant gains. We study a generalization\nof one-hot encoding, similarity encoding, that builds feature vectors from\nsimilarities across categories. We perform a thorough empirical validation on\nnon-curated tables, a problem seldom studied in machine learning. Results on\nseven real-world datasets show that similarity encoding brings significant\ngains in prediction in comparison with known encoding methods for categories or\nstrings, notably one-hot encoding and bag of character n-grams. We draw\npractical recommendations for encoding dirty categories: 3-gram similarity\nappears to be a good choice to capture morphological resemblance. For very\nhigh-cardinality, dimensionality reduction significantly reduces the\ncomputational cost with little loss in performance: random projections or\nchoosing a subset of prototype categories still outperforms classic encoding\napproaches.","url_abs":"http://arxiv.org/abs/1806.00979v1","url_pdf":"http://arxiv.org/pdf/1806.00979v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"similarity-encoding-for-learning-with-dirty","repo_url":"https://github.com/pcerda/ecml-pkdd-2018","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"similarity-encoding-for-learning-with-dirty","repo_url":"https://github.com/jorisvandenbossche/target-encoder-benchmarks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}