{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zipfian-whitening","title":"Zipfian Whitening","arxiv_id":"2411.00680","date":"2024-11-01","proceeding":null,"authors":["Sho Yokoi","Han Bao","Hiroto Kurita","Hidetoshi Shimodaira"],"abstract":"The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the symmetry of an embedding space implicitly assume that the word frequencies are uniform; in reality, word frequencies follow a highly non-uniform distribution, known as Zipf's law. Surprisingly, simply performing PCA whitening weighted by the empirical word frequency that follows Zipf's law significantly improves task performance, surpassing established baselines. From a theoretical perspective, both our approach and existing methods can be clearly categorized: word representations are distributed according to an exponential family with either uniform or Zipfian base measures. By adopting the latter approach, we can naturally emphasize informative low-frequency words in terms of their vector norm, which becomes evident from the information-geometric perspective, and in terms of the loss functions for imbalanced classification. Additionally, our theory corroborates that popular natural language processing methods, such as skip-gram negative sampling, WhiteningBERT, and headless language models, work well just because their word embeddings encode the empirical word frequency into the underlying probabilistic model.","url_abs":"https://arxiv.org/abs/2411.00680v1","url_pdf":"https://arxiv.org/pdf/2411.00680v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zipfian-whitening","repo_url":"https://github.com/cl-tohoku/zipfian-whitening","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"imbalanced-classification","task_name":"imbalanced classification"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"pca","method_name":"PCA"},{"method_slug":"pca-whitening","method_name":"PCA Whitening"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.00680","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.00680"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cl-tohoku/zipfian-whitening","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/princeton-nlp/SimCSE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/PrincetonML/SIF","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/embeddings-benchmark/mteb","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"b9edc5a66c170732","entry":"get_diag_of_cov","repo":"cl-tohoku/zipfian-whitening","repo_kind":"official","path":"src/IsoScore.py","file_url":"https://github.com/cl-tohoku/zipfian-whitening/blob/HEAD/src/IsoScore.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9edc5a66c170732"}},{"code_sha256_prefix":"de1b89dc518ebc26","entry":"cos_sim","repo":"cl-tohoku/zipfian-whitening","repo_kind":"official","path":"src/calc_isotropy_score.py","file_url":"https://github.com/cl-tohoku/zipfian-whitening/blob/HEAD/src/calc_isotropy_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de1b89dc518ebc26"}},{"code_sha256_prefix":"8b4fb7a187d33e21","entry":"normalize_diagonal","repo":"cl-tohoku/zipfian-whitening","repo_kind":"official","path":"src/IsoScore.py","file_url":"https://github.com/cl-tohoku/zipfian-whitening/blob/HEAD/src/IsoScore.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b4fb7a187d33e21"}},{"code_sha256_prefix":"5e0a2765ed1280c0","entry":"pca_normalization","repo":"cl-tohoku/zipfian-whitening","repo_kind":"official","path":"src/IsoScore.py","file_url":"https://github.com/cl-tohoku/zipfian-whitening/blob/HEAD/src/IsoScore.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e0a2765ed1280c0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}