{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-does-bert-capture-semantics-a-closer-look","title":"How does BERT capture semantics? A closer look at polysemous words","arxiv_id":null,"date":"2020-11-01","proceeding":"EMNLP (BlackboxNLP) 2020 11","authors":["David Yenicelik","Florian Schmidt","Yannic Kilcher"],"abstract":"The recent paradigm shift to contextual word embeddings has seen tremendous success across a wide range of down-stream tasks. However, little is known on how the emergent relation of context and semantics manifests geometrically. We investigate polysemous words as one particularly prominent instance of semantic organization. Our rigorous quantitative analysis of linear separability and cluster organization in embedding vectors produced by BERT shows that semantics do not surface as isolated clusters but form seamless structures, tightly coupled with sentiment and syntax.","url_abs":"https://aclanthology.org/2020.blackboxnlp-1.15","url_pdf":"https://aclanthology.org/2020.blackboxnlp-1.15.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-does-bert-capture-semantics-a-closer-look","repo_url":"https://github.com/yenicelik/masterthesis/tree/clusterability/src","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semanticity-prediction","task_name":"Semanticity prediction"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"},{"task_slug":"word-sense-induction","task_name":"Word Sense Induction"},{"task_slug":"word-similarity","task_name":"Word Similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}