{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embedding-words-as-distributions-with-a","title":"Embedding Words as Distributions with a Bayesian Skip-gram Model","arxiv_id":"1711.11027","date":"2017-11-29","proceeding":"COLING 2018 8","authors":["Arthur Bražinskas","Serhii Havrylov","Ivan Titov"],"abstract":"We introduce a method for embedding words as probability densities in a\nlow-dimensional space. Rather than assuming that a word embedding is fixed\nacross the entire text collection, as in standard word embedding methods, in\nour Bayesian model we generate it from a word-specific prior density for each\noccurrence of a given word. Intuitively, for each word, the prior density\nencodes the distribution of its potential 'meanings'. These prior densities are\nconceptually similar to Gaussian embeddings. Interestingly, unlike the Gaussian\nembeddings, we can also obtain context-specific densities: they encode\nuncertainty about the sense of a word given its context and correspond to\nposterior distributions within our model. The context-dependent densities have\nmany potential applications: for example, we show that they can be directly\nused in the lexical substitution task. We describe an effective estimation\nmethod based on the variational autoencoding framework. We also demonstrate\nthat our embeddings achieve competitive results on standard benchmarks.","url_abs":"http://arxiv.org/abs/1711.11027v2","url_pdf":"http://arxiv.org/pdf/1711.11027v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embedding-words-as-distributions-with-a","repo_url":"https://github.com/ixlan/BSG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.11027","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}