{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-word-significance-using-distributed","title":"Measuring Word Significance using Distributed Representations of Words","arxiv_id":"1508.02297","date":"2015-08-10","proceeding":null,"authors":["Adriaan M. J. Schakel","Benjamin J. Wilson"],"abstract":"Distributed representations of words as real-valued vectors in a relatively\nlow-dimensional space aim at extracting syntactic and semantic features from\nlarge text corpora. A recently introduced neural network, named word2vec\n(Mikolov et al., 2013a; Mikolov et al., 2013b), was shown to encode semantic\ninformation in the direction of the word vectors. In this brief report, it is\nproposed to use the length of the vectors, together with the term frequency, as\nmeasure of word significance in a corpus. Experimental evidence using a\ndomain-specific corpus of abstracts is presented to support this proposal. A\nuseful visualization technique for text corpora emerges, where words are mapped\nonto a two-dimensional plane and automatically ranked by significance.","url_abs":"http://arxiv.org/abs/1508.02297v1","url_pdf":"http://arxiv.org/pdf/1508.02297v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measuring-word-significance-using-distributed","repo_url":"https://github.com/vackosar/fasttext-vector-norms-and-oov-words","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1508.02297","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}