{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/problems-with-evaluation-of-word-embeddings","title":"Problems With Evaluation of Word Embeddings Using Word Similarity Tasks","arxiv_id":"1605.02276","date":"2016-05-08","proceeding":"WS 2016 8","authors":["Manaal Faruqui","Yulia Tsvetkov","Pushpendre Rastogi","Chris Dyer"],"abstract":"Lacking standardized extrinsic evaluation methods for vector representations\nof words, the NLP community has relied heavily on word similarity tasks as a\nproxy for intrinsic evaluation of word vectors. Word similarity evaluation,\nwhich correlates the distance between vectors and human judgments of semantic\nsimilarity is attractive, because it is computationally inexpensive and fast.\nIn this paper we present several problems associated with the evaluation of\nword vectors on word similarity datasets, and summarize existing solutions. Our\nstudy suggests that the use of word similarity tasks for evaluation of word\nvectors is not sustainable and calls for further research on evaluation\nmethods.","url_abs":"http://arxiv.org/abs/1605.02276v3","url_pdf":"http://arxiv.org/pdf/1605.02276v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"problems-with-evaluation-of-word-embeddings","repo_url":"https://github.com/avi-jit/SWOW-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"},{"task_slug":"word-similarity","task_name":"Word Similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1605.02276","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}