{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-word2vec-vis-w2v-learning-visually","title":"Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes","arxiv_id":"1511.07067","date":"2015-11-22","proceeding":"CVPR 2016 6","authors":["Satwik Kottur","Ramakrishna Vedantam","José M. F. Moura","Devi Parikh"],"abstract":"We propose a model to learn visually grounded word embeddings (vis-w2v) to\ncapture visual notions of semantic relatedness. While word embeddings trained\nusing text have been extremely successful, they cannot uncover notions of\nsemantic relatedness implicit in our visual world. For instance, although\n\"eats\" and \"stares at\" seem unrelated in text, they share semantics visually.\nWhen people are eating something, they also tend to stare at the food.\nGrounding diverse relations like \"eats\" and \"stares at\" into vision remains\nchallenging, despite recent progress in vision. We note that the visual\ngrounding of words depends on semantics, and not the literal pixels. We thus\nuse abstract scenes created from clipart to provide the visual grounding. We\nfind that the embeddings we learn capture fine-grained, visually grounded\nnotions of semantic relatedness. We show improvements over text-only word\nembeddings (word2vec) on three tasks: common-sense assertion classification,\nvisual paraphrasing and text-based image retrieval. Our code and datasets are\navailable online.","url_abs":"http://arxiv.org/abs/1511.07067v2","url_pdf":"http://arxiv.org/pdf/1511.07067v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-word2vec-vis-w2v-learning-visually","repo_url":"https://github.com/satwikkottur/VisualWord2Vec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.07067","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}