{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/endowing-language-models-with-multimodal","title":"Endowing Language Models with Multimodal Knowledge Graph Representations","arxiv_id":"2206.13163","date":"2022-06-27","proceeding":null,"authors":["Ningyuan Huang","Yash R. Deshpande","Yibo Liu","Houda Alberts","Kyunghyun Cho","Clara Vania","Iacer Calixto"],"abstract":"We propose a method to make natural language understanding models more parameter efficient by storing knowledge in an external knowledge graph (KG) and retrieving from this KG using a dense index. Given (possibly multilingual) downstream task data, e.g., sentences in German, we retrieve entities from the KG and use their multimodal representations to improve downstream task performance. We use the recently released VisualSem KG as our external knowledge repository, which covers a subset of Wikipedia and WordNet entities, and compare a mix of tuple-based and graph-based algorithms to learn entity and relation representations that are grounded on the KG multimodal information. We demonstrate the usefulness of the learned entity representations on two downstream tasks, and show improved performance on the multilingual named entity recognition task by $0.3\\%$--$0.7\\%$ F1, while we achieve up to $2.5\\%$ improvement in accuracy on the visual sense disambiguation task. All our code and data are available in: \\url{https://github.com/iacercalixto/visualsem-kg}.","url_abs":"https://arxiv.org/abs/2206.13163v1","url_pdf":"https://arxiv.org/pdf/2206.13163v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"endowing-language-models-with-multimodal","repo_url":"https://github.com/iacercalixto/visualsem","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"multilingual-named-entity-recognition","task_name":"Multilingual Named Entity Recognition"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.13163","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}