{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/irfl-image-recognition-of-figurative-language","title":"IRFL: Image Recognition of Figurative Language","arxiv_id":"2303.15445","date":"2023-03-27","proceeding":null,"authors":["Ron Yosef","Yonatan Bitton","Dafna Shahaf"],"abstract":"Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code, in hopes of driving the development of models that can better understand figurative language.","url_abs":"https://arxiv.org/abs/2303.15445v3","url_pdf":"https://arxiv.org/pdf/2303.15445v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"irfl-image-recognition-of-figurative-language","repo_url":"https://github.com/irfl-dataset/irfl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"irfl-image-recognition-of-figurative-language","name":"IRFL: Image Recognition of Figurative Language","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/classification-on-irfl-image-recognition-of","task":"Classification","dataset":"IRFL: Image Recognition of Figurative Language","model":"CLIP-RN50x64","rank_in_archive_order":1,"of":1,"metrics":{"1-of-100 Accuracy":"61"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-irfl-image-recognition-of","task":"Visual Reasoning","dataset":"IRFL: Image Recognition of Figurative Language","model":"Humans","rank_in_archive_order":1,"of":1,"metrics":{"1-of-100 Accuracy":"100"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.15445","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}