{"url":"/dataset/irfl-image-recognition-of-figurative-language","name":"IRFL: Image Recognition of Figurative Language","full_name":null,"description_markdown":"The IRFL dataset consists of idioms, similes, and metaphors with matching figurative and literal images, as well as two novel tasks of multimodal figurative understanding and preference. \r\n\r\nWe collected figurative and literal images for textual idioms, metaphors, and similes using an automatic pipeline we created (idioms) and manually (metaphors + similes). We annotated the relations between these images and the figurative phrase they originated from. Using these images we created two novel tasks of figurative understanding and preference. \r\n\r\n\r\nThe figurative understanding task evaluates Vision and Language Pre-Trained Models’ (VL-PTMs) ability to understand the relation between an image and a figurative phrase. The task is to choose the image that best visualizes the figurative phrase out of X candidates. The preference task examines VL-PTMs' preference for figurative images. In this task, the model needs to classify phrase images of different categories correctly based on their ranking by the model matching score.\r\n\r\nThe best models achieve 22%, 30%, and 66% accuracy vs. humans 97%, 99.7%, and 100% on our understanding task for idioms, metaphors, and similes respectively. The best model achieved an F1 score of 61 on the preference task.\r\n\r\nResearchers are welcome to evaluate models on this dataset.","description_withheld":null,"homepage":"https://irfl-dataset.github.io/","introduced_date":"2023-03-27","introduced_date_note":null,"introduced_by":{"paper":"/paper/irfl-image-recognition-of-figurative-language","title":"IRFL: Image Recognition of Figurative Language","first_author":"Ron Yosef","url":null},"license":{"name":"cc-by-4.0","url":null},"modalities":[{"name":"Images","url":"/datasets/modality/images"},{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Classification","url":"/task/classification-1","datasets_with_task":"/datasets/task/classification-1"},{"name":"Visual Reasoning","url":"/task/visual-reasoning","datasets_with_task":"/datasets/task/visual-reasoning"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["IRFL: Image Recognition of Figurative Language"],"data_loaders":[{"repo":"https://github.com/irfl-dataset/irfl","url":"https://huggingface.co/datasets/lampent/IRFL","frameworks":[]}],"num_papers_in_archive":2,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/classification-on-irfl-image-recognition-of","task":"Classification","dataset_variant":"IRFL: Image Recognition of Figurative Language","rows":1,"metrics":["1-of-100 Accuracy"],"first_row_in_archive_order":{"model":"CLIP-RN50x64","paper":"/paper/irfl-image-recognition-of-figurative-language","metrics":{"1-of-100 Accuracy":"61"},"code_links":[{"title":"irfl-dataset/irfl","url":"https://github.com/irfl-dataset/irfl"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/visual-reasoning-on-irfl-image-recognition-of","task":"Visual Reasoning","dataset_variant":"IRFL: Image Recognition of Figurative Language","rows":1,"metrics":["1-of-100 Accuracy"],"first_row_in_archive_order":{"model":"Humans","paper":"/paper/irfl-image-recognition-of-figurative-language","metrics":{"1-of-100 Accuracy":"100"},"code_links":[{"title":"irfl-dataset/irfl","url":"https://github.com/irfl-dataset/irfl"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/irfl-image-recognition-of-figurative-language","title":"IRFL: Image Recognition of Figurative Language","date":"2023-03-27","rows_on_this_dataset":2,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-25T09:33:49+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}