{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/image-retrieval-from-contextual-descriptions-1","title":"Image Retrieval from Contextual Descriptions","arxiv_id":"2203.15867","date":"2022-03-29","proceeding":"ACL 2022 5","authors":["Benno Krojer","Vaibhav Adlakha","Vibhav Vineet","Yash Goyal","Edoardo Ponti","Siva Reddy"],"abstract":"The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we devise a new multimodal challenge, Image Retrieval from Contextual Descriptions (ImageCoDe). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description. As such, each description contains only the details that help distinguish between images. Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. Images are sourced from both static pictures and video frames. We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on ImageCoDe. Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans. Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that ImageCoDE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences.","url_abs":"https://arxiv.org/abs/2203.15867v1","url_pdf":"https://arxiv.org/pdf/2203.15867v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"image-retrieval-from-contextual-descriptions-1","repo_url":"https://github.com/mcgill-nlp/imagecode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"vilbert","method_name":"ViLBERT"}],"datasets_introduced":[{"slug":"imagecode","name":"ImageCoDe","full_name":"Image Retrieval from Contextual Descriptions"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-retrieval-on-imagecode","task":"Image Retrieval","dataset":"ImageCoDe","model":"ContextualCLIP","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"29.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.15867","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.15867"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/CLIP","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mcgill-nlp/imagecode","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"79b7c2af89e9bdad","entry":"find_best_matches","repo":"mcgill-nlp/imagecode","repo_kind":"official","path":"baselines/clip/contextual.py","file_url":"https://github.com/mcgill-nlp/imagecode/blob/HEAD/baselines/clip/contextual.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79b7c2af89e9bdad"}},{"code_sha256_prefix":"64fe66e56c74b1f6","entry":"resume","repo":"mcgill-nlp/imagecode","repo_kind":"official","path":"baselines/crossencoders/volta/train_utils.py","file_url":"https://github.com/mcgill-nlp/imagecode/blob/HEAD/baselines/crossencoders/volta/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64fe66e56c74b1f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}