{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-by-language-for-training-free","title":"Vision-by-Language for Training-Free Compositional Image Retrieval","arxiv_id":"2310.09291","date":"2023-10-13","proceeding":null,"authors":["Shyamgopal Karthik","Karsten Roth","Massimiliano Mancini","Zeynep Akata"],"abstract":"Given an image and a target modification (e.g an image of the Eiffel tower and the text \"without people and at night-time\"), Compositional Image Retrieval (CIR) aims to retrieve the relevant target image in a database. While supervised approaches rely on annotating triplets that is costly (i.e. query image, textual modification, and target image), recent research sidesteps this need by using large-scale vision-language models (VLMs), performing Zero-Shot CIR (ZS-CIR). However, state-of-the-art approaches in ZS-CIR still require training task-specific, customized models over large amounts of image-text pairs. In this work, we propose to tackle CIR in a training-free manner via our Compositional Image Retrieval through Vision-by-Language (CIReVL), a simple, yet human-understandable and scalable pipeline that effectively recombines large-scale VLMs with large language models (LLMs). By captioning the reference image using a pre-trained generative VLM and asking a LLM to recompose the caption based on the textual target modification for subsequent retrieval via e.g. CLIP, we achieve modular language reasoning. In four ZS-CIR benchmarks, we find competitive, in-part state-of-the-art performance - improving over supervised methods. Moreover, the modularity of CIReVL offers simple scalability without re-training, allowing us to both investigate scaling laws and bottlenecks for ZS-CIR while easily scaling up to in parts more than double of previously reported results. Finally, we show that CIReVL makes CIR human-understandable by composing image and text in a modular fashion in the language domain, thereby making it intervenable, allowing to post-hoc re-align failure cases. Code will be released upon acceptance.","url_abs":"https://arxiv.org/abs/2310.09291v2","url_pdf":"https://arxiv.org/pdf/2310.09291v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-by-language-for-training-free","repo_url":"https://github.com/explainableml/vision_by_language","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"zero-shot-composed-image-retrieval-zs-cir","task_name":"Zero-Shot Composed Image Retrieval (ZS-CIR)"}],"methods":[{"method_slug":"blip","method_name":"BLIP"},{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"CIReVL (CLIP G/14)","rank_in_archive_order":15,"of":43,"metrics":{"mAP@10":"27.59"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"CIReVL (CLIP L/14)","rank_in_archive_order":26,"of":43,"metrics":{"mAP@10":"19.01"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"CIReVL (CLIP B/32)","rank_in_archive_order":30,"of":43,"metrics":{"mAP@10":"15.42"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"CIReVL (CLIP G/14)","rank_in_archive_order":10,"of":47,"metrics":{"R@1":"34.65","R@5":"64.29"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"CIReVL (CLIP L/14)","rank_in_archive_order":21,"of":47,"metrics":{"R@1":"24.55","R@5":"52.31"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"CIReVL (CLIP B/32)","rank_in_archive_order":22,"of":47,"metrics":{"R@1":"23.94","R@5":"52.51"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"CIReVL (CLIP G/14)","rank_in_archive_order":20,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"42.28"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"CIReVL (CLIP B/32)","rank_in_archive_order":27,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"38.82"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"CIReVL (CLIP L/14)","rank_in_archive_order":28,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"38.56"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-11","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"GeneCIS","model":"CIReVL (CLIP B/32)","rank_in_archive_order":9,"of":11,"metrics":{"A-R@1":"15.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-11","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"GeneCIS","model":"CIReVL (CLIP L/14)","rank_in_archive_order":10,"of":11,"metrics":{"A-R@1":"15.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-11","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"GeneCIS","model":"CIReVL (CLIP G/14)","rank_in_archive_order":11,"of":11,"metrics":{"A-R@1":"17.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.09291","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09291"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ExplainableML/Vision","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/explainableml/vision_by_language","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b6c1f837e6faaeba","entry":"collate_fn","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6c1f837e6faaeba"}},{"code_sha256_prefix":"e75146036d2d1504","entry":"expand2square","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/datasets.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e75146036d2d1504"}},{"code_sha256_prefix":"55917bc288f17d17","entry":"get_recall","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55917bc288f17d17"}},{"code_sha256_prefix":"9476d6cebb3ecfb9","entry":"targetpad_transform","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9476d6cebb3ecfb9"}},{"code_sha256_prefix":"afe61f6410fb50d2","entry":"circo","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/compute_results.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/compute_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe61f6410fb50d2"}},{"code_sha256_prefix":"0cc283db93cbae1d","entry":"cirr","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/compute_results.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/compute_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cc283db93cbae1d"}},{"code_sha256_prefix":"5ee1f8a4e8bf641d","entry":"fiq","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/compute_results.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/compute_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ee1f8a4e8bf641d"}},{"code_sha256_prefix":"f6c8fb5e60546cec","entry":"openai_completion","repo":"explainableml/vision_by_language","repo_kind":"official","path":"src/openai_api.py","file_url":"https://github.com/explainableml/vision_by_language/blob/HEAD/src/openai_api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6c8fb5e60546cec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}