{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/image-retrieval-on-real-life-images-with-pre","title":"Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models","arxiv_id":"2108.04024","date":"2021-08-09","proceeding":"ICCV 2021 10","authors":["Zheyuan Liu","Cristian Rodriguez-Opazo","Damien Teney","Stephen Gould"],"abstract":"We extend the task of composed image retrieval, where an input query consists of an image and short textual description of how to modify the image. Existing methods have only been applied to non-complex images within narrow domains, such as fashion products, thereby limiting the scope of study on in-depth visual reasoning in rich image and language contexts. To address this issue, we collect the Compose Image Retrieval on Real-life images (CIRR) dataset, which consists of over 36,000 pairs of crowd-sourced, open-domain images with human-generated modifying text. To extend current methods to the open-domain, we propose CIRPLANT, a transformer based model that leverages rich pre-trained vision-and-language (V&L) knowledge for modifying visual features conditioned on natural language. Retrieval is then done by nearest neighbor lookup on the modified features. We demonstrate that with a relatively simple architecture, CIRPLANT outperforms existing methods on open-domain images, while matching state-of-the-art accuracy on the existing narrow datasets, such as fashion. Together with the release of CIRR, we believe this work will inspire further research on composed image retrieval.","url_abs":"https://arxiv.org/abs/2108.04024v1","url_pdf":"https://arxiv.org/pdf/2108.04024v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"image-retrieval-on-real-life-images-with-pre","repo_url":"https://github.com/Cuberick-Orion/CIRPLANT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"image-retrieval-on-real-life-images-with-pre","repo_url":"https://github.com/Cuberick-Orion/CIRR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"image-retrieval-on-real-life-images-with-pre","repo_url":"https://github.com/naver/artemis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"composed-image-retrieval","task_name":"Composed Image Retrieval (CoIR)"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"cirr","name":"CIRR","full_name":"Compose Image Retrieval on Real-life images"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-retrieval-on-cirr","task":"Image Retrieval","dataset":"CIRR","model":"CIRPLANT","rank_in_archive_order":16,"of":17,"metrics":{"(Recall@5+Recall_subset@1)/2":"45.88"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.04024","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.04024"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver/artemis","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Cuberick-Orion/CIRR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Cuberick-Orion/CIRPLANT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2241b22aac3dc66f","entry":"args_formatted_to_file","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"_trainval_base.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/_trainval_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2241b22aac3dc66f"}},{"code_sha256_prefix":"dd1daced3a380bf9","entry":"batched_index_select","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/utils/utils.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd1daced3a380bf9"}},{"code_sha256_prefix":"f4cf22a5a6952c82","entry":"create_model","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"trainval_oscar.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/trainval_oscar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4cf22a5a6952c82"}},{"code_sha256_prefix":"0c36cf6fecae9668","entry":"evaluate_on_nocaps","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/utils/caption_evaluate.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/utils/caption_evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c36cf6fecae9668"}},{"code_sha256_prefix":"5a4d93d8be97ccd9","entry":"instance_bce_with_logits","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/modeling/modeling_bert.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/modeling/modeling_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a4d93d8be97ccd9"}},{"code_sha256_prefix":"7703c53c246a68fa","entry":"load_dataloader","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"trainval_oscar.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/trainval_oscar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7703c53c246a68fa"}},{"code_sha256_prefix":"6ebb002290340830","entry":"load_folder","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/utils/utils.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ebb002290340830"}},{"code_sha256_prefix":"9ae9f4ec47eca173","entry":"load_wordforms","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/utils/cbs.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/utils/cbs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ae9f4ec47eca173"}},{"code_sha256_prefix":"fe97aee1e09f00f2","entry":"pairwise_cossim","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/torch_functions.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/torch_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe97aee1e09f00f2"}},{"code_sha256_prefix":"66d5486f0c42398d","entry":"pairwise_distances","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/torch_functions.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/torch_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66d5486f0c42398d"}},{"code_sha256_prefix":"513a7d2ac738d21f","entry":"select_best_beam_with_constraints","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/utils/cbs.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/utils/cbs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"513a7d2ac738d21f"}},{"code_sha256_prefix":"fc284dbe8d84606e","entry":"setup_logger","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/utils/logger.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc284dbe8d84606e"}},{"code_sha256_prefix":"8164123ea94d7bb5","entry":"top_k_top_p_filtering","repo":"Cuberick-Orion/CIRPLANT","repo_kind":"official","path":"model/OSCAR/modeling/modeling_utils.py","file_url":"https://github.com/Cuberick-Orion/CIRPLANT/blob/HEAD/model/OSCAR/modeling/modeling_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8164123ea94d7bb5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}