{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/isearle-improving-textual-inversion-for-zero","title":"iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval","arxiv_id":"2405.02951","date":"2024-05-05","proceeding":null,"authors":["Lorenzo Agnolucci","Alberto Baldrati","Marco Bertini","Alberto del Bimbo"],"abstract":"Given a query consisting of a reference image and a relative caption, Composed Image Retrieval (CIR) aims to retrieve target images visually similar to the reference one while incorporating the changes specified in the relative caption. The reliance of supervised methods on labor-intensive manually labeled datasets hinders their broad applicability. In this work, we introduce a new task, Zero-Shot CIR (ZS-CIR), that addresses CIR without the need for a labeled training dataset. We propose an approach named iSEARLE (improved zero-Shot composEd imAge Retrieval with textuaL invErsion) that involves mapping the visual information of the reference image into a pseudo-word token in CLIP token embedding space and combining it with the relative caption. To foster research on ZS-CIR, we present an open-domain benchmarking dataset named CIRCO (Composed Image Retrieval on Common Objects in context), the first CIR dataset where each query is labeled with multiple ground truths and a semantic categorization. The experimental results illustrate that iSEARLE obtains state-of-the-art performance on three different CIR datasets -- FashionIQ, CIRR, and the proposed CIRCO -- and two additional evaluation settings, namely domain conversion and object composition. The dataset, the code, and the model are publicly available at https://github.com/miccunifi/SEARLE.","url_abs":"https://arxiv.org/abs/2405.02951v1","url_pdf":"https://arxiv.org/pdf/2405.02951v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"isearle-improving-textual-inversion-for-zero","repo_url":"https://github.com/miccunifi/circo","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"isearle-improving-textual-inversion-for-zero","repo_url":"https://github.com/miccunifi/searle","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"composed-image-retrieval","task_name":"Composed Image Retrieval (CoIR)"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"zero-shot-composed-image-retrieval-zs-cir","task_name":"Zero-Shot Composed Image Retrieval (ZS-CIR)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":32,"of":43,"metrics":{"mAP@10":"13.61"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"iSEARLE-XL-OTI (CLIP L/14)","rank_in_archive_order":36,"of":43,"metrics":{"mAP@10":"12.67"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":38,"of":43,"metrics":{"mAP@10":"11.24"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":39,"of":43,"metrics":{"mAP@10":"10.94"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":34,"of":47,"metrics":{"R@5":"55.69"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":36,"of":47,"metrics":{"R@5":"55.18"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"iSEARLE-XL-OTI (CLIP L/14)","rank_in_archive_order":41,"of":47,"metrics":{"R@5":"54.05"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":42,"of":47,"metrics":{"R@5":"54.00"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-4","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"COCO (Common Objects in Context)","model":"iSEARLE-XL-OTI (CLIP L/14)","rank_in_archive_order":1,"of":10,"metrics":{"Actions Recall@5":"32.55"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-4","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"COCO (Common Objects in Context)","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":3,"of":10,"metrics":{"Actions Recall@5":"30.05"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-4","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"COCO (Common Objects in Context)","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":6,"of":10,"metrics":{"Actions Recall@5":"26.63"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-4","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"COCO (Common Objects in Context)","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":7,"of":10,"metrics":{"Actions Recall@5":"26.40"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"iSEARLE-XL-OTI (CLIP L/14)","rank_in_archive_order":26,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"39.39"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":30,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"38.24"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":35,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"34.93"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":36,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"34.60"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-5","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":1,"of":11,"metrics":{"Average Recall":"24.46"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-5","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet","model":"iSEARLE-XL-OTI (CLIP L/14)","rank_in_archive_order":2,"of":11,"metrics":{"Average Recall":"22.59"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-5","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":8,"of":11,"metrics":{"Average Recall":"16.01"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-5","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":9,"of":11,"metrics":{"Average Recall":"15.62"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"iSEARLE-XL (CLIP L/14)","rank_in_archive_order":5,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"24.46"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"iSEARLE (CLIP B/32)","rank_in_archive_order":11,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"16.01"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"iSEARLE-OTI (CLIP B/32)","rank_in_archive_order":12,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"15.62"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.02951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.02951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miccunifi/searle","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miccunifi/circo","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_honours":1,"ran":2,"unverified":1},"by_repo_kind":{"named_in_paper":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"b6c1f837e6faaeba","entry":"collate_fn","repo":"miccunifi/searle","repo_kind":"named_in_paper","path":"src/data_utils.py","file_url":"https://github.com/miccunifi/searle/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6c1f837e6faaeba"}},{"code_sha256_prefix":"dc736d696fd0df44","entry":"process_phrase","repo":"miccunifi/searle","repo_kind":"named_in_paper","path":"src/gpt_phrases_generation.py","file_url":"https://github.com/miccunifi/searle/blob/HEAD/src/gpt_phrases_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dc736d696fd0df44"}},{"code_sha256_prefix":"9476d6cebb3ecfb9","entry":"targetpad_transform","repo":"miccunifi/searle","repo_kind":"named_in_paper","path":"src/data_utils.py","file_url":"https://github.com/miccunifi/searle/blob/HEAD/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9476d6cebb3ecfb9"}},{"code_sha256_prefix":"f96edb7ca8fe54fe","entry":"contrastive_loss","repo":"miccunifi/searle","repo_kind":"named_in_paper","path":"src/utils.py","file_url":"https://github.com/miccunifi/searle/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f96edb7ca8fe54fe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}