{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reducing-task-discrepancy-of-text-encoders","title":"An Efficient Post-hoc Framework for Reducing Task Discrepancy of Text Encoders for Composed Image Retrieval","arxiv_id":"2406.09188","date":"2024-06-13","proceeding":null,"authors":["Jaeseok Byun","Seokhyeon Jeong","Wonjae Kim","Sanghyuk Chun","Taesup Moon"],"abstract":"Composed Image Retrieval (CIR) aims to retrieve a target image based on a reference image and conditioning text, enabling controllable image searches. The mainstream Zero-Shot (ZS) CIR methods bypass the need for expensive training CIR triplets by projecting image embeddings into the text token embedding space, forming a composed query for retrieval. However, we highlight an inherent limitation in these projection-based CIR: a task discrepancy of text encoders between the original pre-training task of the encoders (text $\\leftrightarrow$ image) and the target CIR task (image + text $\\leftrightarrow$ image), which potentially negatively impacts CIR performance. To reduce such a discrepancy, a naive solution would be to train both image and text encoders with CIR triplets in a supervised manner. Instead, we introduce Reducing Task Discrepancy of Text Encoders (RTD), an efficient text-only post-hoc framework that complements projection-based CIR methods. We devise a novel target-anchored text contrastive learning designed to enhance the capability of the text encoder for CIR. We also propose two key enhancements: (1) a hard negative-based refined batch sampling strategy and (2) a refined concatenation scheme to further mitigate training-inference discrepancy. Integrating RTD into state-of-the-art projection-based methods achieves performance comparable to, or even surpassing, resource-intensive state-of-the-art synthetic CIR triplet-based approaches only with 23 minutes of additional training on 4 A100 GPUs (up to $100\\times$ faster in training). Our code will be available upon acceptance.","url_abs":"https://arxiv.org/abs/2406.09188v2","url_pdf":"https://arxiv.org/pdf/2406.09188v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reducing-task-discrepancy-of-text-encoders","repo_url":"https://github.com/navervision/lincir","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"zero-shot-composed-image-retrieval-zs-cir","task_name":"Zero-Shot Composed Image Retrieval (ZS-CIR)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"RTD + LinCIR (CLIP G/14)","rank_in_archive_order":21,"of":43,"metrics":{"mAP@10":"22.29"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"RTD + LinCIR (CLIP L/14)","rank_in_archive_order":28,"of":43,"metrics":{"mAP@10":"18.11"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"RTD + LinCIR (CLIP G/14)","rank_in_archive_order":23,"of":47,"metrics":{"R@5":"67.47"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"RTD + LinCIR (CLIP L/14)","rank_in_archive_order":33,"of":47,"metrics":{"R@5":"56.17"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"RTD + LinCIR (CLIP G/14)","rank_in_archive_order":1,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"56.74"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"RTD + LinCIR (CLIP L/14)","rank_in_archive_order":23,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"40.66"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09188","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}