{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/magiclens-self-supervised-image-retrieval","title":"MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions","arxiv_id":"2403.19651","date":"2024-03-28","proceeding":null,"authors":["Kai Zhang","Yi Luan","Hexiang Hu","Kenton Lee","Siyuan Qiao","Wenhu Chen","Yu Su","Ming-Wei Chang"],"abstract":"Image retrieval, i.e., finding desired images given a reference image, inherently encompasses rich, multi-faceted search intents that are difficult to capture solely using image-based measures. Recent works leverage text instructions to allow users to more freely express their search intents. However, they primarily focus on image pairs that are visually similar and/or can be characterized by a small set of pre-defined relations. The core thesis of this paper is that text instructions can enable retrieving images with richer relations beyond visual similarity. To show this, we introduce MagicLens, a series of self-supervised image retrieval models that support open-ended instructions. MagicLens is built on a key novel insight: image pairs that naturally occur on the same web pages contain a wide range of implicit relations (e.g., inside view of), and we can bring those implicit relations explicit by synthesizing instructions via foundation models. Trained on 36.7M (query image, instruction, target image) triplets with rich semantic relations mined from the web, MagicLens achieves results comparable with or better than prior best on eight benchmarks of various image retrieval tasks, while maintaining high parameter efficiency with a significantly smaller model size. Additional human analyses on a 1.4M-image unseen corpus further demonstrate the diversity of search intents supported by MagicLens. Code and models are publicly available at https://open-vision-language.github.io/MagicLens/.","url_abs":"https://arxiv.org/abs/2403.19651v2","url_pdf":"https://arxiv.org/pdf/2403.19651v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"magiclens-self-supervised-image-retrieval","repo_url":"https://github.com/google-deepmind/magiclens","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"implicit-relations","task_name":"Implicit Relations"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"supervised-image-retrieval","task_name":"Supervised Image Retrieval"},{"task_slug":"zero-shot-composed-image-retrieval-zs-cir","task_name":"Zero-Shot Composed Image Retrieval (ZS-CIR)"},{"task_slug":"zero-shot-text-to-image-retrieval","task_name":"Zero-shot Text-to-Image Retrieval"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"MagicLens (CoCa L)","rank_in_archive_order":5,"of":43,"metrics":{"mAP@10":"35.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"MagicLens (CoCa B)","rank_in_archive_order":10,"of":43,"metrics":{"mAP@10":"32.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"MagicLens (CLIP L)","rank_in_archive_order":12,"of":43,"metrics":{"mAP@10":"30.8"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"MagicLens (CLIP B)","rank_in_archive_order":20,"of":43,"metrics":{"mAP@10":"23.8"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"MagicLens (CoCa L)","rank_in_archive_order":11,"of":47,"metrics":{"R@1":"33.3","R@10":"77.9","R@5":"67.0","R@50":"94.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"MagicLens (CLIP L)","rank_in_archive_order":14,"of":47,"metrics":{"R@1":"30.1","R@10":"74.4","R@5":"61.7","R@50":"92.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"MagicLens (CoCa B)","rank_in_archive_order":27,"of":47,"metrics":{"R@5":"64.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"MagicLens (CLIP B)","rank_in_archive_order":29,"of":47,"metrics":{"R@5":"58.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"MagicLens (CoCa L)","rank_in_archive_order":7,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"48.1","R@10":"38","R@50":"58.2"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"MagicLens (CoCa B)","rank_in_archive_order":13,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"45.3"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"MagicLens (CLIP L)","rank_in_archive_order":21,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"41.6","R@10":"30.7","R@50":"52.5"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"MagicLens (CLIP B)","rank_in_archive_order":32,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"36.85"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"MagicLens (CoCa L)","rank_in_archive_order":1,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"52.95"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"MagicLens (CoCa B)","rank_in_archive_order":2,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"51.75"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"MagicLens (CLIP L)","rank_in_archive_order":3,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"48"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-6","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"ImageNet-R","model":"MagicLens (CLIP B)","rank_in_archive_order":4,"of":20,"metrics":{"(Recall@10+Recall@50)/2":"36.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.19651","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.19651"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/magiclens","reach":null}],"summary":{"ran_fixture":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b863d2779b183dbc","entry":"largest_square_crop","repo":"google-deepmind/magiclens","repo_kind":"official","path":"model.py","file_url":"https://github.com/google-deepmind/magiclens/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b863d2779b183dbc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}