{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/equivariant-similarity-for-vision-language","title":"Equivariant Similarity for Vision-Language Foundation Models","arxiv_id":"2303.14465","date":"2023-03-25","proceeding":"ICCV 2023 1","authors":["Tan Wang","Kevin Lin","Linjie Li","Chung-Ching Lin","Zhengyuan Yang","Hanwang Zhang","Zicheng Liu","Lijuan Wang"],"abstract":"This study explores the concept of equivariance in vision-language foundation models (VLMs), focusing specifically on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. Unlike the existing image-text similarity objective which only categorizes matched pairs as similar and unmatched pairs as dissimilar, equivariance also requires similarity to vary faithfully according to the semantic changes. This allows VLMs to generalize better to nuanced and unseen multimodal compositions. However, modeling equivariance is challenging as the ground truth of semantic change is difficult to collect. For example, given an image-text pair about a dog, it is unclear to what extent the similarity changes when the pixel is changed from dog to cat? To this end, we propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning. Meanwhile, to further diagnose the equivariance of VLMs, we present a new challenging benchmark EqBen. Compared to the existing evaluation sets, EqBen is the first to focus on \"visual-minimal change\". Extensive experiments show the lack of equivariance in current VLMs and validate the effectiveness of EqSim. Code is available at https://github.com/Wangt-CN/EqBen.","url_abs":"https://arxiv.org/abs/2303.14465v2","url_pdf":"https://arxiv.org/pdf/2303.14465v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"equivariant-similarity-for-vision-language","repo_url":"https://github.com/wangt-cn/eqben","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"text-similarity","task_name":"text similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"FIBER (EqSim)","rank_in_archive_order":8,"of":114,"metrics":{"Group Score":"27.5","Image Score":"32.00","Text Score":"51.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"FIBER (finetuned, Flickr30k)","rank_in_archive_order":9,"of":114,"metrics":{"Group Score":"23.00","Image Score":"26.50","Text Score":"51.25"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"FIBER","rank_in_archive_order":16,"of":114,"metrics":{"Group Score":"22.25","Image Score":"25.75","Text Score":"46.25"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"METER (EqSim)","rank_in_archive_order":20,"of":114,"metrics":{"Group Score":"18.75","Image Score":"22.75","Text Score":"45.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"METER (finetuned, Flickr30k)","rank_in_archive_order":24,"of":114,"metrics":{"Group Score":"14.75","Image Score":"20.75","Text Score":"43.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"METER","rank_in_archive_order":39,"of":114,"metrics":{"Group Score":"12.00","Image Score":"15.75","Text Score":"39.25"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.14465","atlas_url":"https://app.syntology.ai/?focus=2303.14465","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.14465"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangt-cn/eqben","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Wangt-CN/EqBen","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2e0967006961706b","entry":"pairloss_criterion","repo":"Wangt-CN/EqBen","repo_kind":"official","path":"EqSim/FIBER/coarse_grained/fiber/modules/objectives.py","file_url":"https://github.com/Wangt-CN/EqBen/blob/HEAD/EqSim/FIBER/coarse_grained/fiber/modules/objectives.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e0967006961706b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}