{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mitigate-the-gap-investigating-approaches-for","title":"Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP","arxiv_id":"2406.17639","date":"2024-06-25","proceeding":null,"authors":["Sedigheh Eslami","Gerard de Melo"],"abstract":"Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space has been found to have a pronounced modality gap. This gap renders the embedding space overly sparse and disconnected, with different modalities being densely distributed in distinct subregions of the hypersphere. In this work, we aim at answering three main questions: 1. Does sharing the parameter space between the multi-modal encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart the uni-modal embeddings via intra-modality separation? 3. How do these gap reduction approaches affect the downstream performance? We design AlignCLIP, in order to answer these questions and through extensive experiments, we show that AlignCLIP achieves noticeable enhancements in the cross-modal alignment of the embeddings, and thereby, reduces the modality gap, while improving the performance across several zero-shot and fine-tuning downstream evaluations.","url_abs":"https://arxiv.org/abs/2406.17639v3","url_pdf":"https://arxiv.org/pdf/2406.17639v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mitigate-the-gap-investigating-approaches-for","repo_url":"https://github.com/sarahesl/alignclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"},{"task_slug":"text-similarity","task_name":"text similarity"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.17639","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.17639"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sarahesl/alignclip","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":6,"ran":5,"ran_violates":3,"unverified":8},"by_repo_kind":{"official":{"samples":22,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":22,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/tokenizer.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"ba2e1daa7be00f8c","entry":"build_zero_shot_classifier","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/zero_shot_classifier.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/zero_shot_classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba2e1daa7be00f8c"}},{"code_sha256_prefix":"838d6fbdd2edcfe2","entry":"build_zero_shot_classifier_legacy","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/zero_shot_classifier.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/zero_shot_classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"838d6fbdd2edcfe2"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/model.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"4c914b1a132fade6","entry":"get_imagenet","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/data.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c914b1a132fade6"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/model.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/tokenizer.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"7c29b374198bddcd","entry":"is_global_master","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/distributed.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c29b374198bddcd"}},{"code_sha256_prefix":"0377e309e95cfc3e","entry":"is_local_master","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/distributed.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0377e309e95cfc3e"}},{"code_sha256_prefix":"b092791002b40e8a","entry":"is_master","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/distributed.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/distributed.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b092791002b40e8a"}},{"code_sha256_prefix":"4b3a4d17e1df3140","entry":"load_state_dict","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/factory.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b3a4d17e1df3140"}},{"code_sha256_prefix":"d50f37a336cfbd4c","entry":"replace_linear","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/utils.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d50f37a336cfbd4c"}},{"code_sha256_prefix":"8e95157106d1df8c","entry":"trace_model","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/model.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e95157106d1df8c"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/tokenizer.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"9b5765523c6d17fd","entry":"freeze_batch_norm_2d","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/utils.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b5765523c6d17fd"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/loss.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/factory.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"f34d8f34635af394","entry":"get_tokenizer","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/factory.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f34d8f34635af394"}},{"code_sha256_prefix":"2c38b8a1a6553df1","entry":"image_transform","repo":"sarahesl/alignclip","repo_kind":"official","path":"align_clip/transform.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/align_clip/transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c38b8a1a6553df1"}},{"code_sha256_prefix":"c8826eee91bf0f73","entry":"remote_sync","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/file_utils.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8826eee91bf0f73"}},{"code_sha256_prefix":"5ef1e6fce93e5928","entry":"remote_sync_fsspec","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/file_utils.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ef1e6fce93e5928"}},{"code_sha256_prefix":"0cf5fb2e3a3a9114","entry":"remote_sync_s3","repo":"sarahesl/alignclip","repo_kind":"official","path":"main/file_utils.py","file_url":"https://github.com/sarahesl/alignclip/blob/HEAD/main/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cf5fb2e3a3a9114"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}