{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-invariant-causal-mechanism-from","title":"Learning Invariant Causal Mechanism from Vision-Language Models","arxiv_id":"2405.15289","date":"2024-05-24","proceeding":null,"authors":["Zeen Song","Siyu Zhao","Xingyu Zhang","Jiangmeng Li","Changwen Zheng","Wenwen Qiang"],"abstract":"Large-scale pre-trained vision-language models such as CLIP have been widely applied to a variety of downstream scenarios. In real-world applications, the CLIP model is often utilized in more diverse scenarios than those encountered during its training, a challenge known as the out-of-distribution (OOD) problem. However, our experiments reveal that CLIP performs unsatisfactorily in certain domains. Through a causal analysis, we find that CLIP's current prediction process cannot guarantee a low OOD risk. The lowest OOD risk can be achieved when the prediction process is based on invariant causal mechanisms, i.e., predicting solely based on invariant latent factors. However, theoretical analysis indicates that CLIP does not identify these invariant latent factors. Therefore, we propose the Invariant Causal Mechanism for CLIP (CLIP-ICM), a framework that first identifies invariant latent factors using interventional data and then performs invariant predictions across various domains. Our method is simple yet effective, without significant computational overhead. Experimental results demonstrate that CLIP-ICM significantly improves CLIP's performance in OOD scenarios.","url_abs":"https://arxiv.org/abs/2405.15289v2","url_pdf":"https://arxiv.org/pdf/2405.15289v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"causal-inference","method_name":"Causal inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15289","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15289"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZeenSong/CLIP-ICM","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"found_in_text":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"c3c468075fb284f0","entry":"estimate_content","repo":"ZeenSong/CLIP-ICM","repo_kind":"found_in_text","path":"clip_icm.py","file_url":"https://github.com/ZeenSong/CLIP-ICM/blob/HEAD/clip_icm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3c468075fb284f0"}},{"code_sha256_prefix":"0dae99334a1abddb","entry":"get_representation","repo":"ZeenSong/CLIP-ICM","repo_kind":"found_in_text","path":"clip_icm.py","file_url":"https://github.com/ZeenSong/CLIP-ICM/blob/HEAD/clip_icm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dae99334a1abddb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}