{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/causal-attention-for-vision-language-tasks","title":"Causal Attention for Vision-Language Tasks","arxiv_id":"2103.03493","date":"2021-03-05","proceeding":"CVPR 2021 1","authors":["Xu Yang","Hanwang Zhang","GuoJun Qi","Jianfei Cai"],"abstract":"We present a novel attention mechanism: Causal Attention (CATT), to remove the ever-elusive confounding effect in existing attention-based vision-language models. This effect causes harmful bias that misleads the attention module to focus on the spurious correlations in training data, damaging the model generalization. As the confounder is unobserved in general, we use the front-door adjustment to realize the causal intervention, which does not require any knowledge on the confounder. Specifically, CATT is implemented as a combination of 1) In-Sample Attention (IS-ATT) and 2) Cross-Sample Attention (CS-ATT), where the latter forcibly brings other samples into every IS-ATT, mimicking the causal intervention. CATT abides by the Q-K-V convention and hence can replace any attention module such as top-down attention and self-attention in Transformers. CATT improves various popular attention-based vision-language models by considerable margins. In particular, we show that CATT has great potential in large-scale pre-training, e.g., it can promote the lighter LXMERT~\\cite{tan2019lxmert}, which uses fewer data and less computational power, comparable to the heavier UNITER~\\cite{chen2020uniter}. Code is published in \\url{https://github.com/yangxuntu/catt}.","url_abs":"https://arxiv.org/abs/2103.03493v1","url_pdf":"https://arxiv.org/pdf/2103.03493v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.03493","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.03493"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yangxuntu/lxmertcatt","reach":null}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5ccefc0113e7b26e","entry":"BertAttOutput","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ccefc0113e7b26e"}},{"code_sha256_prefix":"b8c495ff176f844a","entry":"BertAttention","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8c495ff176f844a"}},{"code_sha256_prefix":"0e0a344be52d2fdc","entry":"BertCrossattLayer","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e0a344be52d2fdc"}},{"code_sha256_prefix":"61f383a856ac089e","entry":"BertOutput","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61f383a856ac089e"}},{"code_sha256_prefix":"9c6012122feb6f81","entry":"BertSelfattLayer","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c6012122feb6f81"}},{"code_sha256_prefix":"ca650f4eca9f7248","entry":"LXRTXLayer","repo":"yangxuntu/lxmertcatt","repo_kind":"found_in_text","path":"src/lxrt/modeling.py","file_url":"https://github.com/yangxuntu/lxmertcatt/blob/HEAD/src/lxrt/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca650f4eca9f7248"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}