{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alignment-attention-by-matching-key-and-query","title":"Alignment Attention by Matching Key and Query Distributions","arxiv_id":"2110.12567","date":"2021-10-25","proceeding":"NeurIPS 2021 12","authors":["Shujian Zhang","Xinjie Fan","Huangjie Zheng","Korawat Tanwisuth","Mingyuan Zhou"],"abstract":"The neural attention mechanism has been incorporated into deep neural networks to achieve state-of-the-art performance in various domains. Most such models use multi-head self-attention which is appealing for the ability to attend to information from different perspectives. This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. The resulting alignment attention networks can be optimized as an unsupervised regularization in the existing attention framework. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks. We further demonstrate the general applicability of our approach on graph attention and visual question answering, showing the great potential of incorporating our alignment method into various attention-related tasks.","url_abs":"https://arxiv.org/abs/2110.12567v1","url_pdf":"https://arxiv.org/pdf/2110.12567v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alignment-attention-by-matching-key-and-query","repo_url":"https://github.com/szhang42/alignment_attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"graph-attention","task_name":"Graph Attention"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.12567","atlas_url":"https://app.syntology.ai/?focus=2110.12567","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.12567"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/szhang42/alignment_attention","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9d6a7aa341c6be64","entry":"prune_linear_layer","repo":"szhang42/alignment_attention","repo_kind":"official","path":"src/transformers/modeling_albert.py","file_url":"https://github.com/szhang42/alignment_attention/blob/HEAD/src/transformers/modeling_albert.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d6a7aa341c6be64"}},{"code_sha256_prefix":"4c8499f4bae21d95","entry":"AlbertAttention","repo":"szhang42/alignment_attention","repo_kind":"official","path":"src/transformers/modeling_albert.py","file_url":"https://github.com/szhang42/alignment_attention/blob/HEAD/src/transformers/modeling_albert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c8499f4bae21d95"}},{"code_sha256_prefix":"ebb254c6336aec90","entry":"BertSelfAttention","repo":"szhang42/alignment_attention","repo_kind":"official","path":"src/transformers/modeling_albert.py","file_url":"https://github.com/szhang42/alignment_attention/blob/HEAD/src/transformers/modeling_albert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebb254c6336aec90"}},{"code_sha256_prefix":"465b788c5456f812","entry":"load_tf_weights_in_albert","repo":"szhang42/alignment_attention","repo_kind":"official","path":"src/transformers/modeling_albert.py","file_url":"https://github.com/szhang42/alignment_attention/blob/HEAD/src/transformers/modeling_albert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"465b788c5456f812"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}