{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mask-attention-networks-rethinking-and","title":"Mask Attention Networks: Rethinking and Strengthen Transformer","arxiv_id":"2103.13597","date":"2021-03-25","proceeding":"NAACL 2021 4","authors":["Zhihao Fan","Yeyun Gong","Dayiheng Liu","Zhongyu Wei","Siyuan Wang","Jian Jiao","Nan Duan","Ruofei Zhang","Xuanjing Huang"],"abstract":"Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Transformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.","url_abs":"https://arxiv.org/abs/2103.13597v1","url_pdf":"https://arxiv.org/pdf/2103.13597v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mask-attention-networks-rethinking-and","repo_url":"https://github.com/libertfan/man","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/abstractive-text-summarization-on-cnn-daily","task":"Abstractive Text Summarization","dataset":"CNN / Daily Mail","model":"Mask Attention Network","rank_in_archive_order":36,"of":53,"metrics":{"ROUGE-1":"40.98","ROUGE-2":"18.29","ROUGE-L":"37.88"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2014-german","task":"Machine Translation","dataset":"IWSLT2014 German-English","model":"Mask Attention Network (small)","rank_in_archive_order":15,"of":34,"metrics":{"BLEU score":"36.3","Number of Params":"37M"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Mask Attention Network (big)","rank_in_archive_order":11,"of":91,"metrics":{"BLEU score":"30.4","Number of Params":"215M"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Mask Attention Network (base)","rank_in_archive_order":32,"of":91,"metrics":{"BLEU score":"29.1","Number of Params":"63M"},"uses_additional_data":false},{"leaderboard":"/sota/text-summarization-on-gigaword","task":"Text Summarization","dataset":"GigaWord","model":"Mask Attention Network","rank_in_archive_order":21,"of":41,"metrics":{"ROUGE-1":"38.28","ROUGE-2":"19.46","ROUGE-L":"35.46"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.13597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.13597"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/libertfan/man","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/pltrdy/files2rouge","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1},"found_in_text":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"150f87cfae2bfdf2","entry":"MANEncoderLayer","repo":"libertfan/man","repo_kind":"listed","path":"summarization/model/man.py","file_url":"https://github.com/libertfan/man/blob/HEAD/summarization/model/man.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"150f87cfae2bfdf2"}},{"code_sha256_prefix":"c5a79ceafeff9d5e","entry":"line_count","repo":"pltrdy/files2rouge","repo_kind":"found_in_text","path":"files2rouge/utils.py","file_url":"https://github.com/pltrdy/files2rouge/blob/HEAD/files2rouge/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5a79ceafeff9d5e"}},{"code_sha256_prefix":"1d25cd789e57612f","entry":"split_files","repo":"pltrdy/files2rouge","repo_kind":"found_in_text","path":"files2rouge/utils.py","file_url":"https://github.com/pltrdy/files2rouge/blob/HEAD/files2rouge/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d25cd789e57612f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}