{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/synthesizer-rethinking-self-attention-in","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","arxiv_id":"2005.00743","date":"2020-05-02","proceeding":null,"authors":["Yi Tay","Dara Bahri","Donald Metzler","Da-Cheng Juan","Zhe Zhao","Che Zheng"],"abstract":"The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \\textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\\%$ faster but also improves perplexity by a relative $3.5\\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.","url_abs":"https://arxiv.org/abs/2005.00743v3","url_pdf":"https://arxiv.org/pdf/2005.00743v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"synthesizer-rethinking-self-attention-in","repo_url":"https://github.com/10-zin/Synthesizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"document-summarization","task_name":"Document Summarization"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dense-synthesized-attention","method_name":"Dense Synthesized Attention"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"factorized-dense-synthesized-attention","method_name":"Factorized Dense Synthesized Attention"},{"method_slug":"factorized-random-synthesized-attention","method_name":"Factorized Random Synthesized Attention"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"random-synthesized-attention","method_name":"Random Synthesized Attention"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"synthesizer","method_name":"Synthesizer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"factorized-dense-synthesized-attention","name":"Factorized Dense Synthesized Attention","full_name":"Factorized Dense Synthesized Attention"},{"slug":"factorized-random-synthesized-attention","name":"Factorized Random Synthesized Attention","full_name":"Factorized Random Synthesized Attention"}],"results":[{"leaderboard":"/sota/dialogue-generation-on-persona-chat-1","task":"Dialogue Generation","dataset":"Persona-Chat","model":"Synthesizer (R+V)","rank_in_archive_order":6,"of":6,"metrics":{"BLEU-1":"14.7","CIDr":"19.09","METEOR":"6.39","ROUGE-L":"14.79"},"uses_additional_data":true},{"leaderboard":"/sota/document-summarization-on-cnn-daily-mail","task":"Document Summarization","dataset":"CNN / Daily Mail","model":"Synthesizer (R+V)","rank_in_archive_order":23,"of":26,"metrics":{"ROUGE-1":"38.57","ROUGE-2":"16.24","ROUGE-L":"35.95"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-cola-dev","task":"Linguistic Acceptability","dataset":"CoLA Dev","model":"Synthesizer (R+V)","rank_in_archive_order":5,"of":6,"metrics":{"Accuracy":"53.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"Synthesizer (Random + Vanilla)","rank_in_archive_order":20,"of":57,"metrics":{"BLEU score":"41.85"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Synthesizer (Random + Vanilla)","rank_in_archive_order":42,"of":91,"metrics":{"BLEU score":"28.47"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-textual-similarity-on-mrpc-dev","task":"Semantic Textual Similarity","dataset":"MRPC Dev","model":"Synthesizer (R+V)","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"91.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2005.00743","atlas_url":"https://app.syntology.ai/?focus=2005.00743","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.00743"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/10-zin/Synthesizer","reach":null}],"summary":{"ran":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54e8323cea270656","entry":"DenseAttention","repo":"10-zin/Synthesizer","repo_kind":"listed","path":"synth/synthesizer/modules.py","file_url":"https://github.com/10-zin/Synthesizer/blob/HEAD/synth/synthesizer/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e8323cea270656"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}