{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/combiner-full-attention-transformer-with","title":"Combiner: Full Attention Transformer with Sparse Computation Cost","arxiv_id":"2107.05768","date":"2021-07-12","proceeding":"NeurIPS 2021 12","authors":["Hongyu Ren","Hanjun Dai","Zihang Dai","Mengjiao Yang","Jure Leskovec","Dale Schuurmans","Bo Dai"],"abstract":"Transformers provide a class of expressive architectures that are extremely effective for sequence modeling. However, the key limitation of transformers is their quadratic memory and time complexity $\\mathcal{O}(L^2)$ with respect to the sequence length in attention layers, which restricts application in extremely long sequences. Most existing approaches leverage sparsity or low-rank assumptions in the attention matrix to reduce cost, but sacrifice expressiveness. Instead, we propose Combiner, which provides full attention capability in each attention head while maintaining low computation and memory complexity. The key idea is to treat the self-attention mechanism as a conditional expectation over embeddings at each location, and approximate the conditional distribution with a structured factorization. Each location can attend to all other locations, either via direct attention, or through indirect attention to abstractions, which are again conditional expectations of embeddings from corresponding local regions. We show that most sparse attention patterns used in existing sparse transformers are able to inspire the design of such factorization for full attention, resulting in the same sub-quadratic cost ($\\mathcal{O}(L\\log(L))$ or $\\mathcal{O}(L\\sqrt{L})$). Combiner is a drop-in replacement for attention layers in existing transformers and can be easily implemented in common frameworks. An experimental evaluation on both autoregressive and bidirectional sequence tasks demonstrates the effectiveness of this approach, yielding state-of-the-art results on several image and text modeling tasks.","url_abs":"https://arxiv.org/abs/2107.05768v2","url_pdf":"https://arxiv.org/pdf/2107.05768v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"combiner-full-attention-transformer-with","repo_url":"https://github.com/google-research/google-research","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"combiner-full-attention-transformer-with","repo_url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/backbone/CaiT.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-imagenet-64x64","task":"Image Generation","dataset":"ImageNet 64x64","model":"Combiner-Axial","rank_in_archive_order":41,"of":65,"metrics":{"Bits per dim":"3.42"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-64x64","task":"Image Generation","dataset":"ImageNet 64x64","model":"Combiner-Mixture","rank_in_archive_order":46,"of":65,"metrics":{"Bits per dim":"3.504"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wiki-40b","task":"Language Modelling","dataset":"Wiki-40B","model":"Combiner-Axial-8k","rank_in_archive_order":2,"of":3,"metrics":{"Perplexity":"16.49"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wiki-40b","task":"Language Modelling","dataset":"Wiki-40B","model":"Combiner-Fixed-8k","rank_in_archive_order":3,"of":3,"metrics":{"Perplexity":"16.60"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.05768","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.05768"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/google-research","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-research/googleresearch","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-research/longrange-arena","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/backbone/CaiT.py","reach":null}],"summary":{"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"cb6fb711b1ef73ae","entry":"get_fixed_sin_cos_encodings","repo":"google-research/google-research","repo_kind":"official","path":"COSTAR/src/models/utils_transformer.py","file_url":"https://github.com/google-research/google-research/blob/HEAD/COSTAR/src/models/utils_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb6fb711b1ef73ae"}},{"code_sha256_prefix":"5365366fdf3b6d3b","entry":"get_fixed_sin_cos_encodings","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5365366fdf3b6d3b"}},{"code_sha256_prefix":"1a02779f7d9d8d78","entry":"transformer_approx_att_layer","repo":"google-research/google-research","repo_kind":"official","path":"combiner/combiner/tf/models.py","file_url":"https://github.com/google-research/google-research/blob/HEAD/combiner/combiner/tf/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a02779f7d9d8d78"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}