{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-attention-better-than-matrix-decomposition-1","title":"Is Attention Better Than Matrix Decomposition?","arxiv_id":"2109.04553","date":"2021-09-09","proceeding":"ICLR 2021 1","authors":["Zhengyang Geng","Meng-Hao Guo","Hongxu Chen","Xia Li","Ke Wei","Zhouchen Lin"],"abstract":"As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank recovery problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants.","url_abs":"https://arxiv.org/abs/2109.04553v2","url_pdf":"https://arxiv.org/pdf/2109.04553v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-attention-better-than-matrix-decomposition-1","repo_url":"https://github.com/Gsunshine/Enjoy-Hamburger","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"is-attention-better-than-matrix-decomposition-1","repo_url":"https://github.com/plumprc/MTS-Mixers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"is-attention-better-than-matrix-decomposition-1","repo_url":"https://github.com/toqitahamid/gasformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"conditional-image-generation","task_name":"Conditional Image Generation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"hamburger","method_name":"Hamburger"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/conditional-image-generation-on-imagenet","task":"Conditional Image Generation","dataset":"ImageNet 128x128","model":"HamGAN","rank_in_archive_order":18,"of":22,"metrics":{"FID":"14.80","Inception score":"58.75"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Light-Ham (VAN-Huge)","rank_in_archive_order":95,"of":235,"metrics":{"GFLOPs (512 x 512)":"71.8","Params (M)":"61.1","Validation mIoU":"51.5"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Light-Ham (VAN-Large)","rank_in_archive_order":103,"of":235,"metrics":{"GFLOPs (512 x 512)":"55.0","Params (M)":"45.6","Validation mIoU":"51.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Light-Ham (VAN-Base)","rank_in_archive_order":130,"of":235,"metrics":{"GFLOPs (512 x 512)":"34.4","Params (M)":"27.4","Validation mIoU":"49.6"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"HamNet (ResNet-101)","rank_in_archive_order":170,"of":235,"metrics":{"Validation mIoU":"46.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Light-Ham (VAN-Small, D=256)","rank_in_archive_order":195,"of":235,"metrics":{"GFLOPs (512 x 512)":"15.8","Params (M)":"13.8","Validation mIoU":"45.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"Light-Ham (VAN-Huge, 61M, IN-1k, MS)","rank_in_archive_order":45,"of":95,"metrics":{"mIoU":"51.5"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"Light-Ham (VAN-Large, 46M, IN-1k, MS)","rank_in_archive_order":48,"of":95,"metrics":{"mIoU":"51.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"Light-Ham (VAN-Base, 27M, IN-1k, MS)","rank_in_archive_order":58,"of":95,"metrics":{"mIoU":"49.6"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-pascal-context","task":"Semantic Segmentation","dataset":"PASCAL Context","model":"HamNet (ResNet-101)","rank_in_archive_order":31,"of":66,"metrics":{"mIoU":"55.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-pascal-voc-2012","task":"Semantic Segmentation","dataset":"PASCAL VOC 2012 test","model":"HamNet w/o COCO (ResNet-101)","rank_in_archive_order":8,"of":51,"metrics":{"Mean IoU":"85.9%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.04553","atlas_url":"https://app.syntology.ai/?focus=2109.04553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/plumprc/MTS-Mixers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/toqitahamid/gasformer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Gsunshine/Enjoy-Hamburger","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"cbc852fb32a663c7","entry":"FactorizedTemporalMixing","repo":"plumprc/MTS-Mixers","repo_kind":"listed","path":"models/MTSMatrix.py","file_url":"https://github.com/plumprc/MTS-Mixers/blob/HEAD/models/MTSMatrix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbc852fb32a663c7"}},{"code_sha256_prefix":"27b050d7deced77c","entry":"Matrix","repo":"plumprc/MTS-Mixers","repo_kind":"listed","path":"models/MTSMatrix.py","file_url":"https://github.com/plumprc/MTS-Mixers/blob/HEAD/models/MTSMatrix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27b050d7deced77c"}},{"code_sha256_prefix":"e5482e43844f6b74","entry":"Model","repo":"plumprc/MTS-Mixers","repo_kind":"listed","path":"models/MTSMatrix.py","file_url":"https://github.com/plumprc/MTS-Mixers/blob/HEAD/models/MTSMatrix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5482e43844f6b74"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}