{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/monarch-mixer-a-simple-sub-quadratic-gemm-1","title":"Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture","arxiv_id":"2310.12109","date":"2023-10-18","proceeding":"NeurIPS 2023 11","authors":["Daniel Y. Fu","Simran Arora","Jessica Grogan","Isys Johnson","Sabri Eyuboglu","Armin W. Thomas","Benjamin Spector","Michael Poli","Atri Rudra","Christopher Ré"],"abstract":"Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures such as Transformers scale quadratically along both these axes. We ask: are there performant architectures that can scale sub-quadratically along sequence length and model dimension? We introduce Monarch Mixer (M2), a new architecture that uses the same sub-quadratic primitive along both sequence length and model dimension: Monarch matrices, a simple class of expressive structured matrices that captures many linear transforms, achieves high hardware efficiency on GPUs, and scales sub-quadratically. As a proof of concept, we explore the performance of M2 in three domains: non-causal BERT-style language modeling, ViT-style image classification, and causal GPT-style language modeling. For non-causal BERT-style modeling, M2 matches BERT-base and BERT-large in downstream GLUE quality with up to 27% fewer parameters, and achieves up to 9.1$\\times$ higher throughput at sequence length 4K. On ImageNet, M2 outperforms ViT-b by 1% in accuracy, with only half the parameters. Causal GPT-style models introduce a technical challenge: enforcing causality via masking introduces a quadratic bottleneck. To alleviate this bottleneck, we develop a novel theoretical view of Monarch matrices based on multivariate polynomial evaluation and interpolation, which lets us parameterize M2 to be causal while remaining sub-quadratic. Using this parameterization, M2 matches GPT-style Transformers at 360M parameters in pretraining perplexity on The PILE--showing for the first time that it may be possible to match Transformer quality without attention or MLPs.","url_abs":"https://arxiv.org/abs/2310.12109v1","url_pdf":"https://arxiv.org/pdf/2310.12109v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"monarch-mixer-a-simple-sub-quadratic-gemm-1","repo_url":"https://github.com/HazyResearch/m2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"4k","task_name":"4k"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.12109","atlas_url":"https://app.syntology.ai/?focus=2310.12109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/m2","reach":null}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"listed":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b8b4a54fca0d06f8","entry":"ExponentialModulation","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8b4a54fca0d06f8"}},{"code_sha256_prefix":"0bc2ee1177b6118f","entry":"HyenaFilter","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bc2ee1177b6118f"}},{"code_sha256_prefix":"c3fc52e16deda095","entry":"PositionalEmbedding","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3fc52e16deda095"}},{"code_sha256_prefix":"46738ca5fd203f92","entry":"Sin","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46738ca5fd203f92"}},{"code_sha256_prefix":"38673f48ffeede10","entry":"MonarchMixerSequenceMixing","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38673f48ffeede10"}},{"code_sha256_prefix":"6f2ce45de2079602","entry":"OptimModule","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f2ce45de2079602"}},{"code_sha256_prefix":"1ea1c3c4388f11da","entry":"fftconv_ref","repo":"HazyResearch/m2","repo_kind":"listed","path":"bert/src/mm/monarch_mixer_sequence_mixer.py","file_url":"https://github.com/HazyResearch/m2/blob/HEAD/bert/src/mm/monarch_mixer_sequence_mixer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ea1c3c4388f11da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}