{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/switch-transformers-scaling-to-trillion","title":"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity","arxiv_id":"2101.03961","date":"2021-01-11","proceeding":null,"authors":["William Fedus","Barret Zoph","Noam Shazeer"],"abstract":"In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the \"Colossal Clean Crawled Corpus\" and achieve a 4x speedup over the T5-XXL model.","url_abs":"https://arxiv.org/abs/2101.03961v3","url_pdf":"https://arxiv.org/pdf/2101.03961v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/tensorflow/mesh","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/efeslab/fiddler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/lingjzhu/charsiug2p","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/vikparuchuri/marker","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/zyphra/blackmamba","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/MindCode-4/code-5/tree/main/time_series_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"switch-transformers-scaling-to-trillion","repo_url":"https://github.com/pwc-1/Paper-10/tree/main/switch_transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"switch-ffn","method_name":"Switch FFN"},{"method_slug":"switch-transformer","method_name":"Switch Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"switch-ffn","name":"Switch FFN","full_name":"Switch FFN"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2101.03961","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.03961"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-10/tree/main/switch_transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lingjzhu/charsiug2p","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyphra/blackmamba","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-5/tree/main/time_series_transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vikparuchuri/marker","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/efeslab/fiddler","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tensorflow/mesh","reach":null}],"summary":{"ran":3,"ran_draft_wrong":4,"unverified":6},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1},"listed":{"samples":8,"ran":6,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ff4dfeb7df35fff6","entry":"MLP","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff4dfeb7df35fff6"}},{"code_sha256_prefix":"1db09a0131e40faf","entry":"MambaConfig","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1db09a0131e40faf"}},{"code_sha256_prefix":"74374c911af39f6c","entry":"MixtralBlockSparseTop2MLP","repo":"efeslab/fiddler","repo_kind":"listed","path":"benchmarks/mixtral_offloading/src/custom_layers.py","file_url":"https://github.com/efeslab/fiddler/blob/HEAD/benchmarks/mixtral_offloading/src/custom_layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74374c911af39f6c"}},{"code_sha256_prefix":"58a3d812be2f2018","entry":"init_method_normal","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58a3d812be2f2018"}},{"code_sha256_prefix":"c319770f5ead4490","entry":"scaled_init_method_normal","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c319770f5ead4490"}},{"code_sha256_prefix":"ba7b868fec181b57","entry":"selective_scan_ref","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ba7b868fec181b57"}},{"code_sha256_prefix":"bfbd162761e2e475","entry":"sinkhorn","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfbd162761e2e475"}},{"code_sha256_prefix":"02e5457338807a13","entry":"SwitchMLP","repo":"zyphra/blackmamba","repo_kind":"listed","path":"switch_mlp.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/switch_mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02e5457338807a13"}},{"code_sha256_prefix":"898627b88bfd6d44","entry":"_add_token_emb_to_gate_inputs","repo":"tensorflow/mesh","repo_kind":"official","path":"mesh_tensorflow/transformer/moe.py","file_url":"https://github.com/tensorflow/mesh/blob/HEAD/mesh_tensorflow/transformer/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"898627b88bfd6d44"}},{"code_sha256_prefix":"7179ac118906b2a5","entry":"_router_z_loss","repo":"tensorflow/mesh","repo_kind":"official","path":"mesh_tensorflow/transformer/moe.py","file_url":"https://github.com/tensorflow/mesh/blob/HEAD/mesh_tensorflow/transformer/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7179ac118906b2a5"}},{"code_sha256_prefix":"da4ea018ee62163d","entry":"_stochastically_use_non_top_expert","repo":"tensorflow/mesh","repo_kind":"official","path":"mesh_tensorflow/transformer/moe.py","file_url":"https://github.com/tensorflow/mesh/blob/HEAD/mesh_tensorflow/transformer/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da4ea018ee62163d"}},{"code_sha256_prefix":"2d4e03e49ac8f264","entry":"_switch_gating","repo":"tensorflow/mesh","repo_kind":"official","path":"mesh_tensorflow/transformer/moe.py","file_url":"https://github.com/tensorflow/mesh/blob/HEAD/mesh_tensorflow/transformer/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d4e03e49ac8f264"}},{"code_sha256_prefix":"c48b9d686b548496","entry":"selective_scan_fn","repo":"zyphra/blackmamba","repo_kind":"listed","path":"ops/selective_scan_interface.py","file_url":"https://github.com/zyphra/blackmamba/blob/HEAD/ops/selective_scan_interface.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c48b9d686b548496"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}