{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/outrageously-large-neural-networks-the","title":"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer","arxiv_id":"1701.06538","date":"2017-01-23","proceeding":null,"authors":["Noam Shazeer","Azalia Mirhoseini","Krzysztof Maziarz","Andy Davis","Quoc Le","Geoffrey Hinton","Jeff Dean"],"abstract":"The capacity of a neural network to absorb information is limited by its\nnumber of parameters. Conditional computation, where parts of the network are\nactive on a per-example basis, has been proposed in theory as a way of\ndramatically increasing model capacity without a proportional increase in\ncomputation. In practice, however, there are significant algorithmic and\nperformance challenges. In this work, we address these challenges and finally\nrealize the promise of conditional computation, achieving greater than 1000x\nimprovements in model capacity with only minor losses in computational\nefficiency on modern GPU clusters. We introduce a Sparsely-Gated\nMixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward\nsub-networks. A trainable gating network determines a sparse combination of\nthese experts to use for each example. We apply the MoE to the tasks of\nlanguage modeling and machine translation, where model capacity is critical for\nabsorbing the vast quantities of knowledge available in the training corpora.\nWe present model architectures in which a MoE with up to 137 billion parameters\nis applied convolutionally between stacked LSTM layers. On large language\nmodeling and machine translation benchmarks, these models achieve significantly\nbetter results than state-of-the-art at lower computational cost.","url_abs":"http://arxiv.org/abs/1701.06538v1","url_pdf":"http://arxiv.org/pdf/1701.06538v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"outrageously-large-neural-networks-the","repo_url":"https://github.com/davidmrau/mixture-of-experts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"outrageously-large-neural-networks-the","repo_url":"https://github.com/jsuarez5341/Efficient-Dynamic-Batching","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"outrageously-large-neural-networks-the","repo_url":"https://github.com/ma921/XRDidentifier","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"outrageously-large-neural-networks-the","repo_url":"https://github.com/unconst/MACH","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"High-Budget MoE","rank_in_archive_order":16,"of":27,"metrics":{"Number of params":"5B","PPL":" 28.0"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-one-billion-word","task":"Language Modelling","dataset":"One Billion Word","model":"Low-Budget MoE","rank_in_archive_order":21,"of":27,"metrics":{"Number of params":"5B","PPL":"34.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"MoE","rank_in_archive_order":31,"of":57,"metrics":{"BLEU score":"40.56","Hardware Burden":"142G"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"MoE","rank_in_archive_order":64,"of":91,"metrics":{"BLEU score":"26.03","Hardware Burden":"24G"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1701.06538","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1701.06538"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unconst/MACH","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jsuarez5341/Efficient-Dynamic-Batching","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/davidmrau/mixture-of-experts","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ma921/XRDidentifier","reach":null}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"listed":{"samples":6,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"798b423437ee44c0","entry":"MLP","repo":"davidmrau/mixture-of-experts","repo_kind":"listed","path":"moe.py","file_url":"https://github.com/davidmrau/mixture-of-experts/blob/HEAD/moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"798b423437ee44c0"}},{"code_sha256_prefix":"dad41f3c016e1b4a","entry":"MOE","repo":"jsuarez5341/Efficient-Dynamic-Batching","repo_kind":"listed","path":"MOE.py","file_url":"https://github.com/jsuarez5341/Efficient-Dynamic-Batching/blob/HEAD/MOE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dad41f3c016e1b4a"}},{"code_sha256_prefix":"bd4f0758341bbaf5","entry":"SparseDispatcher","repo":"davidmrau/mixture-of-experts","repo_kind":"listed","path":"moe.py","file_url":"https://github.com/davidmrau/mixture-of-experts/blob/HEAD/moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bd4f0758341bbaf5"}},{"code_sha256_prefix":"2284f451aa61fc6b","entry":"TwoLayerNet","repo":"jsuarez5341/Efficient-Dynamic-Batching","repo_kind":"listed","path":"MOE.py","file_url":"https://github.com/jsuarez5341/Efficient-Dynamic-Batching/blob/HEAD/MOE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2284f451aa61fc6b"}},{"code_sha256_prefix":"c31aa0d1c4d7c7dc","entry":"MoE","repo":"davidmrau/mixture-of-experts","repo_kind":"listed","path":"moe.py","file_url":"https://github.com/davidmrau/mixture-of-experts/blob/HEAD/moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c31aa0d1c4d7c7dc"}},{"code_sha256_prefix":"f9ed34e946f92347","entry":"list","repo":"jsuarez5341/Efficient-Dynamic-Batching","repo_kind":"listed","path":"MOE.py","file_url":"https://github.com/jsuarez5341/Efficient-Dynamic-Batching/blob/HEAD/MOE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9ed34e946f92347"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}