{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-laws-for-fine-grained-mixture-of","title":"Scaling Laws for Fine-Grained Mixture of Experts","arxiv_id":"2402.07871","date":"2024-02-12","proceeding":null,"authors":["Jakub Krajewski","Jan Ludziejewski","Kamil Adamczewski","Maciej Pióro","Michał Krutul","Szymon Antoniak","Kamil Ciebiera","Krystian Król","Tomasz Odrzygóźdź","Piotr Sankowski","Marek Cygan","Sebastian Jaszczur"],"abstract":"Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.","url_abs":"https://arxiv.org/abs/2402.07871v1","url_pdf":"https://arxiv.org/pdf/2402.07871v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-laws-for-fine-grained-mixture-of","repo_url":"https://github.com/llm-random/llm-random","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.07871","atlas_url":"https://app.syntology.ai/?focus=2402.07871","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07871"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/llm-random/llm-random","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bedb49cd1f1a0de9","entry":"DenseEinMix","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/misc.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bedb49cd1f1a0de9"}},{"code_sha256_prefix":"547620f3cabb823c","entry":"decode_bias_string","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/llm.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"547620f3cabb823c"}},{"code_sha256_prefix":"ab08b9a954599b80","entry":"get_checkpoint_from_path","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/train/load_and_save_model.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/train/load_and_save_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab08b9a954599b80"}},{"code_sha256_prefix":"60ccfd22a4396cc8","entry":"get_init_fun","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/initialization.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/initialization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60ccfd22a4396cc8"}},{"code_sha256_prefix":"598beaf614d19cd9","entry":"get_init_weight","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/initialization.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/initialization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"598beaf614d19cd9"}},{"code_sha256_prefix":"4a3ee9eeb13ba96e","entry":"get_latest_checkpoint","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/train/load_and_save_model.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/train/load_and_save_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a3ee9eeb13ba96e"}},{"code_sha256_prefix":"e1f8a72b6a93def9","entry":"get_registered_name","repo":"llm-random/llm-random","repo_kind":"official","path":"research/token_reduction/layer_manager.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/research/token_reduction/layer_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1f8a72b6a93def9"}},{"code_sha256_prefix":"4a31b2005d286506","entry":"init_kaiming_uniform","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/initialization.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/initialization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a31b2005d286506"}},{"code_sha256_prefix":"17f4154f5359d704","entry":"stop_gradient","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/misc.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/misc.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17f4154f5359d704"}},{"code_sha256_prefix":"4418dc5a6351b911","entry":"einsum","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/misc.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4418dc5a6351b911"}},{"code_sha256_prefix":"01ee119a712b62ff","entry":"wrap_in_ddp","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/distributed.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01ee119a712b62ff"}},{"code_sha256_prefix":"ef09afdda4279503","entry":"wrap_in_fsdp","repo":"llm-random/llm-random","repo_kind":"official","path":"lizrd/core/distributed.py","file_url":"https://github.com/llm-random/llm-random/blob/HEAD/lizrd/core/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef09afdda4279503"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}