{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/read-me-refactorizing-llms-as-router","title":"Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design","arxiv_id":"2410.19123","date":"2024-10-24","proceeding":null,"authors":["Ruisi Cai","Yeonju Ro","Geon-Woo Kim","Peihao Wang","Babak Ehteshami Bejnordi","Aditya Akella","Zhangyang Wang"],"abstract":"The proliferation of large language models (LLMs) has led to the adoption of Mixture-of-Experts (MoE) architectures that dynamically leverage specialized subnetworks for improved efficiency and performance. Despite their benefits, MoE models face significant challenges during inference, including inefficient memory management and suboptimal batching, due to misaligned design choices between the model architecture and the system policies. Furthermore, the conventional approach of training MoEs from scratch is increasingly prohibitive in terms of cost. In this paper, we propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models (in contrast to \"upcycling\" generalist MoEs), avoiding the high costs of ground-up training. Our approach employs activation sparsity to extract experts. To compose experts, we examine the widely-adopted layer-wise router design and show its redundancy, and thus we introduce the pre-gating router decoupled from the MoE backbone that facilitates system-friendly pre-computing and lookahead scheduling, enhancing expert-aware batching and caching. Our codesign therefore addresses critical gaps on both the algorithmic and system fronts, establishing a scalable and efficient alternative for LLM inference in resource-constrained settings. Read-ME outperforms other popular open-source dense models of similar scales, achieving improvements of up to 10.1% on MMLU, and improving mean end-to-end latency up to 6.1%. Codes are available at: https://github.com/VITA-Group/READ-ME.","url_abs":"https://arxiv.org/abs/2410.19123v1","url_pdf":"https://arxiv.org/pdf/2410.19123v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"read-me-refactorizing-llms-as-router","repo_url":"https://github.com/vita-group/read-me","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.19123","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19123"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/VITA-Group/READ-ME","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/read-me","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"c2877c234b2d0e3f","entry":"loss_func","repo":"vita-group/read-me","repo_kind":"official","path":"moe_fication/train_red.py","file_url":"https://github.com/vita-group/read-me/blob/HEAD/moe_fication/train_red.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2877c234b2d0e3f"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/activations_tf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"utils/create_dummy_models.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"utils/create_dummy_models.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"dd3166a87bc972ff","entry":"get_configuration_file","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/configuration_utils.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/configuration_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd3166a87bc972ff"}},{"code_sha256_prefix":"834b335b08e359db","entry":"get_intermediate","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"moe_fication/get_activations.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/moe_fication/get_activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"834b335b08e359db"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"utils/create_dummy_models.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"9a09cbf63a4ddff4","entry":"hertz_to_mel","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/audio_utils.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/audio_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a09cbf63a4ddff4"}},{"code_sha256_prefix":"01f915e04e3d24d0","entry":"main","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"moe_fication/train_red.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/moe_fication/train_red.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01f915e04e3d24d0"}},{"code_sha256_prefix":"7e7c601df5b22499","entry":"mel_filter_bank","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/audio_utils.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/audio_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e7c601df5b22499"}},{"code_sha256_prefix":"2650530d2f0dd9b2","entry":"mel_to_hertz","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/audio_utils.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/audio_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2650530d2f0dd9b2"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/activations_tf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/src/transformers/activations_tf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}},{"code_sha256_prefix":"15b393fa2449cd6e","entry":"save_activations","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"moe_fication/get_activations.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/moe_fication/get_activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15b393fa2449cd6e"}},{"code_sha256_prefix":"16b1aa5f7cc4afe6","entry":"thresholding","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"moe_fication/generate_mask.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/moe_fication/generate_mask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16b1aa5f7cc4afe6"}},{"code_sha256_prefix":"2d40c8a96666defc","entry":"train_predictor","repo":"VITA-Group/READ-ME","repo_kind":"official","path":"moe_fication/train_red.py","file_url":"https://github.com/VITA-Group/READ-ME/blob/HEAD/moe_fication/train_red.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d40c8a96666defc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}