{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/router-tuning-a-simple-and-effective-approach","title":"Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers","arxiv_id":"2410.13184","date":"2024-10-17","proceeding":null,"authors":["Shwai He","Tao Ge","Guoheng Sun","Bowei Tian","Xiaoyang Wang","Ang Li","Dong Yu"],"abstract":"Traditional transformer models often allocate a fixed amount of computational resources to every input token, leading to inefficient and unnecessary computation. To address this, the Mixture of Depths (MoD) was introduced to dynamically adjust the computational depth by skipping less important layers. Despite its promise, current MoD approaches remain under-explored and face two main challenges: (1) \\textit{high training costs due to the need to train the entire model along with the routers that determine which layers to skip}, and (2) \\textit{the risk of performance degradation when important layers are bypassed}. In response to the first issue, we propose Router-Tuning, a method that fine-tunes only the router on a small dataset, drastically reducing the computational overhead associated with full model training. For the second challenge, we propose MindSkip, which deploys \\textit{Attention with Dynamic Depths}. This method preserves the model's performance while significantly enhancing computational and memory efficiency. Extensive experiments demonstrate that our approach delivers competitive results while dramatically improving the computation efficiency, e.g., 21\\% speedup and only a 0.2\\% performance drop. The code is released at \\url{https://github.com/CASE-Lab-UMD/Router-Tuning}.","url_abs":"https://arxiv.org/abs/2410.13184v2","url_pdf":"https://arxiv.org/pdf/2410.13184v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"router-tuning-a-simple-and-effective-approach","repo_url":"https://github.com/case-lab-umd/router-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.13184","atlas_url":"https://app.syntology.ai/?focus=2410.13184","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13184"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/case-lab-umd/router-tuning","reach":{"status":"ok"}}],"summary":{"ran":11,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"9a39a79cccd7fa9f","entry":"convert_to_type","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/cache.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/cache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a39a79cccd7fa9f"}},{"code_sha256_prefix":"1cb0eeb16cf04ee3","entry":"encode_with_messages_format","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"entrypoints/finetune/finetune_router_tuning.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/entrypoints/finetune/finetune_router_tuning.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cb0eeb16cf04ee3"}},{"code_sha256_prefix":"88b8bf1223d522a4","entry":"encode_with_prompt_completion_format","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"entrypoints/finetune/finetune_router_tuning.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/entrypoints/finetune/finetune_router_tuning.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88b8bf1223d522a4"}},{"code_sha256_prefix":"44817f3718b05bf8","entry":"find_factors_with_minimal_sum","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/algorithms.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44817f3718b05bf8"}},{"code_sha256_prefix":"b317237cdf13806f","entry":"find_files","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/io.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b317237cdf13806f"}},{"code_sha256_prefix":"7ffd1b8c6cf87361","entry":"get_demonstrations","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ffd1b8c6cf87361"}},{"code_sha256_prefix":"ebe6dd6a79dacbd5","entry":"instagger_prompt","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/evaluation/prompts.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/evaluation/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebe6dd6a79dacbd5"}},{"code_sha256_prefix":"8f2ead0d177cde5d","entry":"llama2_prompt","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/evaluation/prompts.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/evaluation/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f2ead0d177cde5d"}},{"code_sha256_prefix":"d080647f9e54b2b9","entry":"load_compressed_file_7z","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/io.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d080647f9e54b2b9"}},{"code_sha256_prefix":"7127466930d8fc22","entry":"load_compressed_file_gz","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/io.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7127466930d8fc22"}},{"code_sha256_prefix":"41701d57e69b77ea","entry":"load_data","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41701d57e69b77ea"}},{"code_sha256_prefix":"b174c668ab8e8ebc","entry":"supports_router_tuning_patch","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/model/model_patch.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/model/model_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b174c668ab8e8ebc"}},{"code_sha256_prefix":"b9379d07b346863f","entry":"vicuna_prompt","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/evaluation/prompts.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/evaluation/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9379d07b346863f"}},{"code_sha256_prefix":"a0357baa562f6a38","entry":"apply_router_tuning_patch","repo":"case-lab-umd/router-tuning","repo_kind":"official","path":"utils/model/model_patch.py","file_url":"https://github.com/case-lab-umd/router-tuning/blob/HEAD/utils/model/model_patch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0357baa562f6a38"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}