{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/amoeballm-constructing-any-shape-large","title":"AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment","arxiv_id":"2411.10606","date":"2024-11-15","proceeding":null,"authors":["Yonggan Fu","Zhongzhi Yu","Junwei Li","Jiayi Qian","Yongan Zhang","Xiangchi Yuan","Dachuan Shi","Roman Yakunin","Yingyan Celine Lin"],"abstract":"Motivated by the transformative capabilities of large language models (LLMs) across various natural language tasks, there has been a growing demand to deploy these models effectively across diverse real-world applications and platforms. However, the challenge of efficiently deploying LLMs has become increasingly pronounced due to the varying application-specific performance requirements and the rapid evolution of computational platforms, which feature diverse resource constraints and deployment flows. These varying requirements necessitate LLMs that can adapt their structures (depth and width) for optimal efficiency across different platforms and application specifications. To address this critical gap, we propose AmoebaLLM, a novel framework designed to enable the instant derivation of LLM subnets of arbitrary shapes, which achieve the accuracy-efficiency frontier and can be extracted immediately after a one-time fine-tuning. In this way, AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications. Specifically, AmoebaLLM integrates three innovative components: (1) a knowledge-preserving subnet selection strategy that features a dynamic-programming approach for depth shrinking and an importance-driven method for width shrinking; (2) a shape-aware mixture of LoRAs to mitigate gradient conflicts among subnets during fine-tuning; and (3) an in-place distillation scheme with loss-magnitude balancing as the fine-tuning objective. Extensive experiments validate that AmoebaLLM not only sets new standards in LLM adaptability but also successfully delivers subnets that achieve state-of-the-art trade-offs between accuracy and efficiency.","url_abs":"https://arxiv.org/abs/2411.10606v1","url_pdf":"https://arxiv.org/pdf/2411.10606v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"amoeballm-constructing-any-shape-large","repo_url":"https://github.com/GATECH-EIC/AmoebaLLM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.10606","atlas_url":"https://app.syntology.ai/?focus=2411.10606","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.10606"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GATECH-EIC/AmoebaLLM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":3,"unverified":7},"by_repo_kind":{"official":{"samples":13,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"dee8be97b2cf4692","entry":"get_wikitext2","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"width_shrink/data.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/width_shrink/data.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dee8be97b2cf4692"}},{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"lm-evaluation-harness/lm_eval/api/model.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/lm-evaluation-harness/lm_eval/api/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256_prefix":"400ea4f64ccce5ef","entry":"parse_eval_args","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"lm-evaluation-harness/lm_eval/__main__.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/lm-evaluation-harness/lm_eval/__main__.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"400ea4f64ccce5ef"}},{"code_sha256_prefix":"c44830a7722ff5f7","entry":"request_caching_arg_to_dict","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"lm-evaluation-harness/lm_eval/evaluator.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/lm-evaluation-harness/lm_eval/evaluator.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c44830a7722ff5f7"}},{"code_sha256_prefix":"3bf1c3c00271897f","entry":"batch_split","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"eval_func.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/eval_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bf1c3c00271897f"}},{"code_sha256_prefix":"89809a6f227449c9","entry":"compute_metrics","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"eval_func.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/eval_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89809a6f227449c9"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"transformers/utils/create_dummy_models.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/transformers/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"7630a157a3b32ab9","entry":"get_c4","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"width_shrink/data.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/width_shrink/data.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7630a157a3b32ab9"}},{"code_sha256_prefix":"16994008353af958","entry":"get_loaders","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"width_shrink/data.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/width_shrink/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16994008353af958"}},{"code_sha256_prefix":"a5c450766875733e","entry":"prepare_calibration_input","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"width_shrink/prune.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/width_shrink/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5c450766875733e"}},{"code_sha256_prefix":"5dc37098fd3e0e07","entry":"prepare_input","repo":"GATECH-EIC/AmoebaLLM","repo_kind":"official","path":"eval_func.py","file_url":"https://github.com/GATECH-EIC/AmoebaLLM/blob/HEAD/eval_func.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dc37098fd3e0e07"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}