{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-smart-accelerating-large-language","title":"Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization","arxiv_id":"2409.12903","date":"2024-09-19","proceeding":null,"authors":["Mohammad Samragh","Iman Mirzadeh","Keivan Alizadeh Vahid","Fartash Faghri","Minsik Cho","Moin Nabi","Devang Naik","Mehrdad Farajtabar"],"abstract":"The pre-training phase of language models often begins with randomly initialized parameters. With the current trends in scaling models, training their large number of parameters can be extremely slow and costly. In contrast, small language models are less expensive to train, but they often cannot achieve the accuracy of large models. In this paper, we explore an intriguing idea to connect these two different regimes: Can we develop a method to initialize large language models using smaller pre-trained models? Will such initialization bring any benefits in terms of training time and final accuracy? In this paper, we introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. Our method ensures that the larger model retains the functionality of the smaller model. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts. We demonstrate that training such an initialized model results in significant savings in terms of GPU hours required for pre-training large language models.","url_abs":"https://arxiv.org/abs/2409.12903v2","url_pdf":"https://arxiv.org/pdf/2409.12903v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-smart-accelerating-large-language","repo_url":"https://github.com/apple/ml-hypercloning","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.12903","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.12903"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apple/ml-hypercloning","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"5bf5346c016ad1b1","entry":"add_noise","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/common.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5bf5346c016ad1b1"}},{"code_sha256_prefix":"2691c1608d583ad0","entry":"clone_gemma_qkv_bias","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/gemma_cloning.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/gemma_cloning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2691c1608d583ad0"}},{"code_sha256_prefix":"51832898f65d3183","entry":"clone_matrix","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/common.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"51832898f65d3183"}},{"code_sha256_prefix":"f6f5bad0bcf6b857","entry":"clone_pythia_qkv_bias","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/pythia_cloning.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/pythia_cloning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f6f5bad0bcf6b857"}},{"code_sha256_prefix":"ca5e5e781ea94791","entry":"get_noise_with_snr","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/common.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ca5e5e781ea94791"}},{"code_sha256_prefix":"40a641f5658d24d4","entry":"reorder_swiglu","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/olmo_cloning.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/olmo_cloning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"40a641f5658d24d4"}},{"code_sha256_prefix":"1f21241d294209d6","entry":"reorder_weights","repo":"apple/ml-hypercloning","repo_kind":"official","path":"hypercloning/olmo_cloning.py","file_url":"https://github.com/apple/ml-hypercloning/blob/HEAD/hypercloning/olmo_cloning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1f21241d294209d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}