{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-the-law-of-capacity-gap-in-distilling","title":"Towards the Law of Capacity Gap in Distilling Language Models","arxiv_id":"2311.07052","date":"2023-11-13","proceeding":null,"authors":["Chen Zhang","Dawei Song","Zheyu Ye","Yan Gao"],"abstract":"Language model (LM) distillation is a trending area that aims to distil the knowledge residing in a large teacher LM to a small student one. While various methods have been proposed to maximize the effectiveness of the distillation, significant challenges persist, particularly when there is a substantial capacity gap between the teacher and student LMs. This issue, often referred to as the \\textit{curse} of capacity gap, suggests that a larger teacher does not necessarily result in a superior student compared to one distilled from a smaller teacher. In other words, there is likely an optimal teacher yielding the best student along the scaling course of the teacher. However, the curse of capacity gap can not be tackled without notable compute overhead, as indicated in previous studies. In the context of large LMs (LLMs), previously viable approaches become much less meaningful, as it is an impossible triangle to distill an expected student from an optimal teacher student with small compute overhead. Fortunately, the impossible triangle can fortunately be possible provided an inducted \\textit{law} of capacity gap. In this paper, we take the spirits of scaling law and reveal that the optimal teacher scale almost consistently follows a linear scaling with the student scale across different model architectures and data scales. The law later guides us to distil a 3B student LM (termed \\textsc{MiniMA}) from LLaMA2-7B. \\textsc{MiniMA} is demonstrated to outperform a wide range of 3B competitors and could even compete with several 7B models.","url_abs":"https://arxiv.org/abs/2311.07052v3","url_pdf":"https://arxiv.org/pdf/2311.07052v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-the-law-of-capacity-gap-in-distilling","repo_url":"https://github.com/genezc/minima","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2311.07052","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.07052"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/genezc/minima","reach":null}],"summary":{"ran_fixture":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"005e006117789ac1","entry":"acc","repo":"genezc/minima","repo_kind":"official","path":"minima/run_distillation_llama_ds.py","file_url":"https://github.com/genezc/minima/blob/HEAD/minima/run_distillation_llama_ds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"005e006117789ac1"}},{"code_sha256_prefix":"5ea641b9e771d680","entry":"soft_cross_entropy","repo":"genezc/minima","repo_kind":"official","path":"minima/run_distillation_llama_ds.py","file_url":"https://github.com/genezc/minima/blob/HEAD/minima/run_distillation_llama_ds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ea641b9e771d680"}},{"code_sha256_prefix":"3d58cb57472675a1","entry":"gather","repo":"genezc/minima","repo_kind":"official","path":"minima/run_distillation_llama_ds.py","file_url":"https://github.com/genezc/minima/blob/HEAD/minima/run_distillation_llama_ds.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d58cb57472675a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}