{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distillation-quantification-for-large","title":"Distillation Quantification for Large Language Models","arxiv_id":"2501.12619","date":"2025-01-22","proceeding":null,"authors":["Sunbowen Lee","Junting Zhou","Chang Ao","Kaige Li","Xinrun Du","Sirui He","Jiaheng Liu","Min Yang","Zhoufutu Wen","Shiwen Ni"],"abstract":"Model distillation is a technique for transferring knowledge from large language models (LLMs) to smaller ones, aiming to create resource-efficient yet high-performing models. However, excessive distillation can lead to homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification.","url_abs":"https://arxiv.org/abs/2501.12619v2","url_pdf":"https://arxiv.org/pdf/2501.12619v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distillation-quantification-for-large","repo_url":"https://github.com/aegis1863/llms-distillation-quantification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.12619","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.12619"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aegis1863/llms-distillation-quantification","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Aegis1863/LLMs-Distillation-Quantification","reach":{"status":"ok"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"9758dd55551e8559","entry":"Evaluator","repo":"Aegis1863/LLMs-Distillation-Quantification","repo_kind":"official","path":"easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","file_url":"https://github.com/Aegis1863/LLMs-Distillation-Quantification/blob/HEAD/easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9758dd55551e8559"}},{"code_sha256_prefix":"6840e4733c1591dd","entry":"EvaluatorClassificationGetScore","repo":"Aegis1863/LLMs-Distillation-Quantification","repo_kind":"official","path":"easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","file_url":"https://github.com/Aegis1863/LLMs-Distillation-Quantification/blob/HEAD/easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6840e4733c1591dd"}},{"code_sha256_prefix":"03e2455f600b42bd","entry":"Instance","repo":"Aegis1863/LLMs-Distillation-Quantification","repo_kind":"official","path":"easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","file_url":"https://github.com/Aegis1863/LLMs-Distillation-Quantification/blob/HEAD/easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03e2455f600b42bd"}},{"code_sha256_prefix":"57235e7502859cc3","entry":"JailbreakDataset","repo":"Aegis1863/LLMs-Distillation-Quantification","repo_kind":"official","path":"easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","file_url":"https://github.com/Aegis1863/LLMs-Distillation-Quantification/blob/HEAD/easyjailbreak/metrics/Evaluator/Evaluator_ClassificationGetScore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57235e7502859cc3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}