{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisit-knowledge-distillation-a-teacher-free","title":"Revisiting Knowledge Distillation via Label Smoothing Regularization","arxiv_id":"1909.11723","date":"2019-09-25","proceeding":"CVPR 2020 6","authors":["Li Yuan","Francis E. H. Tay","Guilin Li","Tao Wang","Jiashi Feng"],"abstract":"Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65\\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.","url_abs":"https://arxiv.org/abs/1909.11723v3","url_pdf":"https://arxiv.org/pdf/1909.11723v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisit-knowledge-distillation-a-teacher-free","repo_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"revisit-knowledge-distillation-a-teacher-free","repo_url":"https://github.com/yoshitomo-matsubara/torchdistill","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"self-knowledge-distillation","task_name":"Self-Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1909.11723","atlas_url":"https://app.syntology.ai/?focus=1909.11723","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.11723"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yoshitomo-matsubara/torchdistill","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a0f7289cfc868ad7","entry":"accuracy","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0f7289cfc868ad7"}},{"code_sha256_prefix":"d5eeef2165bfe6dd","entry":"accuracy","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/resnext.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/resnext.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5eeef2165bfe6dd"}},{"code_sha256_prefix":"b436429d77a35725","entry":"accuracy","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/wrn.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/wrn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b436429d77a35725"}},{"code_sha256_prefix":"cbcbff49b39a9e81","entry":"channel_shuffle","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/shufflenetv2.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/shufflenetv2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbcbff49b39a9e81"}},{"code_sha256_prefix":"258131c8ef1fa602","entry":"channel_split","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/shufflenetv2.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/shufflenetv2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"258131c8ef1fa602"}},{"code_sha256_prefix":"8219ebf427193dd2","entry":"fetch_dataloader","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"data_loader.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8219ebf427193dd2"}},{"code_sha256_prefix":"769fae72b2c05484","entry":"fetch_subset_dataloader","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"data_loader.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"769fae72b2c05484"}},{"code_sha256_prefix":"2fabab362b4fd333","entry":"loss_fn","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fabab362b4fd333"}},{"code_sha256_prefix":"135ebbf5ed793416","entry":"loss_fn","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/resnext.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/resnext.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"135ebbf5ed793416"}},{"code_sha256_prefix":"106d033266830257","entry":"loss_fn","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"model/wrn.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/model/wrn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"106d033266830257"}},{"code_sha256_prefix":"b6f5faa941f756ff","entry":"loss_kd","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"my_loss_function.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/my_loss_function.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6f5faa941f756ff"}},{"code_sha256_prefix":"3c1d36c3f9c4dcea","entry":"loss_kd_regularization","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"my_loss_function.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/my_loss_function.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c1d36c3f9c4dcea"}},{"code_sha256_prefix":"75117863b01e3f2a","entry":"loss_kd_self","repo":"yuanli2333/Teacher-free-Knowledge-Distillation","repo_kind":"official","path":"my_loss_function.py","file_url":"https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation/blob/HEAD/my_loss_function.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75117863b01e3f2a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}