{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-distillation-as-instance-specific-label","title":"Self-Distillation as Instance-Specific Label Smoothing","arxiv_id":"2006.05065","date":"2020-06-09","proceeding":"NeurIPS 2020 12","authors":["Zhilu Zhang","Mert R. Sabuncu"],"abstract":"It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.","url_abs":"https://arxiv.org/abs/2006.05065v2","url_pdf":"https://arxiv.org/pdf/2006.05065v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-distillation-as-instance-specific-label","repo_url":"https://github.com/ZhiluZhang123/neurips_2020_distillation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"label-smoothing","method_name":"Label Smoothing"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.05065","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.05065"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZhiluZhang123/neurips_2020_distillation","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b5e4e66a4c429046","entry":"get_entropy","repo":"ZhiluZhang123/neurips_2020_distillation","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/ZhiluZhang123/neurips_2020_distillation/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5e4e66a4c429046"}},{"code_sha256_prefix":"c7405034391e333f","entry":"soft_beta_loss","repo":"ZhiluZhang123/neurips_2020_distillation","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/ZhiluZhang123/neurips_2020_distillation/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7405034391e333f"}},{"code_sha256_prefix":"4697272499d11677","entry":"soft_cce_loss","repo":"ZhiluZhang123/neurips_2020_distillation","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/ZhiluZhang123/neurips_2020_distillation/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4697272499d11677"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}