{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/logit-standardization-in-knowledge","title":"Logit Standardization in Knowledge Distillation","arxiv_id":"2403.01427","date":"2024-03-03","proceeding":"CVPR 2024 1","authors":["Shangquan Sun","Wenqi Ren","Jingzhi Li","Rui Wang","Xiaochun Cao"],"abstract":"Knowledge distillation involves transferring soft labels from a teacher to a student using a shared temperature-based softmax function. However, the assumption of a shared temperature between teacher and student implies a mandatory exact match between their logits in terms of logit range and variance. This side-effect limits the performance of student, considering the capacity discrepancy between them and the finding that the innate logit relations of teacher are sufficient for student to learn. To address this issue, we propose setting the temperature as the weighted standard deviation of logit and performing a plug-and-play Z-score pre-process of logit standardization before applying softmax and Kullback-Leibler divergence. Our pre-process enables student to focus on essential logit relations from teacher rather than requiring a magnitude match, and can improve the performance of existing logit-based distillation methods. We also show a typical case where the conventional setting of sharing temperature between teacher and student cannot reliably yield the authentic distillation evaluation; nonetheless, this challenge is successfully alleviated by our Z-score. We extensively evaluate our method for various student and teacher models on CIFAR-100 and ImageNet, showing its significant superiority. The vanilla knowledge distillation powered by our pre-process can achieve favorable performance against state-of-the-art methods, and other distillation variants can obtain considerable gain with the assistance of our pre-process.","url_abs":"https://arxiv.org/abs/2403.01427v1","url_pdf":"https://arxiv.org/pdf/2403.01427v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"logit-standardization-in-knowledge","repo_url":"https://github.com/sunshangquan/logit-standardardization-kd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"shufflenet-v2(T:resnet-32x4, S:shufflenet-v2)","rank_in_archive_order":2,"of":27,"metrics":{"Top-1 Accuracy (%)":"78.76"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"resnet8x4\n(T: resnet32x4 S: resnet8x4)","rank_in_archive_order":4,"of":27,"metrics":{"Top-1 Accuracy (%)":"78.28"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.01427","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01427"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sunshangquan/logit-standardardization-kd","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"e528493badedd2da","entry":"DistillKL_logit_stand","repo":"sunshangquan/logit-standardardization-kd","repo_kind":"official","path":"CTKD/distiller_zoo/KD.py","file_url":"https://github.com/sunshangquan/logit-standardardization-kd/blob/HEAD/CTKD/distiller_zoo/KD.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e528493badedd2da"}},{"code_sha256_prefix":"52e1d206ab58139c","entry":"normalize","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"52e1d206ab58139c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}