{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-knowledge-distillation-via-1","title":"Improving Knowledge Distillation via Regularizing Feature Norm and Direction","arxiv_id":"2305.17007","date":"2023-05-26","proceeding":null,"authors":["Yuzhu Wang","Lechao Cheng","Manni Duan","Yongheng Wang","Zunlei Feng","Shu Kong"],"abstract":"Knowledge distillation (KD) exploits a large well-trained model (i.e., teacher) to train a small student model on the same dataset for the same task. Treating teacher features as knowledge, prevailing methods of knowledge distillation train student by aligning its features with the teacher's, e.g., by minimizing the KL-divergence between their logits or L2 distance between their intermediate features. While it is natural to believe that better alignment of student features to the teacher better distills teacher knowledge, simply forcing this alignment does not directly contribute to the student's performance, e.g., classification accuracy. In this work, we propose to align student features with class-mean of teacher features, where class-mean naturally serves as a strong classifier. To this end, we explore baseline techniques such as adopting the cosine distance based loss to encourage the similarity between student features and their corresponding class-means of the teacher. Moreover, we train the student to produce large-norm features, inspired by other lines of work (e.g., model pruning and domain adaptation), which find the large-norm features to be more significant. Finally, we propose a rather simple loss term (dubbed ND loss) to simultaneously (1) encourage student to produce large-\\emph{norm} features, and (2) align the \\emph{direction} of student features and teacher class-means. Experiments on standard benchmarks demonstrate that our explored techniques help existing KD methods achieve better performance, i.e., higher classification accuracy on ImageNet and CIFAR100 datasets, and higher detection precision on COCO dataset. Importantly, our proposed ND loss helps the most, leading to the state-of-the-art performance on these benchmarks. The source code is available at \\url{https://github.com/WangYZ1608/Knowledge-Distillation-via-ND}.","url_abs":"https://arxiv.org/abs/2305.17007v1","url_pdf":"https://arxiv.org/pdf/2305.17007v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-knowledge-distillation-via-1","repo_url":"https://github.com/wangyz1608/knowledge-distillation-via-nd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"ReviewKD++(T:resnet-32x4, S:shufflenet-v2)","rank_in_archive_order":6,"of":27,"metrics":{"Top-1 Accuracy (%)":"77.93"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"ReviewKD++(T:resnet-32x4, S:shufflenet-v1)","rank_in_archive_order":7,"of":27,"metrics":{"Top-1 Accuracy (%)":"77.68"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"DKD++(T:resnet-32x4, S:resnet-8x4)","rank_in_archive_order":11,"of":27,"metrics":{"Top-1 Accuracy (%)":"76.28"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"ReviewKD++(T:WRN-40-2, S:WRN-40-1)","rank_in_archive_order":13,"of":27,"metrics":{"Top-1 Accuracy (%)":"75.66"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"KD++(T:resnet56, S:resnet20)","rank_in_archive_order":22,"of":27,"metrics":{"Top-1 Accuracy (%)":"72.53"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-cifar-100","task":"Knowledge Distillation","dataset":"CIFAR-100","model":"DKD++(T:resnet50, S:mobilenetv2)","rank_in_archive_order":26,"of":27,"metrics":{"Top-1 Accuracy (%)":"70.82"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-coco-2017-val","task":"Knowledge Distillation","dataset":"COCO 2017 val","model":"ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))","rank_in_archive_order":1,"of":3,"metrics":{"AP@0.5":"61.80","AP@0.75":"44.94","mAP":"41.03"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-coco-2017-val","task":"Knowledge Distillation","dataset":"COCO 2017 val","model":"ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))","rank_in_archive_order":2,"of":3,"metrics":{"AP@0.5":"57.96","AP@0.75":"40.15","mAP":"37.43"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-coco-2017-val","task":"Knowledge Distillation","dataset":"COCO 2017 val","model":"ReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))","rank_in_archive_order":3,"of":3,"metrics":{"AP@0.5":"55.18","AP@0.75":"37.21","mAP":"34.51"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T: regnety-16GF S:ViT-B)","rank_in_archive_order":5,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"83.60","model size":"87M"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:resnet-152 S:resnet-101)","rank_in_archive_order":16,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"79.15","model size":"44.5M"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:resnet-152 S:resnet-50)","rank_in_archive_order":19,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"77.48"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:resnet152 S:resnet34)","rank_in_archive_order":25,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"75.53"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"ReviewKD++(T:resnet50, S:mobilenet-v1)","rank_in_archive_order":27,"of":52,"metrics":{"Top-1 accuracy %":"72.96"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:resnet-152 S:resnet18)","rank_in_archive_order":28,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"72.54"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:renset101 S:resnet18)","rank_in_archive_order":29,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"72.54"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:resnet50 S:resnet18)","rank_in_archive_order":30,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"72.53"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T: ResNet-34 S:ResNet-18)","rank_in_archive_order":35,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"72.07"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T:ViT-B, S:resnet18)","rank_in_archive_order":39,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"71.84"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"KD++(T: ViT-S, S:resnet18)","rank_in_archive_order":46,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"71.46"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.17007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.17007"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangyz1608/knowledge-distillation-via-nd","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b57663499e3c25d8","entry":"reduce_tensor","repo":"wangyz1608/knowledge-distillation-via-nd","repo_kind":"official","path":"ImageNet/train_dkd.py","file_url":"https://github.com/wangyz1608/knowledge-distillation-via-nd/blob/HEAD/ImageNet/train_dkd.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b57663499e3c25d8"}},{"code_sha256_prefix":"15fc62159d2fc40a","entry":"train","repo":"wangyz1608/knowledge-distillation-via-nd","repo_kind":"official","path":"ImageNet/train_dkd.py","file_url":"https://github.com/wangyz1608/knowledge-distillation-via-nd/blob/HEAD/ImageNet/train_dkd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15fc62159d2fc40a"}},{"code_sha256_prefix":"5bf784d9365d24ac","entry":"validate","repo":"wangyz1608/knowledge-distillation-via-nd","repo_kind":"official","path":"ImageNet/train_dkd.py","file_url":"https://github.com/wangyz1608/knowledge-distillation-via-nd/blob/HEAD/ImageNet/train_dkd.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bf784d9365d24ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}