{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/norm-knowledge-distillation-via-n-to-one","title":"NORM: Knowledge Distillation via N-to-One Representation Matching","arxiv_id":"2305.13803","date":"2023-05-23","proceeding":null,"authors":["Xiaolong Liu","Lujun Li","Chao Li","Anbang Yao"],"abstract":"Existing feature distillation methods commonly adopt the One-to-one Representation Matching between any pre-selected teacher-student layer pair. In this paper, we present N-to-One Representation (NORM), a new two-stage knowledge distillation method, which relies on a simple Feature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, during training, our FT module is merely inserted after the last convolutional layer of the student network. The first linear layer projects the student representation to a feature space having N times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space. By sequentially splitting the expanded student representation into N non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair. After training, such an FT module will be naturally merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modifications to the student network at inference. Extensive experiments on different visual recognition benchmarks demonstrate the leading performance of our method. For instance, the ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches 72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an absolute improvement of 2.01%|4.63%|3.03% against the individually trained counterpart. Code is available at https://github.com/OSVAI/NORM","url_abs":"https://arxiv.org/abs/2305.13803v1","url_pdf":"https://arxiv.org/pdf/2305.13803v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"norm-knowledge-distillation-via-n-to-one","repo_url":"https://github.com/osvai/norm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"linear-layer","method_name":"Linear Layer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.13803","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.13803"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OSVAI/NORM","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osvai/norm","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b4fb4c6c1894352c","entry":"EMDLoss","repo":"osvai/norm","repo_kind":"official","path":"distiller_zoo/FitNet.py","file_url":"https://github.com/osvai/norm/blob/HEAD/distiller_zoo/FitNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b4fb4c6c1894352c"}},{"code_sha256_prefix":"fcabf3b0cfff6f97","entry":"SinkhornDistance","repo":"osvai/norm","repo_kind":"official","path":"distiller_zoo/FitNet.py","file_url":"https://github.com/osvai/norm/blob/HEAD/distiller_zoo/FitNet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fcabf3b0cfff6f97"}},{"code_sha256_prefix":"50e6e0d8738ab5cb","entry":"dropout","repo":"OSVAI/NORM","repo_kind":"official","path":"distiller_zoo/FitNet.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/distiller_zoo/FitNet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"50e6e0d8738ab5cb"}},{"code_sha256_prefix":"abaca697293cd77b","entry":"get_cifar100_dataloaders","repo":"OSVAI/NORM","repo_kind":"official","path":"dataset/cifar100.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/dataset/cifar100.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"abaca697293cd77b"}},{"code_sha256_prefix":"4a8a3145b463068f","entry":"get_cifar100_dataloaders_sample","repo":"OSVAI/NORM","repo_kind":"official","path":"dataset/cifar100.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/dataset/cifar100.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4a8a3145b463068f"}},{"code_sha256_prefix":"c17ab84fea4709b4","entry":"get_imagenet_dataloaders","repo":"OSVAI/NORM","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c17ab84fea4709b4"}},{"code_sha256_prefix":"90eacca6c2155dec","entry":"get_imagenet_dataloaders_sample","repo":"OSVAI/NORM","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"90eacca6c2155dec"}},{"code_sha256_prefix":"ed6c3e9f7fe06186","entry":"get_test_loader","repo":"OSVAI/NORM","repo_kind":"official","path":"dataset/imagenet.py","file_url":"https://github.com/OSVAI/NORM/blob/HEAD/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ed6c3e9f7fe06186"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}