{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/in-defense-of-feature-mimicking-for-knowledge","title":"Distilling Knowledge by Mimicking Features","arxiv_id":"2011.01424","date":"2020-11-03","proceeding":null,"authors":["Guo-Hua Wang","Yifan Ge","Jianxin Wu"],"abstract":"Knowledge distillation (KD) is a popular method to train efficient networks (\"student\") with the help of high-capacity networks (\"teacher\"). Traditional methods use the teacher's soft logits as extra supervision to train the student network. In this paper, we argue that it is more advantageous to make the student mimic the teacher's features in the penultimate layer. Not only the student can directly learn more effective information from the teacher feature, feature mimicking can also be applied for teachers trained without a softmax layer. Experiments show that it can achieve higher accuracy than traditional KD. To further facilitate feature mimicking, we decompose a feature vector into the magnitude and the direction. We argue that the teacher should give more freedom to the student feature's magnitude, and let the student pay more attention on mimicking the feature direction. To meet this requirement, we propose a loss term based on locality-sensitive hashing (LSH). With the help of this new loss, our method indeed mimics feature directions more accurately, relaxes constraints on feature magnitudes, and achieves state-of-the-art distillation accuracy. We provide theoretical analyses of how LSH facilitates feature direction mimicking, and further extend feature mimicking to multi-label recognition and object detection.","url_abs":"https://arxiv.org/abs/2011.01424v2","url_pdf":"https://arxiv.org/pdf/2011.01424v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-defense-of-feature-mimicking-for-knowledge","repo_url":"https://github.com/DoctorKey/LSHFM.detection","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"in-defense-of-feature-mimicking-for-knowledge","repo_url":"https://github.com/DoctorKey/LSHFM.multiclassification","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"in-defense-of-feature-mimicking-for-knowledge","repo_url":"https://github.com/DoctorKey/LSHFM.singleclassification","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/knowledge-distillation-on-coco","task":"Knowledge Distillation","dataset":"COCO (Common Objects in Context)","model":"LSHFM (T: ResNet101 S: ResNet50)","rank_in_archive_order":3,"of":4,"metrics":{"mAP":"77.16"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-coco","task":"Knowledge Distillation","dataset":"COCO (Common Objects in Context)","model":"LSHFM (T: ResNet101 S: MobileNetV2)","rank_in_archive_order":4,"of":4,"metrics":{"mAP":"73.73"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"LSHFM (T: ResNet-34 S:ResNet-18)","rank_in_archive_order":40,"of":52,"metrics":{"Top-1 accuracy %":"71.72"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-pascal-voc","task":"Knowledge Distillation","dataset":"PASCAL VOC","model":"LSHFM (T: ResNet101 S: ResNet50)","rank_in_archive_order":1,"of":2,"metrics":{"mAP":"93.17"},"uses_additional_data":false},{"leaderboard":"/sota/knowledge-distillation-on-pascal-voc","task":"Knowledge Distillation","dataset":"PASCAL VOC","model":"LSHFM (T: ResNet101 S: MobileNetV2)","rank_in_archive_order":2,"of":2,"metrics":{"mAP":"90.14"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2011.01424","atlas_url":"https://app.syntology.ai/?focus=2011.01424","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.01424"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DoctorKey/LSHFM.multiclassification","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DoctorKey/LSHFM.singleclassification","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DoctorKey/LSHFM.detection","reach":null}],"summary":{"ran_fixture":3,"ran_draft_wrong":1,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"131a82fd65128218","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"131a82fd65128218"}},{"code_sha256_prefix":"4b4f0463521302e8","entry":"get_teacher_name","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4b4f0463521302e8"}},{"code_sha256_prefix":"e48a7a2383524bda","entry":"validate","repo":"DoctorKey/LSHFM.singleclassification","repo_kind":"official","path":"imagenet_lsh.py","file_url":"https://github.com/DoctorKey/LSHFM.singleclassification/blob/HEAD/imagenet_lsh.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e48a7a2383524bda"}},{"code_sha256_prefix":"60427af0bdab606c","entry":"vis_bbox","repo":"DoctorKey/LSHFM.detection","repo_kind":"official","path":"src/visual.py","file_url":"https://github.com/DoctorKey/LSHFM.detection/blob/HEAD/src/visual.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60427af0bdab606c"}},{"code_sha256_prefix":"9975d86336dcfa7e","entry":"vis_image","repo":"DoctorKey/LSHFM.detection","repo_kind":"official","path":"src/visual.py","file_url":"https://github.com/DoctorKey/LSHFM.detection/blob/HEAD/src/visual.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9975d86336dcfa7e"}},{"code_sha256_prefix":"c1601433df697b2a","entry":"train","repo":"DoctorKey/LSHFM.singleclassification","repo_kind":"official","path":"imagenet_lsh.py","file_url":"https://github.com/DoctorKey/LSHFM.singleclassification/blob/HEAD/imagenet_lsh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1601433df697b2a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}