{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hierarchical-self-supervised-augmented","title":"Hierarchical Self-supervised Augmented Knowledge Distillation","arxiv_id":"2107.13715","date":"2021-07-29","proceeding":null,"authors":["Chuanguang Yang","Zhulin An","Linhang Cai","Yongjun Xu"],"abstract":"Knowledge distillation often involves how to define and transfer knowledge from teacher to student effectively. Although recent self-supervised contrastive knowledge achieves the best performance, forcing the network to learn such knowledge may damage the representation learning of the original class recognition task. We therefore adopt an alternative self-supervised augmented task to guide the network to learn the joint distribution of the original recognition task and self-supervised auxiliary task. It is demonstrated as a richer knowledge to improve the representation power without losing the normal classification capability. Moreover, it is incomplete that previous methods only transfer the probabilistic knowledge between the final layers. We propose to append several auxiliary classifiers to hierarchical intermediate feature maps to generate diverse self-supervised knowledge and perform the one-to-one transfer to teach the student network thoroughly. Our method significantly surpasses the previous SOTA SSKD with an average improvement of 2.56\\% on CIFAR-100 and an improvement of 0.77\\% on ImageNet across widely used network pairs. Codes are available at https://github.com/winycg/HSAKD.","url_abs":"https://arxiv.org/abs/2107.13715v2","url_pdf":"https://arxiv.org/pdf/2107.13715v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hierarchical-self-supervised-augmented","repo_url":"https://github.com/winycg/HSAKD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/knowledge-distillation-on-imagenet","task":"Knowledge Distillation","dataset":"ImageNet","model":"HSAKD  (T: ResNet-34  S:ResNet-18)","rank_in_archive_order":31,"of":52,"metrics":{"CRD training setting":"✘","Top-1 accuracy %":"72.39"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.13715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.13715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/winycg/HSAKD","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3619a293af4bee7f","entry":"mbv2_mbv2","repo":"winycg/HSAKD","repo_kind":"official","path":"models/mutual_kd_net.py","file_url":"https://github.com/winycg/HSAKD/blob/HEAD/models/mutual_kd_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3619a293af4bee7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}