{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-centered-kernel-alignment-in","title":"Rethinking Centered Kernel Alignment in Knowledge Distillation","arxiv_id":"2401.11824","date":"2024-01-22","proceeding":null,"authors":["Zikai Zhou","Yunhang Shen","Shitong Shao","Linrui Gong","Shaohui Lin"],"abstract":"Knowledge distillation has emerged as a highly effective method for bridging the representation discrepancy between large-scale models and lightweight models. Prevalent approaches involve leveraging appropriate metrics to minimize the divergence or distance between the knowledge extracted from the teacher model and the knowledge learned by the student model. Centered Kernel Alignment (CKA) is widely used to measure representation similarity and has been applied in several knowledge distillation methods. However, these methods are complex and fail to uncover the essence of CKA, thus not answering the question of how to use CKA to achieve simple and effective distillation properly. This paper first provides a theoretical perspective to illustrate the effectiveness of CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy~(MMD) and a constant term. Drawing from this, we propose a novel Relation-Centered Kernel Alignment~(RCKA) framework, which practically establishes a connection between CKA and MMD. Furthermore, we dynamically customize the application of CKA based on the characteristics of each task, with less computational source yet comparable performance than the previous methods. The extensive experiments on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs for image classification and object detection, validating the effectiveness of our approaches. Our code is available in https://github.com/Klayand/PCKA","url_abs":"https://arxiv.org/abs/2401.11824v4","url_pdf":"https://arxiv.org/pdf/2401.11824v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-centered-kernel-alignment-in","repo_url":"https://github.com/klayand/pcka","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.11824","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.11824"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/klayand/pcka","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Klayand/PCKA","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"179a3e25e2e7ca16","entry":"capitalize","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmyolo/.dev_scripts/print_registers.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmyolo/.dev_scripts/print_registers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"179a3e25e2e7ca16"}},{"code_sha256_prefix":"7aa14690a60bba8a","entry":"compare_metric","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmrazor/.dev_scripts/benchmark_summary_analyse.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmrazor/.dev_scripts/benchmark_summary_analyse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7aa14690a60bba8a"}},{"code_sha256_prefix":"464ead2b752c6eb7","entry":"load_modules_from_dir","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmyolo/.dev_scripts/print_registers.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmyolo/.dev_scripts/print_registers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"464ead2b752c6eb7"}},{"code_sha256_prefix":"c2c28aecae545cec","entry":"parse_repo_name","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmyolo/.dev_scripts/print_registers.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmyolo/.dev_scripts/print_registers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2c28aecae545cec"}},{"code_sha256_prefix":"4d5e2d377341d2fd","entry":"process_checkpoint","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmyolo/.dev_scripts/gather_models.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmyolo/.dev_scripts/gather_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d5e2d377341d2fd"}},{"code_sha256_prefix":"8ff99a6bd2ebf598","entry":"ordered_yaml_dump","repo":"Klayand/PCKA","repo_kind":"official","path":"PCKA/mmyolo/.dev_scripts/gather_models.py","file_url":"https://github.com/Klayand/PCKA/blob/HEAD/PCKA/mmyolo/.dev_scripts/gather_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ff99a6bd2ebf598"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}