{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-kernel-transfer-in-knowledge","title":"Improved Knowledge Distillation via Full Kernel Matrix Transfer","arxiv_id":"2009.14416","date":"2020-09-30","proceeding":null,"authors":["Qi Qian","Hao Li","Juhua Hu"],"abstract":"Knowledge distillation is an effective way for model compression in deep learning. Given a large model (i.e., teacher model), it aims to improve the performance of a compact model (i.e., student model) by transferring the information from the teacher. Various information for distillation has been studied. Recently, a number of works propose to transfer the pairwise similarity between examples to distill relative information. However, most of efforts are devoted to developing different similarity measurements, while only a small matrix consisting of examples within a mini-batch is transferred at each iteration that can be inefficient for optimizing the pairwise similarity over the whole data set. In this work, we aim to transfer the full similarity matrix effectively. The main challenge is from the size of the full matrix that is quadratic to the number of examples. To address the challenge, we decompose the original full matrix with Nystr{\\\"{o}}m method. By selecting appropriate landmark points, our theoretical analysis indicates that the loss for transfer can be further simplified. Concretely, we find that the difference between the original full kernel matrices between teacher and student can be well bounded by that of the corresponding partial matrices, which only consists of similarities between original examples and landmark points. Compared with the full matrix, the size of the partial matrix is linear in the number of examples, which improves the efficiency of optimization significantly. The empirical study on benchmark data sets demonstrates the effectiveness of the proposed algorithm. Code is available at \\url{https://github.com/idstcv/KDA}.","url_abs":"https://arxiv.org/abs/2009.14416v2","url_pdf":"https://arxiv.org/pdf/2009.14416v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-kernel-transfer-in-knowledge","repo_url":"https://github.com/idstcv/kda","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model-compression","task_name":"Model Compression"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.14416","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.14416"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idstcv/kda","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"131a82fd65128218","entry":"accuracy","repo":"idstcv/kda","repo_kind":"official","path":"main_kda.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/main_kda.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"131a82fd65128218"}},{"code_sha256_prefix":"611ab0db5344681b","entry":"resnet18","repo":"idstcv/kda","repo_kind":"official","path":"resnet_cifar.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"611ab0db5344681b"}},{"code_sha256_prefix":"5ad4d4f85b904e35","entry":"resnet34","repo":"idstcv/kda","repo_kind":"official","path":"resnet_cifar.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ad4d4f85b904e35"}},{"code_sha256_prefix":"c2e0ae17093793fa","entry":"resnet50","repo":"idstcv/kda","repo_kind":"official","path":"resnet_cifar.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2e0ae17093793fa"}},{"code_sha256_prefix":"f5802bd2a5722722","entry":"train","repo":"idstcv/kda","repo_kind":"official","path":"main_kda.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/main_kda.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5802bd2a5722722"}},{"code_sha256_prefix":"c680d9099214f2fb","entry":"validate","repo":"idstcv/kda","repo_kind":"official","path":"main_kda.py","file_url":"https://github.com/idstcv/kda/blob/HEAD/main_kda.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c680d9099214f2fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}