{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/correlation-congruence-for-knowledge","title":"Correlation Congruence for Knowledge Distillation","arxiv_id":"1904.01802","date":"2019-04-03","proceeding":"ICCV 2019 10","authors":["Baoyun Peng","Xiao Jin","Jiaheng Liu","Shunfeng Zhou","Yi-Chao Wu","Yu Liu","Dongsheng Li","Zhaoning Zhang"],"abstract":"Most teacher-student frameworks based on knowledge distillation (KD) depend\non a strong congruent constraint on instance level. However, they usually\nignore the correlation between multiple instances, which is also valuable for\nknowledge transfer. In this work, we propose a new framework named correlation\ncongruence for knowledge distillation (CCKD), which transfers not only the\ninstance-level information, but also the correlation between instances.\nFurthermore, a generalized kernel method based on Taylor series expansion is\nproposed to better capture the correlation between instances. Empirical\nexperiments and ablation studies on image classification tasks (including\nCIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face\nRecognition) show that the proposed CCKD substantially outperforms the original\nKD and achieves state-of-the-art accuracy compared with other SOTA KD-based\nmethods. The CCKD can be easily deployed in the majority of the teacher-student\nframework such as KD and hint-based learning methods.","url_abs":"http://arxiv.org/abs/1904.01802v1","url_pdf":"http://arxiv.org/pdf/1904.01802v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"correlation-congruence-for-knowledge","repo_url":"https://github.com/SimonZsx/nasproj","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"correlation-congruence-for-knowledge","repo_url":"https://github.com/yoshitomo-matsubara/torchdistill","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"face-recognition","task_name":"Face Recognition"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.01802","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}