{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-knowledge-distillation","title":"Decoupled Knowledge Distillation","arxiv_id":"2203.08679","date":"2022-03-16","proceeding":"CVPR 2022 1","authors":["Borui Zhao","Quan Cui","RenJie Song","Yiyu Qiu","Jiajun Liang"],"abstract":"State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the \"difficulty\" of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller.","url_abs":"https://arxiv.org/abs/2203.08679v2","url_pdf":"https://arxiv.org/pdf/2203.08679v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoupled-knowledge-distillation","repo_url":"https://github.com/megvii-research/mdistiller","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.08679","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.08679"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-research/mdistiller","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/megviiresearch/mdistiller","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":2,"ran_fixture":3,"ran_honours":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"ff70b2d07fb71583","entry":"DKD","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff70b2d07fb71583"}},{"code_sha256_prefix":"5102cbbc15c6cc90","entry":"Distiller","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5102cbbc15c6cc90"}},{"code_sha256_prefix":"c414ec11055678a1","entry":"_get_gt_mask","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c414ec11055678a1"}},{"code_sha256_prefix":"6e77e58e6d85d7e4","entry":"_get_other_mask","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e77e58e6d85d7e4"}},{"code_sha256_prefix":"a05175b39afcdcfe","entry":"cat_mask","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a05175b39afcdcfe"}},{"code_sha256_prefix":"4f5a2ac497ae4e80","entry":"dkd_loss","repo":"megvii-research/mdistiller","repo_kind":"official","path":"mdistiller/distillers/DKD.py","file_url":"https://github.com/megvii-research/mdistiller/blob/HEAD/mdistiller/distillers/DKD.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f5a2ac497ae4e80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}