{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meta-learning-for-knowledge-distillation","title":"BERT Learns to Teach: Knowledge Distillation with Meta Learning","arxiv_id":"2106.04570","date":"2021-06-08","proceeding":"ACL 2022 5","authors":["Wangchunshu Zhou","Canwen Xu","Julian McAuley"],"abstract":"We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an improved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with traditional KD algorithms and is less sensitive to the choice of different student capacity and hyperparameters, facilitating the use of KD on different tasks and models.","url_abs":"https://arxiv.org/abs/2106.04570v3","url_pdf":"https://arxiv.org/pdf/2106.04570v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meta-learning-for-knowledge-distillation","repo_url":"https://github.com/JetRunner/MetaDistil","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04570","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04570"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JetRunner/MetaDistil","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/models/resnet.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"7765583b9e540679","entry":"conv_1x1_bn","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/models/mobilenetv2.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7765583b9e540679"}},{"code_sha256_prefix":"2f7853ff01cbbc29","entry":"conv_bn","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/models/mobilenetv2.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f7853ff01cbbc29"}},{"code_sha256_prefix":"4b4f0463521302e8","entry":"get_teacher_name","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4b4f0463521302e8"}},{"code_sha256_prefix":"421fd14c036bc279","entry":"mobilenetv2_T_w","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/models/mobilenetv2.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"421fd14c036bc279"}},{"code_sha256_prefix":"f0c5a648a4b38bad","entry":"conv3x3","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/models/meta_resnet.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/models/meta_resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c5a648a4b38bad"}},{"code_sha256_prefix":"a10e2be36131a0cc","entry":"get_dataloader_sample","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/dataset/imagenet.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a10e2be36131a0cc"}},{"code_sha256_prefix":"16aace58531277b4","entry":"get_imagenet_dataloader","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/dataset/imagenet.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16aace58531277b4"}},{"code_sha256_prefix":"60ec2c0ae31874b4","entry":"get_test_loader","repo":"JetRunner/MetaDistil","repo_kind":"official","path":"cv/dataset/imagenet.py","file_url":"https://github.com/JetRunner/MetaDistil/blob/HEAD/cv/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60ec2c0ae31874b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}