{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/torchdistill-a-modular-configuration-driven","title":"torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation","arxiv_id":"2011.12913","date":"2020-11-25","proceeding":null,"authors":["Yoshitomo Matsubara"],"abstract":"While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks to lower barriers to such high-quality, reproducible deep learning research. Several researchers voluntarily published frameworks used in their knowledge distillation studies to help other interested researchers reproduce their original work. Such frameworks, however, are usually neither well generalized nor maintained, thus researchers are still required to write a lot of code to refactor/build on the frameworks for introducing new methods, models, datasets and designing experiments. In this paper, we present our developed open-source framework built on PyTorch and dedicated for knowledge distillation studies. The framework is designed to enable users to design experiments by declarative PyYAML configuration files, and helps researchers complete the recently proposed ML Code Completeness Checklist. Using the developed framework, we demonstrate its various efficient training strategies, and implement a variety of knowledge distillation methods. We also reproduce some of their original experimental results on the ImageNet and COCO datasets presented at major machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including recent state-of-the-art methods. All the source code, configurations, log files and trained model weights are publicly available at https://github.com/yoshitomo-matsubara/torchdistill .","url_abs":"https://arxiv.org/abs/2011.12913v2","url_pdf":"https://arxiv.org/pdf/2011.12913v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"torchdistill-a-modular-configuration-driven","repo_url":"https://github.com/yoshitomo-matsubara/torchdistill","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"neural-network-compression","task_name":"Neural Network Compression"},{"task_slug":"object-detection","task_name":"Object Detection"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (PAD-L2 w/ ResNet-34 teacher)","rank_in_archive_order":1005,"of":1060,"metrics":{"Top 1 Accuracy":"71.71%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (FT w/ ResNet-34 teacher)","rank_in_archive_order":1008,"of":1060,"metrics":{"Top 1 Accuracy":"71.56%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (KD w/ ResNet-34 teacher)","rank_in_archive_order":1011,"of":1060,"metrics":{"Top 1 Accuracy":"71.37%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (L2 w/ ResNet-34 teacher)","rank_in_archive_order":1014,"of":1060,"metrics":{"Top 1 Accuracy":"71.08%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (CRD w/ ResNet-34 teacher)","rank_in_archive_order":1016,"of":1060,"metrics":{"Top 1 Accuracy":"70.93%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (tf-KD w/ ResNet-18 teacher)","rank_in_archive_order":1020,"of":1060,"metrics":{"Top 1 Accuracy":"70.52%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ResNet-18 (SSKD w/ ResNet-34 teacher)","rank_in_archive_order":1024,"of":1060,"metrics":{"Top 1 Accuracy":"70.09%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"Mask R-CNN (Bottleneck-injected ResNet-50, FPN)","rank_in_archive_order":101,"of":112,"metrics":{"mask AP":"33.6"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"Mask R-CNN (Bottleneck-injected ResNet-50, FPN)","rank_in_archive_order":220,"of":225,"metrics":{"box mAP":"36.9"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"Faster R-CNN (Bottleneck-injected ResNet-50 and FPN)","rank_in_archive_order":225,"of":225,"metrics":{"box mAP":"35.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2011.12913","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.12913"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yoshitomo-matsubara/torchdistill","reach":null}],"summary":{"ran_draft_wrong":10,"unverified":16},"by_repo_kind":{"official":{"samples":26,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c3ed65ba6ea90613","entry":"_resolve_module_path_flag","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ed65ba6ea90613"}},{"code_sha256_prefix":"3254f2af805330f9","entry":"build_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3254f2af805330f9"}},{"code_sha256_prefix":"87b46952aaf35391","entry":"get_device_index","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87b46952aaf35391"}},{"code_sha256_prefix":"fd6d7563a4a31ea5","entry":"get_proc_config","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd6d7563a4a31ea5"}},{"code_sha256_prefix":"4b491e5e58b0f881","entry":"register_forward_hook_with_dict","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b491e5e58b0f881"}},{"code_sha256_prefix":"c92bc6d24855c49a","entry":"register_post_epoch_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c92bc6d24855c49a"}},{"code_sha256_prefix":"7c61d41d0c3e500d","entry":"register_post_forward_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c61d41d0c3e500d"}},{"code_sha256_prefix":"b354561e7df46cce","entry":"register_pre_epoch_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b354561e7df46cce"}},{"code_sha256_prefix":"6a583218d3dff95c","entry":"register_pre_forward_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a583218d3dff95c"}},{"code_sha256_prefix":"f44f1f4a4378af5a","entry":"set_hooks","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f44f1f4a4378af5a"}},{"code_sha256_prefix":"5e932d9e557d1152","entry":"DistillationBox","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e932d9e557d1152"}},{"code_sha256_prefix":"733dc5424bd2b106","entry":"ForwardHookManager","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"733dc5424bd2b106"}},{"code_sha256_prefix":"62cb0c77042128ad","entry":"_extract_module","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62cb0c77042128ad"}},{"code_sha256_prefix":"3f67b3fedf426f90","entry":"clear_io_dict","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f67b3fedf426f90"}},{"code_sha256_prefix":"0b996e294219e230","entry":"clear_io_dict_values","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b996e294219e230"}},{"code_sha256_prefix":"7b5cddaa746c727a","entry":"default_post_epoch_process_with_teacher","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b5cddaa746c727a"}},{"code_sha256_prefix":"25556f7f6770ba26","entry":"default_post_forward_process","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25556f7f6770ba26"}},{"code_sha256_prefix":"4d443b9aef2436a9","entry":"default_pre_epoch_process_with_teacher","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d443b9aef2436a9"}},{"code_sha256_prefix":"aaa26d595f56f023","entry":"default_pre_forward_process","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aaa26d595f56f023"}},{"code_sha256_prefix":"d6b164151ea048bc","entry":"get_forward_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6b164151ea048bc"}},{"code_sha256_prefix":"c2780237c1682ad8","entry":"get_post_epoch_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2780237c1682ad8"}},{"code_sha256_prefix":"e78710c20c16f470","entry":"get_post_forward_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e78710c20c16f470"}},{"code_sha256_prefix":"d75c1131fcde7500","entry":"get_pre_epoch_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d75c1131fcde7500"}},{"code_sha256_prefix":"7f13327e06b694aa","entry":"get_pre_forward_proc_func","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f13327e06b694aa"}},{"code_sha256_prefix":"3933a969d27c0f3a","entry":"update_io_dict","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3933a969d27c0f3a"}},{"code_sha256_prefix":"0c2e416b9fd8f9e7","entry":"wrap_model","repo":"yoshitomo-matsubara/torchdistill","repo_kind":"official","path":"torchdistill/core/distillation.py","file_url":"https://github.com/yoshitomo-matsubara/torchdistill/blob/HEAD/torchdistill/core/distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c2e416b9fd8f9e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}