{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/switchable-online-knowledge-distillation","title":"Switchable Online Knowledge Distillation","arxiv_id":"2209.04996","date":"2022-09-12","proceeding":null,"authors":["Biao Qian","Yang Wang","Hongzhi Yin","Richang Hong","Meng Wang"],"abstract":"Online Knowledge Distillation (OKD) improves the involved models by reciprocally exploiting the difference between teacher and student. Several crucial bottlenecks over the gap between them -- e.g., Why and when does a large gap harm the performance, especially for student? How to quantify the gap between teacher and student? -- have received limited formal study. In this paper, we propose Switchable Online Knowledge Distillation (SwitOKD), to answer these questions. Instead of focusing on the accuracy gap at test phase by the existing arts, the core idea of SwitOKD is to adaptively calibrate the gap at training phase, namely distillation gap, via a switching strategy between two modes -- expert mode (pause the teacher while keep the student learning) and learning mode (restart the teacher). To possess an appropriate distillation gap, we further devise an adaptive switching threshold, which provides a formal criterion as to when to switch to learning mode or expert mode, and thus improves the student's performance. Meanwhile, the teacher benefits from our adaptive switching threshold and keeps basically on a par with other online arts. We further extend SwitOKD to multiple networks with two basis topologies. Finally, extensive experiments and analysis validate the merits of SwitOKD for classification over the state-of-the-arts. Our code is available at https://github.com/hfutqian/SwitOKD.","url_abs":"https://arxiv.org/abs/2209.04996v1","url_pdf":"https://arxiv.org/pdf/2209.04996v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"switchable-online-knowledge-distillation","repo_url":"https://github.com/hfutqian/SwitOKD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.04996","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.04996"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hfutqian/SwitOKD","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran":4,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"00e569acd6b45ef0","entry":"conv3x3","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/CIFAR-10/WRN-16-1_8/models/wrnet16_1.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/CIFAR-10/WRN-16-1_8/models/wrnet16_1.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00e569acd6b45ef0"}},{"code_sha256_prefix":"7765583b9e540679","entry":"conv_1x1_bn","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7765583b9e540679"}},{"code_sha256_prefix":"2f7853ff01cbbc29","entry":"conv_bn","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f7853ff01cbbc29"}},{"code_sha256_prefix":"403277d0bfbb477b","entry":"find_classes","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"403277d0bfbb477b"}},{"code_sha256_prefix":"ab4109634b75ef8b","entry":"is_image_file","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab4109634b75ef8b"}},{"code_sha256_prefix":"27e7954cc9b19461","entry":"loadPILImage","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/Tiny-ImageNet/Train/SwitOKD/tiny_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27e7954cc9b19461"}},{"code_sha256_prefix":"421fd14c036bc279","entry":"mobilenetv2_T_w","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/CIFAR-100/WRN16-2_0.5mobilenetv2/models/mobilenetv2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"421fd14c036bc279"}},{"code_sha256_prefix":"9e7bf674714e06d5","entry":"dumps_pyarrow","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e7bf674714e06d5"}},{"code_sha256_prefix":"b4c154b6f0568b6b","entry":"loads_pyarrow","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4c154b6f0568b6b"}},{"code_sha256_prefix":"51738010a21f76a1","entry":"raw_reader","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51738010a21f76a1"}},{"code_sha256_prefix":"b5d3e2b97d86d2dd","entry":"resnet18","repo":"hfutqian/SwitOKD","repo_kind":"official","path":"SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","file_url":"https://github.com/hfutqian/SwitOKD/blob/HEAD/SwitOKD_code/ImageNet/ResNet18_34/models/resnet18.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5d3e2b97d86d2dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}