{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/boosting-residual-networks-with-group","title":"Boosting Residual Networks with Group Knowledge","arxiv_id":"2308.13772","date":"2023-08-26","proceeding":null,"authors":["Shengji Tang","Peng Ye","Baopu Li","Weihao Lin","Tao Chen","Tong He","Chong Yu","Wanli Ouyang"],"abstract":"Recent research understands the residual networks from a new perspective of the implicit ensemble model. From this view, previous methods such as stochastic depth and stimulative training have further improved the performance of the residual network by sampling and training of its subnets. However, they both use the same supervision for all subnets of different capacities and neglect the valuable knowledge generated by subnets during training. In this manuscript, we mitigate the significant knowledge distillation gap caused by using the same kind of supervision and advocate leveraging the subnets to provide diverse knowledge. Based on this motivation, we propose a group knowledge based training framework for boosting the performance of residual networks. Specifically, we implicitly divide all subnets into hierarchical groups by subnet-in-subnet sampling, aggregate the knowledge of different subnets in each group during training, and exploit upper-level group knowledge to supervise lower-level subnet groups. Meanwhile, We also develop a subnet sampling strategy that naturally samples larger subnets, which are found to be more helpful than smaller subnets in boosting performance for hierarchical groups. Compared with typical subnet training and other methods, our method achieves the best efficiency and performance trade-offs on multiple datasets and network structures. The code is at https://github.com/tsj-001/AAAI24-GKT.","url_abs":"https://arxiv.org/abs/2308.13772v2","url_pdf":"https://arxiv.org/pdf/2308.13772v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"boosting-residual-networks-with-group","repo_url":"https://github.com/tsj-001/aaai24-gkt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.13772","atlas_url":"https://app.syntology.ai/?focus=2308.13772","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.13772"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsj-001/aaai24-gkt","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":2,"ran_draft_wrong":2,"ran":3,"ran_violates":2},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5c1da0fe7a052b06","entry":"cast_tuple","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/pit_ddp.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/pit_ddp.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c1da0fe7a052b06"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/big_resnet.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/big_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/big_resnet.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/big_resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"cc07f242a3837d36","entry":"conv_output_size","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/pit_ddp.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/pit_ddp.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc07f242a3837d36"}},{"code_sha256_prefix":"f3c67c589b630c22","entry":"drop_path","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/cait_ddp.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/cait_ddp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3c67c589b630c22"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/cait_ddp.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/cait_ddp.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"012a56f3a60ed47b","entry":"make_divisible","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/big_resnet_ddw.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/big_resnet_ddw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"012a56f3a60ed47b"}},{"code_sha256_prefix":"c8b5501808912cf3","entry":"make_eight_divisible","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/big_resnet_ddw.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/big_resnet_ddw.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8b5501808912cf3"}},{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"tsj-001/aaai24-gkt","repo_kind":"official","path":"gkt/model_bank/cait_ddp.py","file_url":"https://github.com/tsj-001/aaai24-gkt/blob/HEAD/gkt/model_bank/cait_ddp.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}