{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stimulative-training-go-beyond-the","title":"Stimulative Training++: Go Beyond The Performance Limits of Residual Networks","arxiv_id":"2305.02507","date":"2023-05-04","proceeding":null,"authors":["Peng Ye","Tong He","Shengji Tang","Baopu Li","Tao Chen","Lei Bai","Wanli Ouyang"],"abstract":"Residual networks have shown great success and become indispensable in recent deep neural network models. In this work, we aim to re-investigate the training process of residual networks from a novel social psychology perspective of loafing, and further propose a new training scheme as well as three improved strategies for boosting residual networks beyond their performance limits. Previous research has suggested that residual networks can be considered as ensembles of shallow networks, which implies that the final performance of a residual network is influenced by a group of subnetworks. We identify a previously overlooked problem that is analogous to social loafing, where subnetworks within a residual network are prone to exert less effort when working as part of a group compared to working alone. We define this problem as \\textit{network loafing}. Similar to the decreased individual productivity and overall performance as demonstrated in society, network loafing inevitably causes sub-par performance. Inspired by solutions from social psychology, we first propose a novel training scheme called stimulative training, which randomly samples a residual subnetwork and calculates the KL divergence loss between the sampled subnetwork and the given residual network for extra supervision. In order to unleash the potential of stimulative training, we further propose three simple-yet-effective strategies, including a novel KL- loss that only aligns the network logits direction, random smaller inputs for subnetworks, and inter-stage sampling rules. Comprehensive experiments and analysis verify the effectiveness of stimulative training as well as its three improved strategies.","url_abs":"https://arxiv.org/abs/2305.02507v1","url_pdf":"https://arxiv.org/pdf/2305.02507v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stimulative-training-go-beyond-the","repo_url":"https://github.com/sunshine-ye/nips22-st","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.02507","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.02507"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sunshine-ye/nips22-st","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":3,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e4c841a535ce14d5","entry":"CutoutDefault","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"aug_lib.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/aug_lib.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4c841a535ce14d5"}},{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"models/pyramidnet_randwidth.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/models/pyramidnet_randwidth.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"6af95ebe99af2e36","entry":"conv3x3","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"models/resnet_randdepth.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/models/resnet_randdepth.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6af95ebe99af2e36"}},{"code_sha256_prefix":"012a56f3a60ed47b","entry":"make_divisible","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"models/randwidth_ops.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/models/randwidth_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"012a56f3a60ed47b"}},{"code_sha256_prefix":"8733e0d87f232d11","entry":"mixup_target","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"mixup.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/mixup.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8733e0d87f232d11"}},{"code_sha256_prefix":"4728dc2025227bba","entry":"one_hot","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"mixup.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/mixup.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4728dc2025227bba"}},{"code_sha256_prefix":"3913c620f6926f42","entry":"rand_bbox","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"mixup.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/mixup.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3913c620f6926f42"}},{"code_sha256_prefix":"3434fff9b5d4912e","entry":"float_parameter","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"aug_lib.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/aug_lib.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3434fff9b5d4912e"}},{"code_sha256_prefix":"84a10f5778846b16","entry":"get_logger","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"utils/setlogger.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/utils/setlogger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84a10f5778846b16"}},{"code_sha256_prefix":"b3a390f777b2388c","entry":"int_parameter","repo":"sunshine-ye/nips22-st","repo_kind":"official","path":"aug_lib.py","file_url":"https://github.com/sunshine-ye/nips22-st/blob/HEAD/aug_lib.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a390f777b2388c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}