{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automated-progressive-learning-for-efficient","title":"Automated Progressive Learning for Efficient Training of Vision Transformers","arxiv_id":"2203.14509","date":"2022-03-28","proceeding":"CVPR 2022 1","authors":["Changlin Li","Bohan Zhuang","Guangrun Wang","Xiaodan Liang","Xiaojun Chang","Yi Yang"],"abstract":"Recent advances in vision Transformers (ViTs) have come with a voracious appetite for computing power, high-lighting the urgent need to develop efficient training methods for ViTs. Progressive learning, a training scheme where the model capacity grows progressively during training, has started showing its ability in efficient training. In this paper, we take a practical step towards efficient training of ViTs by customizing and automating progressive learning. First, we develop a strong manual baseline for progressive learning of ViTs, by introducing momentum growth (MoGrow) to bridge the gap brought by model growth. Then, we propose automated progressive learning (AutoProg), an efficient training scheme that aims to achieve lossless acceleration by automatically increasing the training overload on-the-fly; this is achieved by adaptively deciding whether, where and how much should the model grow during progressive learning. Specifically, we first relax the optimization of the growth schedule to sub-network architecture optimization problem, then propose one-shot estimation of the sub-network performance via an elastic supernet. The searching overhead is reduced to minimal by recycling the parameters of the supernet. Extensive experiments of efficient training on ImageNet with two representative ViT models, DeiT and VOLO, demonstrate that AutoProg can accelerate ViTs training by up to 85.1% with no performance drop. Code: https://github.com/changlin31/AutoProg","url_abs":"https://arxiv.org/abs/2203.14509v1","url_pdf":"https://arxiv.org/pdf/2203.14509v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automated-progressive-learning-for-efficient","repo_url":"https://github.com/changlin31/autoprog","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"automated-progressive-learning-for-efficient","repo_url":"https://github.com/changlin31/autoprog-zero","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"deit","method_name":"DeiT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.14509","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.14509"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/changlin31/autoprog-zero","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/changlin31/autoprog","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_honours":1,"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1},"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"c6689f2469faefd0","entry":"SmoothMeter","repo":"changlin31/autoprog-zero","repo_kind":"listed","path":"train_apz.py","file_url":"https://github.com/changlin31/autoprog-zero/blob/HEAD/train_apz.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6689f2469faefd0"}},{"code_sha256_prefix":"24400bc088ed228d","entry":"make_divisible","repo":"changlin31/autoprog","repo_kind":"official","path":"prog/progressive.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/prog/progressive.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24400bc088ed228d"}},{"code_sha256_prefix":"ba7779c554538760","entry":"outlooker_blocks","repo":"changlin31/autoprog","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/models/volo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba7779c554538760"}},{"code_sha256_prefix":"c42ac1c1628c699d","entry":"auto_grow","repo":"changlin31/autoprog-zero","repo_kind":"listed","path":"train_apz.py","file_url":"https://github.com/changlin31/autoprog-zero/blob/HEAD/train_apz.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c42ac1c1628c699d"}},{"code_sha256_prefix":"87a34d0211098d18","entry":"get_block","repo":"changlin31/autoprog","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/models/volo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"87a34d0211098d18"}},{"code_sha256_prefix":"453b65e88ebc0a36","entry":"get_resume_epoch","repo":"changlin31/autoprog","repo_kind":"official","path":"prog/helpers.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/prog/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"453b65e88ebc0a36"}},{"code_sha256_prefix":"6725eb03753f6554","entry":"load_state_dict","repo":"changlin31/autoprog","repo_kind":"official","path":"prog/helpers.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/prog/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6725eb03753f6554"}},{"code_sha256_prefix":"ec909a92ce99e3b8","entry":"progressive_schedule","repo":"changlin31/autoprog","repo_kind":"official","path":"prog/progressive.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/prog/progressive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec909a92ce99e3b8"}},{"code_sha256_prefix":"b66088e90761117d","entry":"rand_bbox","repo":"changlin31/autoprog","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/models/volo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b66088e90761117d"}},{"code_sha256_prefix":"fbde52072d8c3118","entry":"resume_checkpoint","repo":"changlin31/autoprog","repo_kind":"official","path":"prog/helpers.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/prog/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fbde52072d8c3118"}},{"code_sha256_prefix":"a8dddad99b3c4841","entry":"train_one_epoch_super","repo":"changlin31/autoprog-zero","repo_kind":"listed","path":"train_apz.py","file_url":"https://github.com/changlin31/autoprog-zero/blob/HEAD/train_apz.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8dddad99b3c4841"}},{"code_sha256_prefix":"c3177cb184234e06","entry":"validate","repo":"changlin31/autoprog","repo_kind":"official","path":"main_prog.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/main_prog.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3177cb184234e06"}},{"code_sha256_prefix":"1bb06b1e2166bced","entry":"validate_trainset","repo":"changlin31/autoprog","repo_kind":"official","path":"main_prog.py","file_url":"https://github.com/changlin31/autoprog/blob/HEAD/main_prog.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bb06b1e2166bced"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}