{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trainable-weight-averaging-for-fast","title":"Trainable Weight Averaging: A General Approach for Subspace Training","arxiv_id":"2205.13104","date":"2022-05-26","proceeding":null,"authors":["Tao Li","Zhehao Huang","Yingwen Wu","Zhengbao He","Qinghua Tao","Xiaolin Huang","Chih-Jen Lin"],"abstract":"Training deep neural networks (DNNs) in low-dimensional subspaces is a promising direction for achieving efficient training and better generalization performance. Our previous work extracts the subspaces by performing the dimension reduction method over the training trajectory, which verifies that DNN could be well-trained in a tiny subspace. However, that method is inefficient for subspace extraction and numerically unstable, limiting its applicability to more general tasks. In this paper, we connect subspace training to weight averaging and propose \\emph{Trainable Weight Averaging} (TWA), a general approach for subspace training. TWA is efficient in terms of subspace extraction and easy to use, making it a promising new optimizer for DNN's training. Our design also includes an efficient scheme that allows parallel training across multiple nodes to handle large-scale problems and evenly distribute the memory and computation burden to each node. TWA can be used for both efficient training and generalization enhancement, for different neural network architectures, and for various tasks from image classification and object detection, to neural language processing. The code of implementation is available at https://github.com/nblt/TWA, which includes extensive experiments covering various benchmark computer vision and neural language processing tasks with various architectures.","url_abs":"https://arxiv.org/abs/2205.13104v3","url_pdf":"https://arxiv.org/pdf/2205.13104v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trainable-weight-averaging-for-fast","repo_url":"https://github.com/nblt/twa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"},{"task_slug":"efficient-neural-network","task_name":"Efficient Neural Network"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"stochastic-weight-averaging","method_name":"Stochastic Weight Averaging"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.13104","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.13104"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/nblt/TWA","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlfoundations/model-soups","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nblt/twa","reach":null}],"summary":{"ran_honours":4,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d9e78073505760c","entry":"get_model_param_vec","repo":"nblt/TWA","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/TWA/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d9e78073505760c"}},{"code_sha256_prefix":"7f894ad74097173f","entry":"get_model_param_vec_torch","repo":"nblt/TWA","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/TWA/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f894ad74097173f"}},{"code_sha256_prefix":"14ebf34e2f003912","entry":"adjust_learning_rate","repo":"nblt/twa","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/twa/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14ebf34e2f003912"}},{"code_sha256_prefix":"e8e9688a1aa24874","entry":"get_model_grad_vec","repo":"nblt/TWA","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/TWA/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8e9688a1aa24874"}},{"code_sha256_prefix":"63990b8c88b3c8f7","entry":"get_model_grad_vec","repo":"nblt/twa","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/twa/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63990b8c88b3c8f7"}},{"code_sha256_prefix":"3ba4f28978b774dd","entry":"P_SGD","repo":"nblt/twa","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/twa/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ba4f28978b774dd"}},{"code_sha256_prefix":"4a023162808ea147","entry":"train","repo":"nblt/twa","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/twa/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a023162808ea147"}},{"code_sha256_prefix":"e502b62be0b7d826","entry":"update_grad","repo":"nblt/twa","repo_kind":"official","path":"train_twa.py","file_url":"https://github.com/nblt/twa/blob/HEAD/train_twa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e502b62be0b7d826"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}