{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/regularized-adaptive-momentum-dual-averaging","title":"Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network","arxiv_id":"2403.14398","date":"2024-03-21","proceeding":null,"authors":["Zih-Syuan Huang","Ching-pei Lee"],"abstract":"We propose a Regularized Adaptive Momentum Dual Averaging (RAMDA) algorithm for training structured neural networks. Similar to existing regularized adaptive methods, the subproblem for computing the update direction of RAMDA involves a nonsmooth regularizer and a diagonal preconditioner, and therefore does not possess a closed-form solution in general. We thus also carefully devise an implementable inexactness condition that retains convergence guarantees similar to the exact versions, and propose a companion efficient solver for the subproblems of both RAMDA and existing methods to make them practically feasible. We leverage the theory of manifold identification in variational analysis to show that, even in the presence of such inexactness, the iterates of RAMDA attain the ideal structure induced by the regularizer at the stationary point of asymptotic convergence. This structure is locally optimal near the point of convergence, so RAMDA is guaranteed to obtain the best structure possible among all methods converging to the same point, making it the first regularized adaptive method outputting models that possess outstanding predictive performance while being (locally) optimally structured. Extensive numerical experiments in large-scale modern computer vision, language modeling, and speech tasks show that the proposed RAMDA is efficient and consistently outperforms state of the art for training structured neural network. Implementation of our algorithm is available at https://www.github.com/ismoptgroup/RAMDA/.","url_abs":"https://arxiv.org/abs/2403.14398v2","url_pdf":"https://arxiv.org/pdf/2403.14398v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"regularized-adaptive-momentum-dual-averaging","repo_url":"https://github.com/ismoptgroup/ramda","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"regularized-adaptive-momentum-dual-averaging","repo_url":"https://github.com/ismoptgroup/ramda_exp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.14398","atlas_url":"https://app.syntology.ai/?focus=2403.14398","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.14398"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ismoptgroup/RAMDA","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ismoptgroup/ramda","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ismoptgroup/ramda_exp","reach":null}],"summary":{"ran":4,"ran_fixture":7,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"73fdb5d7e2ed91da","entry":"RAMDA","repo":"ismoptgroup/ramda_exp","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda_exp/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73fdb5d7e2ed91da"}},{"code_sha256_prefix":"347bd076390d800a","entry":"_3d_to_4d","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"347bd076390d800a"}},{"code_sha256_prefix":"d4f21a337fadf91d","entry":"_4d_to_3d","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4f21a337fadf91d"}},{"code_sha256_prefix":"31290575c2ad4e3e","entry":"group_model","repo":"ismoptgroup/RAMDA","repo_kind":"official","path":"Core/group.py","file_url":"https://github.com/ismoptgroup/RAMDA/blob/HEAD/Core/group.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31290575c2ad4e3e"}},{"code_sha256_prefix":"8f35b83115cafdab","entry":"is_4d","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f35b83115cafdab"}},{"code_sha256_prefix":"a445e118c7ac9c43","entry":"pgd","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a445e118c7ac9c43"}},{"code_sha256_prefix":"17df8509337a0c9b","entry":"prox_glasso","repo":"ismoptgroup/ramda_exp","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda_exp/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17df8509337a0c9b"}},{"code_sha256_prefix":"34920252c14342fe","entry":"prox_group_lasso","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34920252c14342fe"}},{"code_sha256_prefix":"694f00802cc3ca3d","entry":"prox_nuclear","repo":"ismoptgroup/ramda_exp","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda_exp/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"694f00802cc3ca3d"}},{"code_sha256_prefix":"8bacd8274fd5963f","entry":"prox_nuclear_norm","repo":"ismoptgroup/RAMDA","repo_kind":"official","path":"Core/prox_fns.py","file_url":"https://github.com/ismoptgroup/RAMDA/blob/HEAD/Core/prox_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bacd8274fd5963f"}},{"code_sha256_prefix":"f5f94494634e3836","entry":"reweight_fn","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5f94494634e3836"}},{"code_sha256_prefix":"67c24c64abe117e1","entry":"subproblem_objective","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67c24c64abe117e1"}},{"code_sha256_prefix":"60acd49acf9d0a09","entry":"RAMDA","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60acd49acf9d0a09"}},{"code_sha256_prefix":"b17a2749515de759","entry":"pgd_solver_glasso","repo":"ismoptgroup/ramda_exp","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda_exp/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b17a2749515de759"}},{"code_sha256_prefix":"2ac3ac6563e02e1e","entry":"pgd_solver_group_lasso","repo":"ismoptgroup/ramda","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ac3ac6563e02e1e"}},{"code_sha256_prefix":"3f6f1a2fc10c3c36","entry":"pgd_solver_nuclear","repo":"ismoptgroup/ramda_exp","repo_kind":"official","path":"Core/optimizer.py","file_url":"https://github.com/ismoptgroup/ramda_exp/blob/HEAD/Core/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f6f1a2fc10c3c36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}