{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stable-nonconvex-nonconcave-training-via-1","title":"Stable Nonconvex-Nonconcave Training via Linear Interpolation","arxiv_id":"2310.13459","date":"2023-10-20","proceeding":"NeurIPS 2023 11","authors":["Thomas Pethick","Wanyun Xie","Volkan Cevher"],"abstract":"This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear interpolation can help by leveraging the theory of nonexpansive operators. We construct a new optimization scheme called relaxed approximate proximal point (RAPP), which is the first explicit method without anchoring to achieve last iterate convergence rates for $\\rho$-comonotone problems while only requiring $\\rho > -\\tfrac{1}{2L}$. The construction extends to constrained and regularized settings. By replacing the inner optimizer in RAPP we rediscover the family of Lookahead algorithms for which we establish convergence in cohypomonotone problems even when the base optimizer is taken to be gradient descent ascent. The range of cohypomonotone problems in which Lookahead converges is further expanded by exploiting that Lookahead inherits the properties of the base optimizer. We corroborate the results with experiments on generative adversarial networks which demonstrates the benefits of the linear interpolation present in both RAPP and Lookahead.","url_abs":"https://arxiv.org/abs/2310.13459v4","url_pdf":"https://arxiv.org/pdf/2310.13459v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stable-nonconvex-nonconcave-training-via-1","repo_url":"https://github.com/LIONS-EPFL/linear-interpolation","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"lookahead","method_name":"Lookahead"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.13459","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.13459"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LIONS-EPFL/linear-interpolation","reach":{"status":"ok"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"a3071df5e17ff3a1","entry":"define_model_loss","repo":"LIONS-EPFL/linear-interpolation","repo_kind":"official","path":"rapp/losses.py","file_url":"https://github.com/LIONS-EPFL/linear-interpolation/blob/HEAD/rapp/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3071df5e17ff3a1"}},{"code_sha256_prefix":"e32a5060b0d5785b","entry":"tensorify","repo":"LIONS-EPFL/linear-interpolation","repo_kind":"official","path":"rapp/models/utils.py","file_url":"https://github.com/LIONS-EPFL/linear-interpolation/blob/HEAD/rapp/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e32a5060b0d5785b"}},{"code_sha256_prefix":"66e44e5708005e29","entry":"tensorify_grad","repo":"LIONS-EPFL/linear-interpolation","repo_kind":"official","path":"rapp/models/utils.py","file_url":"https://github.com/LIONS-EPFL/linear-interpolation/blob/HEAD/rapp/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66e44e5708005e29"}},{"code_sha256_prefix":"8e2be65057bbefbd","entry":"vangan_loss_dis","repo":"LIONS-EPFL/linear-interpolation","repo_kind":"official","path":"rapp/losses.py","file_url":"https://github.com/LIONS-EPFL/linear-interpolation/blob/HEAD/rapp/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e2be65057bbefbd"}},{"code_sha256_prefix":"8614eb30aa3b4b78","entry":"vangan_loss_gen","repo":"LIONS-EPFL/linear-interpolation","repo_kind":"official","path":"rapp/losses.py","file_url":"https://github.com/LIONS-EPFL/linear-interpolation/blob/HEAD/rapp/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8614eb30aa3b4b78"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}