{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optex-expediting-first-order-optimization","title":"OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations","arxiv_id":"2402.11427","date":"2024-02-18","proceeding":null,"authors":["Yao Shu","Jiongfeng Fang","Ying Tiffany He","Fei Richard Yu"],"abstract":"First-order optimization (FOO) algorithms are pivotal in numerous computational domains such as machine learning and signal denoising. However, their application to complex tasks like neural network training often entails significant inefficiencies due to the need for many sequential iterations for convergence. In response, we introduce first-order optimization expedited with approximately parallelized iterations (OptEx), the first framework that enhances the efficiency of FOO by leveraging parallel computing to mitigate its iterative bottleneck. OptEx employs kernelized gradient estimation to make use of gradient history for future gradient prediction, enabling parallelization of iterations -- a strategy once considered impractical because of the inherent iterative dependency in FOO. We provide theoretical guarantees for the reliability of our kernelized gradient estimation and the iteration complexity of SGD-based OptEx, confirming that estimation errors diminish to zero as historical gradients accumulate and that SGD-based OptEx enjoys an effective acceleration rate of $\\Omega(\\sqrt{N})$ over standard SGD given parallelism of N. We also use extensive empirical studies, including synthetic functions, reinforcement learning tasks, and neural network training across various datasets, to underscore the substantial efficiency improvements achieved by OptEx.","url_abs":"https://arxiv.org/abs/2402.11427v2","url_pdf":"https://arxiv.org/pdf/2402.11427v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optex-expediting-first-order-optimization","repo_url":"https://github.com/youyve/OptEx","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.11427","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11427"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/youyve/OptEx","reach":null}],"summary":{"ran_violates":1,"ran":1,"ran_draft_wrong":3,"ran_fixture":2},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"99b2c601b3157cc1","entry":"cosine_similarity","repo":"youyve/optex","repo_kind":"official","path":"optex_jax/OptEx_JAX.py","file_url":"https://github.com/youyve/optex/blob/HEAD/optex_jax/OptEx_JAX.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99b2c601b3157cc1"}},{"code_sha256_prefix":"6711451b16a989df","entry":"OptEx","repo":"youyve/OptEx","repo_kind":"official","path":"optex/optex.py","file_url":"https://github.com/youyve/OptEx/blob/HEAD/optex/optex.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6711451b16a989df"}},{"code_sha256_prefix":"de95a5e71990cbdb","entry":"ackley_function","repo":"youyve/optex","repo_kind":"official","path":"examples/test_optex.py","file_url":"https://github.com/youyve/optex/blob/HEAD/examples/test_optex.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de95a5e71990cbdb"}},{"code_sha256_prefix":"f9758c897165ff0f","entry":"get_proxy_grad_func","repo":"youyve/optex","repo_kind":"official","path":"optex_jax/OptEx_JAX.py","file_url":"https://github.com/youyve/optex/blob/HEAD/optex_jax/OptEx_JAX.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9758c897165ff0f"}},{"code_sha256_prefix":"1252c181b90235ec","entry":"rosenbrock_function","repo":"youyve/optex","repo_kind":"official","path":"examples/test_optex.py","file_url":"https://github.com/youyve/optex/blob/HEAD/examples/test_optex.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1252c181b90235ec"}},{"code_sha256_prefix":"df1f3211b0f748b6","entry":"sphere_function","repo":"youyve/optex","repo_kind":"official","path":"examples/test_optex.py","file_url":"https://github.com/youyve/optex/blob/HEAD/examples/test_optex.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df1f3211b0f748b6"}},{"code_sha256_prefix":"acfeda556d2c7bab","entry":"tuning_mattern","repo":"youyve/optex","repo_kind":"official","path":"optex_jax/OptEx_JAX.py","file_url":"https://github.com/youyve/optex/blob/HEAD/optex_jax/OptEx_JAX.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acfeda556d2c7bab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}