{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/apollo-an-adaptive-parameter-wise-diagonal","title":"Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization","arxiv_id":"2009.13586","date":"2020-09-28","proceeding":null,"authors":["Xuezhe Ma"],"abstract":"In this paper, we introduce Apollo, a quasi-Newton method for nonconvex stochastic optimization, which dynamically incorporates the curvature of the loss function by approximating the Hessian via a diagonal matrix. Importantly, the update and storage of the diagonal approximation of Hessian is as efficient as adaptive first-order optimization methods with linear complexity for both time and memory. To handle nonconvexity, we replace the Hessian with its rectified absolute value, which is guaranteed to be positive-definite. Experiments on three tasks of vision and language show that Apollo achieves significant improvements over other stochastic optimization methods, including SGD and variants of Adam, in term of both convergence speed and generalization performance. The implementation of the algorithm is available at https://github.com/XuezheMax/apollo.","url_abs":"https://arxiv.org/abs/2009.13586v6","url_pdf":"https://arxiv.org/pdf/2009.13586v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"apollo-an-adaptive-parameter-wise-diagonal","repo_url":"https://github.com/XuezheMax/apollo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"apollo-an-adaptive-parameter-wise-diagonal","repo_url":"https://github.com/XuezheMax/fairseq-apollo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"apollo-an-adaptive-parameter-wise-diagonal","repo_url":"https://github.com/evanatyourservice/Apollo-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"apollo","method_name":"Apollo"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[{"slug":"apollo","name":"Apollo","full_name":"Adaptive Parameter-wise Diagonal Quasi-Newton Method"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.13586","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.13586"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XuezheMax/fairseq-apollo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/XuezheMax/apollo","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/evanatyourservice/Apollo-tf","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9999fec2e6f3632f","entry":"is_distributed","repo":"XuezheMax/apollo","repo_kind":"official","path":"classification/imagenet.py","file_url":"https://github.com/XuezheMax/apollo/blob/HEAD/classification/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9999fec2e6f3632f"}},{"code_sha256_prefix":"20afae6213308bf1","entry":"is_master","repo":"XuezheMax/apollo","repo_kind":"official","path":"classification/imagenet.py","file_url":"https://github.com/XuezheMax/apollo/blob/HEAD/classification/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20afae6213308bf1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}