{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bidirectional-looking-with-a-novel-double","title":"Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers","arxiv_id":"2307.00631","date":"2023-07-02","proceeding":null,"authors":["Yineng Chen","Zuchao Li","Lefei Zhang","Bo Du","Hai Zhao"],"abstract":"Optimizer is an essential component for the success of deep learning, which guides the neural network to update the parameters according to the loss on the training set. SGD and Adam are two classical and effective optimizers on which researchers have proposed many variants, such as SGDM and RAdam. In this paper, we innovatively combine the backward-looking and forward-looking aspects of the optimizer algorithm and propose a novel \\textsc{Admeta} (\\textbf{A} \\textbf{D}ouble exponential \\textbf{M}oving averag\\textbf{E} \\textbf{T}o \\textbf{A}daptive and non-adaptive momentum) optimizer framework. For backward-looking part, we propose a DEMA variant scheme, which is motivated by a metric in the stock market, to replace the common exponential moving average scheme. While in the forward-looking part, we present a dynamic lookahead strategy which asymptotically approaches a set value, maintaining its speed at early stage and high convergence performance at final stage. Based on this idea, we provide two optimizer implementations, \\textsc{AdmetaR} and \\textsc{AdmetaS}, the former based on RAdam and the latter based on SGDM. Through extensive experiments on diverse tasks, we find that the proposed \\textsc{Admeta} optimizer outperforms our base optimizers and shows advantages over recently proposed competitive optimizers. We also provide theoretical proof of these two algorithms, which verifies the convergence of our proposed \\textsc{Admeta}.","url_abs":"https://arxiv.org/abs/2307.00631v1","url_pdf":"https://arxiv.org/pdf/2307.00631v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bidirectional-looking-with-a-novel-double","repo_url":"https://github.com/chernyn/admeta-optimizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"radam","method_name":"RAdam"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.00631","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.00631"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chernyn/admeta-optimizer","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ebe71d5f25ff7ba","entry":"create_initializer","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/models/initializer.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/models/initializer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6ebe71d5f25ff7ba"}},{"code_sha256_prefix":"bc8c7f73dc3f1039","entry":"cross_entropy_loss","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/losses/label_smoothing.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/losses/label_smoothing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bc8c7f73dc3f1039"}},{"code_sha256_prefix":"72e2bbc696d16422","entry":"cutmix","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/collators/cutmix.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/collators/cutmix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"72e2bbc696d16422"}},{"code_sha256_prefix":"5617086f7f208033","entry":"mixup","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/collators/mixup.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/collators/mixup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5617086f7f208033"}},{"code_sha256_prefix":"fc5b24dc25537ab8","entry":"onehot_encoding","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/losses/label_smoothing.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/losses/label_smoothing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fc5b24dc25537ab8"}},{"code_sha256_prefix":"07c163e0bbda72d3","entry":"ricap","repo":"chernyn/admeta-optimizer","repo_kind":"official","path":"admeta-code/admeta-cifar/pytorch_image_classification/collators/ricap.py","file_url":"https://github.com/chernyn/admeta-optimizer/blob/HEAD/admeta-code/admeta-cifar/pytorch_image_classification/collators/ricap.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"07c163e0bbda72d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}