{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adopt-modified-adam-can-converge-with-any-b-2","title":"ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate","arxiv_id":"2411.02853","date":"2024-11-05","proceeding":null,"authors":["Shohei Taniguchi","Keno Harada","Gouki Minegishi","Yuta Oshima","Seong Cheol Jeong","Go Nagahara","Tomoshi Iiyama","Masahiro Suzuki","Yusuke Iwasawa","Yutaka Matsuo"],"abstract":"Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., $\\beta_2$, in a problem-dependent manner. There have been many attempts to fix the non-convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise is uniformly bounded. In this paper, we propose a new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of $\\mathcal{O} ( 1 / \\sqrt{T} )$ with any choice of $\\beta_2$ without depending on the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and the normalization by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves superior results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning. The implementation is available at https://github.com/iShohei220/adopt.","url_abs":"https://arxiv.org/abs/2411.02853v3","url_pdf":"https://arxiv.org/pdf/2411.02853v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adopt-modified-adam-can-converge-with-any-b-2","repo_url":"https://github.com/ishohei220/adopt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"adopt-modified-adam-can-converge-with-any-b-2","repo_url":"https://github.com/huggingface/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adopt","method_name":"ADOPT"},{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[{"slug":"adopt","name":"ADOPT","full_name":"ADaptive gradient method with the OPTimal convergence rate"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.02853","atlas_url":"https://app.syntology.ai/?focus=2411.02853","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02853"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/NVlabs/NVAE","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/iShohei220/adopt","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/UiPath/torchvision","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/pytorch-image-models","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ishohei220/adopt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":1,"ran_draft_wrong":2,"unverified":19},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1},"found_in_text":{"samples":18,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d95e28fac98427f","entry":"accuracy","repo":"iShohei220/adopt","repo_kind":"official","path":"imagenet/utils.py","file_url":"https://github.com/iShohei220/adopt/blob/HEAD/imagenet/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d95e28fac98427f"}},{"code_sha256_prefix":"cf7081fc34608ea3","entry":"channel_shuffle","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/shufflenetv2.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/shufflenetv2.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cf7081fc34608ea3"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/resnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/resnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"8ccdf1f64321d691","entry":"alexnet","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/alexnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/alexnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8ccdf1f64321d691"}},{"code_sha256_prefix":"4eca7815147e1708","entry":"average_checkpoints","repo":"iShohei220/adopt","repo_kind":"official","path":"imagenet/utils.py","file_url":"https://github.com/iShohei220/adopt/blob/HEAD/imagenet/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4eca7815147e1708"}},{"code_sha256_prefix":"9eb4b3b03ed7a1c9","entry":"densenet121","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/densenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9eb4b3b03ed7a1c9"}},{"code_sha256_prefix":"e945c7af558f65a9","entry":"densenet161","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/densenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e945c7af558f65a9"}},{"code_sha256_prefix":"858b62a0718acf3c","entry":"densenet169","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/densenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"858b62a0718acf3c"}},{"code_sha256_prefix":"e36bcfb158ae8f9c","entry":"get_args_parser","repo":"iShohei220/adopt","repo_kind":"official","path":"imagenet/train_quantization.py","file_url":"https://github.com/iShohei220/adopt/blob/HEAD/imagenet/train_quantization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e36bcfb158ae8f9c"}},{"code_sha256_prefix":"b2ea55703711dd12","entry":"get_module","repo":"iShohei220/adopt","repo_kind":"official","path":"imagenet/presets.py","file_url":"https://github.com/iShohei220/adopt/blob/HEAD/imagenet/presets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b2ea55703711dd12"}},{"code_sha256_prefix":"cc86d582f7c75a6d","entry":"googlenet","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/googlenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/googlenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cc86d582f7c75a6d"}},{"code_sha256_prefix":"d5f1aa4870d7ad8b","entry":"inception_v3","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/inception.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/inception.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d5f1aa4870d7ad8b"}},{"code_sha256_prefix":"fd93b84b5a89ac58","entry":"load_meta_file","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/datasets/imagenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/datasets/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fd93b84b5a89ac58"}},{"code_sha256_prefix":"095cb1dfd64f93c1","entry":"mnasnet0_5","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/mnasnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/mnasnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"095cb1dfd64f93c1"}},{"code_sha256_prefix":"561d4224329b8802","entry":"mnasnet0_75","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/mnasnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/mnasnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"561d4224329b8802"}},{"code_sha256_prefix":"541eb8cf9b978129","entry":"mnasnet1_0","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/mnasnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/mnasnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"541eb8cf9b978129"}},{"code_sha256_prefix":"57f30bbef481b9c8","entry":"mobilenet_v2","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/mobilenetv2.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/mobilenetv2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"57f30bbef481b9c8"}},{"code_sha256_prefix":"d523a6fcfaf250c7","entry":"parse_devkit_archive","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/datasets/imagenet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/datasets/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d523a6fcfaf250c7"}},{"code_sha256_prefix":"6af00ec71cf439d8","entry":"resnet18","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/resnet.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/resnet.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6af00ec71cf439d8"}},{"code_sha256_prefix":"3edb47a32cd6a773","entry":"shufflenet_v2_x0_5","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/shufflenetv2.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/shufflenetv2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3edb47a32cd6a773"}},{"code_sha256_prefix":"97931550315207a2","entry":"shufflenet_v2_x1_0","repo":"UiPath/torchvision","repo_kind":"found_in_text","path":"torchvision/models/shufflenetv2.py","file_url":"https://github.com/UiPath/torchvision/blob/HEAD/torchvision/models/shufflenetv2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"97931550315207a2"}},{"code_sha256_prefix":"d92101f375faa8a6","entry":"store_model_weights","repo":"iShohei220/adopt","repo_kind":"official","path":"imagenet/utils.py","file_url":"https://github.com/iShohei220/adopt/blob/HEAD/imagenet/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d92101f375faa8a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}