{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adai-separating-the-effects-of-adaptive","title":"Adai: Separating the Effects of Adaptive Learning Rate and Momentum Inertia","arxiv_id":"2006.15815","date":"2020-06-29","proceeding":null,"authors":["Zeke Xie","Xinrui Wang","Huishuai Zhang","Issei Sato","Masashi Sugiyama"],"abstract":"Adaptive Momentum Estimation (Adam), which combines Adaptive Learning Rate and Momentum, is the most popular stochastic optimizer for accelerating the training of deep neural networks. However, empirically Adam often generalizes worse than Stochastic Gradient Descent (SGD). We unveil the mystery of this behavior based on the diffusion theoretical framework. Specifically, we disentangle the effects of Adaptive Learning Rate and Momentum of the Adam dynamics on saddle-point escaping and minima selection. We prove that Adaptive Learning Rate can escape saddle points efficiently, but cannot select flat minima as SGD does. In contrast, Momentum provides a drift effect to help the training process pass through saddle points, and almost does not affect flat minima selection. This theoretically explains why SGD (with Momentum) generalizes better, while Adam generalizes worse but converges faster. Furthermore, motivated by the analysis, we design a novel adaptive optimization framework named Adaptive Inertia, which uses parameter-wise adaptive inertia to accelerate the training and provably favors flat minima as well as SGD. Our extensive experiments demonstrate that the proposed adaptive inertia method can generalize significantly better than SGD and conventional adaptive gradient methods.","url_abs":"https://arxiv.org/abs/2006.15815v9","url_pdf":"https://arxiv.org/pdf/2006.15815v9.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adai-separating-the-effects-of-adaptive","repo_url":"https://github.com/zeke-xie/adaptive-inertia-adai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.15815","atlas_url":"https://app.syntology.ai/?focus=2006.15815","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.15815"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zeke-xie/adaptive-inertia-adai","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dfb395bae3d07d76","entry":"DenseNet121","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/densenet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/densenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb395bae3d07d76"}},{"code_sha256_prefix":"f295fe3c209ecf78","entry":"DenseNet169","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/densenet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/densenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f295fe3c209ecf78"}},{"code_sha256_prefix":"6383aee1c940e7a5","entry":"DenseNet201","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/densenet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/densenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6383aee1c940e7a5"}},{"code_sha256_prefix":"0f23e7e681845fe0","entry":"ResNet18","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f23e7e681845fe0"}},{"code_sha256_prefix":"ac3dd06d1a08c549","entry":"ResNet34","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac3dd06d1a08c549"}},{"code_sha256_prefix":"66f25b3cc4217531","entry":"ResNet50","repo":"zeke-xie/adaptive-inertia-adai","repo_kind":"official","path":"model/resnet.py","file_url":"https://github.com/zeke-xie/adaptive-inertia-adai/blob/HEAD/model/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66f25b3cc4217531"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}