{"url":"/method/adamod","slug":"adamod","name":"AdaMod","full_name":"AdaMod","full_name_withheld":false,"description_markdown":"**AdaMod** is a stochastic optimizer that restricts adaptive learning rates with adaptive and momental upper bounds. The dynamic learning rate bounds are based on the exponential moving averages of the adaptive learning rates themselves, which smooth out unexpected large learning rates and stabilize the training of deep neural networks.\r\n\r\n\r\nThe weight updates are performed as:\r\n\r\n\r\n$$ g\\_{t} = \\nabla{f}\\_{t}\\left(\\theta\\_{t-1}\\right) $$\r\n\r\n$$ m\\_{t} = \\beta\\_{1}m\\_{t-1} + \\left(1-\\beta\\_{1}\\right)g\\_{t} $$\r\n\r\n$$ v\\_{t} = \\beta\\_{2}v\\_{t-1} + \\left(1-\\beta\\_{2}\\right)g\\_{t}^{2} $$\r\n\r\n$$ \\hat{m}\\_{t} = m\\_{t} / \\left(1 - \\beta^{t}\\_{1}\\right)$$\r\n\r\n$$ \\hat{v}\\_{t} = v\\_{t} / \\left(1 - \\beta^{t}\\_{2}\\right)$$\r\n\r\n$$ \\eta\\_{t} = \\alpha\\_{t} / \\left(\\sqrt{\\hat{v}\\_{t}} + \\epsilon\\right) $$\r\n\r\n$$ s\\_{t} = \\beta\\_{3}s\\_{t-1} + (1-\\beta\\_{3})\\eta\\_{t} $$\r\n\r\n$$ \\hat{\\eta}\\_{t} = \\text{min}\\left(\\eta\\_{t}, s\\_{t}\\right) $$\r\n\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} - \\hat{\\eta}\\_{t}\\hat{m}\\_{t} $$","description_state":"present","introduced_year":null,"introduced_by":{"title":"An Adaptive and Momental Bound Method for Stochastic Learning","paper":"/paper/an-adaptive-and-momental-bound-method-for","first_author":"Jianbang Ding","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/an-adaptive-and-momental-bound-method-for"},"source":{"url":"https://arxiv.org/abs/1910.12249v1","title":"An Adaptive and Momental Bound Method for Stochastic Learning","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/jettify/pytorch-optimizer/blob/155246597d66dd774156599be0f07a8c6f7758aa/torch_optimizer/adamod.py#L11","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/an-adaptive-and-momental-bound-method-for","title":"An Adaptive and Momental Bound Method for Stochastic Learning","date":"2019-10-27","arxiv_id":"1910.12249","n_code_links":2,"syntology":{"ran":0,"of":2,"unverified":2,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":1}],"tasks_shown":1,"n_tasks":1,"usage_by_year":[{"year":"2019","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adamod"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}