{"url":"/method/adashift","slug":"adashift","name":"AdaShift","full_name":"AdaShift","full_name_withheld":false,"description_markdown":"**AdaShift** is a type of adaptive stochastic optimizer that decorrelates $v\\_{t}$ and $g\\_{t}$ in [Adam](https://paperswithcode.com/method/adam) by temporal shifting, i.e., using temporally shifted gradient $g\\_{t−n}$ to calculate $v\\_{t}$. The authors argue that an inappropriate correlation between gradient $g\\_{t}$ and the second-moment term $v\\_{t}$ exists in Adam, which results in a large gradient being likely to have a small step size while a small gradient may have a large step size. The authors argue that such biased step sizes are the fundamental cause of non-convergence of Adam.\r\n\r\nThe AdaShift updates, based on the idea of temporal independence between gradients, are as follows:\r\n\r\n$$ g\\_{t} = \\nabla{f\\_{t}}\\left(\\theta\\_{t}\\right) $$\r\n\r\n$$ m\\_{t} = \\sum^{n-1}\\_{i=0}\\beta^{i}\\_{1}g\\_{t-i}/\\sum^{n-1}\\_{i=0}\\beta^{i}\\_{1} $$\r\n\r\nThen for $i=1$ to $M$:\r\n\r\n$$ v\\_{t}\\left[i\\right] = \\beta\\_{2}v\\_{t-1}\\left[i\\right] + \\left(1-\\beta\\_{2}\\right)\\phi\\left(g^{2}\\_{t-n}\\left[i\\right]\\right) $$\r\n\r\n$$ \\theta\\_{t}\\left[i\\right] = \\theta\\_{t-1}\\left[i\\right] - \\alpha\\_{t}/\\sqrt{v\\_{t}\\left[i\\right]}\\cdot{m\\_{t}\\left[i\\right]} $$","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1810.00143v4","title":"AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/MichaelKonobeev/adashift/blob/bf86b021d42e922078a39246770f0f875300a6f3/adashift/optimizers.py#L8","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":4,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift Factor","date":"2024-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/momentum-centering-and-asynchronous-update","title":"Momentum Centering and Asynchronous Update for Adaptive Gradient Methods","date":"2021-10-11","arxiv_id":"2110.05454","n_code_links":2,"syntology":null},{"paper":"/paper/an-adaptive-and-momental-bound-method-for","title":"An Adaptive and Momental Bound Method for Stochastic Learning","date":"2019-10-27","arxiv_id":"1910.12249","n_code_links":2,"syntology":{"ran":0,"of":2,"unverified":2,"pointer_only":0}},{"paper":"/paper/adashift-decorrelation-and-convergence-of","title":"AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods","date":"2018-09-29","arxiv_id":"1810.00143","n_code_links":3,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}}],"papers_shown":4,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/representation-learning","name":"Representation Learning","papers":1},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1}],"tasks_shown":4,"n_tasks":4,"usage_by_year":[{"year":"2018","papers":1},{"year":"2019","papers":1},{"year":"2021","papers":1},{"year":"2024","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adashift"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}