Methods › General › Stochastic Optimization › AdaShift
AdaShift
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdaShift is a type of adaptive stochastic optimizer that decorrelates vₜ and gₜ in Adam by temporal shifting, i.e., using temporally shifted gradient gₜ₋ₙ to calculate vₜ. The authors argue that an inappropriate correlation between gradient gₜ and the second-moment term vₜ exists in Adam, which results in a large gradient being likely to have a small step size while a small gradient may have a large step size. The authors argue that such biased step sizes are the fundamental cause of non-convergence of Adam.
The AdaShift updates, based on the idea of temporal independence between gradients, are as follows:
gₜ = ∇fₜ(θₜ)
mₜ = ∑ⁿ⁻¹ᵢ₌₀βⁱ₁gₜ₋ᵢ/∑ⁿ⁻¹ᵢ₌₀βⁱ₁
Then for i=1 to M:
vₜ[i] = β₂vₜ₋₁[i] + (1-β₂)ϕ(g²ₜ₋ₙ[i])
θₜ[i] = θₜ₋₁[i] - αₜ/√(vₜ[i])·mₜ[i]
Papers archive 2025-07-28
4 shown of 4, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift Factor 1 Jan 2024 · 0 repositories
-
Momentum Centering and Asynchronous Update for Adaptive Gradient Methods 11 Oct 2021 · 2 repositories · arXiv:2110.05454
-
An Adaptive and Momental Bound Method for Stochastic Learning 27 Oct 2019 · 2 repositories · arXiv:1910.12249Syntology ran 0 of 2 samples · 2 unverified
-
AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods 29 Sep 2018 · 3 repositories · arXiv:1810.00143Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
Tasks archive 2025-07-28
4 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Image Classification | 1 |
| Representation Learning | 1 |
| Stochastic Optimization | 1 |
| image-classification | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections