Methods › General › Momentum Rules › Demon
Demon
Introduced by John Chen et al. in Demon: Improved Neural Network Training with Momentum Decay
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Decaying Momentum, or Demon, is a stochastic optimizer motivated by decaying the total contribution of a gradient to all future updates. By decaying the momentum parameter, the total contribution of a gradient to all future updates is decayed. A particular gradient term gₜ contributes a total of η∑ᵢβⁱ of its "energy" to all future gradient updates, and this results in the geometric sum, ∑^∞ᵢ₌₁βⁱ = β∑^∞ᵢ₌₀βⁱ = β/(1-β). Decaying this sum results in the Demon algorithm. Letting βᵢₙᵢₜ be the initial β; then at the current step t with total T steps, the decay routine is given by solving the below for βₜ:
βₜ/(1-βₜ) = (1-t/T)βᵢₙᵢₜ/(1-βᵢₙᵢₜ)
Where (1-t/T) refers to the proportion of iterations remaining. Note that Demon typically requires no hyperparameter tuning as it is usually decayed to $0$ or a small negative value at time T. Improved performance is observed by delaying the decaying. Demon can be applied to any gradient descent algorithm with a momentum parameter.
Papers archive 2025-07-28
16 shown of 16, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Representation and Interpretation in Artificial and Natural Computing 14 Feb 2025 · 0 repositories · arXiv:2502.10383
-
DEMONet: Underwater Acoustic Target Recognition based on Multi-Expert Network and Cross-Temporal Variational Autoencoder 5 Nov 2024 · 0 repositories · arXiv:2411.02758
-
Training-free Diffusion Model Alignment with Sampling Demons 8 Oct 2024 · 1 repository · arXiv:2410.05760Syntology ran 2 of 3 samples · 1 unverified
-
Neural Entropy 5 Sep 2024 · 0 repositories · arXiv:2409.03817
-
Str-L Pose: Integrating Point and Structured Line for Relative Pose Estimation in Dual-Graph 28 Aug 2024 · 0 repositories · arXiv:2408.15750
-
A Decentralized and Self-Adaptive Approach for Monitoring Volatile Edge Environments 13 May 2024 · 1 repository · arXiv:2405.07806
-
Reflective Linguistic Programming (RLP): A Stepping Stone in Socially-Aware AGI (SocialAGI) 22 May 2023 · 0 repositories · arXiv:2305.12647
-
How to train your demon to do fast information erasure without heat production 17 May 2023 · 0 repositories · arXiv:2305.10607
-
Thermodynamic AI and the fluctuation frontier 9 Feb 2023 · 0 repositories · arXiv:2302.06584
-
Differentiable Neural Computers with Memory Demon 5 Nov 2022 · 1 repository · arXiv:2211.02987Syntology ran 3 of 5 samples · 2 unverified · 5 pointer-only (licence)
-
Static Knowledge vs. Dynamic Argumentation: A Dual Theory Based on Kripke Semantics 27 Sep 2022 · 0 repositories · arXiv:2209.13082
-
Boosting Adversarial Transferability of MLP-Mixer 26 Apr 2022 · 0 repositories · arXiv:2204.12204
-
Learning Relational Rules from Rewards 25 Mar 2022 · 1 repository · arXiv:2203.13599
-
Nonequilibrium thermodynamics of self-supervised learning 16 Jun 2021 · 0 repositories · arXiv:2106.08981
-
Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty 16 Apr 2021 · 0 repositories · arXiv:2104.08278
-
Demon: Improved Neural Network Training with Momentum Decay 11 Oct 2019 · 2 repositories · arXiv:1910.04952
Tasks archive 2025-07-28
18 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections