{"url":"/method/demon","slug":"demon","name":"Demon","full_name":"Demon","full_name_withheld":false,"description_markdown":"**Decaying Momentum**, or **Demon**, is a stochastic optimizer motivated by decaying the total contribution of a gradient to all future updates. By decaying the momentum parameter, the total contribution of a gradient to all future updates is decayed. A particular gradient term $g\\_{t}$ contributes a total of  $\\eta\\sum\\_{i}\\beta^{i}$ of its \"energy\" to all future gradient updates, and this results in the geometric sum, $\\sum^{\\infty}\\_{i=1}\\beta^{i} = \\beta\\sum^{\\infty}\\_{i=0}\\beta^{i} = \\frac{\\beta}{\\left(1-\\beta\\right)}$. Decaying this sum results in the Demon algorithm. Letting $\\beta\\_{init}$ be the initial $\\beta$; then at the current step $t$ with total $T$ steps, the decay routine is given by solving the below for $\\beta\\_{t}$:\r\n\r\n$$ \\frac{\\beta\\_{t}}{\\left(1-\\beta\\_{t}\\right)} =  \\left(1-t/T\\right)\\beta\\_{init}/\\left(1-\\beta\\_{init}\\right)$$\r\n\r\nWhere $\\left(1-t/T\\right)$ refers to the proportion of iterations remaining. Note that Demon typically requires no hyperparameter tuning as it is usually decayed to $0$ or a small negative value at time \r\n$T$. Improved performance is observed by delaying the decaying. Demon can be applied to any gradient descent algorithm with a momentum parameter.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Demon: Improved Neural Network Training with Momentum Decay","paper":"/paper/decaying-momentum-helps-neural-network","first_author":"John Chen","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/decaying-momentum-helps-neural-network"},"source":{"url":"https://arxiv.org/abs/1910.04952v4","title":"Demon: Improved Neural Network Training with Momentum Decay","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/JRC1995/DemonRangerOptimizer/blob/5a3e6e352ab766f96cd8d20eabd5b71843c595fe/optimizers.py#L205","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Momentum Rules","url":"/methods/category/momentum-rules","pwc_aliases":[]}],"n_papers_tagged":16,"archive_num_papers":16,"papers_newest_first":[{"paper":null,"title":"Representation and Interpretation in Artificial and Natural Computing","date":"2025-02-14","arxiv_id":"2502.10383","n_code_links":0,"syntology":null},{"paper":null,"title":"DEMONet: Underwater Acoustic Target Recognition based on Multi-Expert Network and Cross-Temporal Variational Autoencoder","date":"2024-11-05","arxiv_id":"2411.02758","n_code_links":0,"syntology":null},{"paper":"/paper/training-free-diffusion-model-alignment-with","title":"Training-free Diffusion Model Alignment with Sampling Demons","date":"2024-10-08","arxiv_id":"2410.05760","n_code_links":1,"syntology":{"ran":2,"of":3,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Neural Entropy","date":"2024-09-05","arxiv_id":"2409.03817","n_code_links":0,"syntology":null},{"paper":null,"title":"Str-L Pose: Integrating Point and Structured Line for Relative Pose Estimation in Dual-Graph","date":"2024-08-28","arxiv_id":"2408.15750","n_code_links":0,"syntology":null},{"paper":"/paper/a-decentralized-and-self-adaptive-approach","title":"A Decentralized and Self-Adaptive Approach for Monitoring Volatile Edge Environments","date":"2024-05-13","arxiv_id":"2405.07806","n_code_links":1,"syntology":null},{"paper":null,"title":"Reflective Linguistic Programming (RLP): A Stepping Stone in Socially-Aware AGI (SocialAGI)","date":"2023-05-22","arxiv_id":"2305.12647","n_code_links":0,"syntology":null},{"paper":null,"title":"How to train your demon to do fast information erasure without heat production","date":"2023-05-17","arxiv_id":"2305.10607","n_code_links":0,"syntology":null},{"paper":null,"title":"Thermodynamic AI and the fluctuation frontier","date":"2023-02-09","arxiv_id":"2302.06584","n_code_links":0,"syntology":null},{"paper":"/paper/differentiable-neural-computers-with-memory","title":"Differentiable Neural Computers with Memory Demon","date":"2022-11-05","arxiv_id":"2211.02987","n_code_links":1,"syntology":{"ran":3,"of":5,"unverified":2,"pointer_only":5}},{"paper":null,"title":"Static Knowledge vs. Dynamic Argumentation: A Dual Theory Based on Kripke Semantics","date":"2022-09-27","arxiv_id":"2209.13082","n_code_links":0,"syntology":null},{"paper":null,"title":"Boosting Adversarial Transferability of MLP-Mixer","date":"2022-04-26","arxiv_id":"2204.12204","n_code_links":0,"syntology":null},{"paper":"/paper/learning-relational-rules-from-rewards","title":"Learning Relational Rules from Rewards","date":"2022-03-25","arxiv_id":"2203.13599","n_code_links":1,"syntology":null},{"paper":null,"title":"Nonequilibrium thermodynamics of self-supervised learning","date":"2021-06-16","arxiv_id":"2106.08981","n_code_links":0,"syntology":null},{"paper":null,"title":"Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty","date":"2021-04-16","arxiv_id":"2104.08278","n_code_links":0,"syntology":null},{"paper":"/paper/decaying-momentum-helps-neural-network","title":"Demon: Improved Neural Network Training with Momentum Decay","date":"2019-10-11","arxiv_id":"1910.04952","n_code_links":2,"syntology":null}],"papers_shown":16,"tasks":[{"task":"/task/graph-neural-network","name":"Graph Neural Network","papers":2},{"task":"/task/pose-estimation","name":"Pose Estimation","papers":2},{"task":"/task/adversarial-attack","name":"Adversarial Attack","papers":1},{"task":"/task/atari-games","name":"Atari Games","papers":1},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":1},{"task":"/task/camera-pose-estimation","name":"Camera Pose Estimation","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/edge-computing","name":"Edge-computing","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/image-generation","name":"Image Generation","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/scene-understanding","name":"Scene Understanding","papers":1},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":1},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":1},{"task":"/task/text-to-image-generation-1","name":"Text to Image Generation","papers":1},{"task":"/task/text-to-image-generation","name":"Text-to-Image Generation","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1}],"tasks_shown":18,"n_tasks":18,"usage_by_year":[{"year":"2019","papers":1},{"year":"2021","papers":2},{"year":"2022","papers":4},{"year":"2023","papers":3},{"year":"2024","papers":5},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/demon"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}