{"url":"/method/radam","slug":"radam","name":"RAdam","full_name":"RAdam","full_name_withheld":false,"description_markdown":"**Rectified Adam**, or **RAdam**, is a variant of the [Adam](https://paperswithcode.com/method/adam) stochastic optimizer that introduces a term to rectify the variance of the adaptive learning rate. It seeks to tackle the bad convergence problem suffered by Adam. The authors argue that the root cause of this behaviour is that the adaptive learning rate has undesirably large variance in the early stage of model training, due to the limited amount of training samples being used. Thus, to reduce such variance, it is better to use smaller learning rates in the first few epochs of training - which justifies the warmup heuristic. This heuristic motivates RAdam which rectifies the variance problem:\r\n\r\n$$g\\_{t} = \\nabla\\_{\\theta}f\\_{t}\\left(\\theta\\_{t-1}\\right) $$\r\n\r\n$$v\\_{t} = 1/\\beta\\_{2}v\\_{t-1} + \\left(1-\\beta\\_{2}\\right)g^{2}\\_{t} $$\r\n\r\n$$m\\_{t} = \\beta\\_{1}m\\_{t-1} + \\left(1-\\beta\\_{1}\\right)g\\_{t} $$\r\n\r\n$$ \\hat{m\\_{t}} = m\\_{t} / \\left(1-\\beta^{t}\\_{1}\\right) $$\r\n\r\n$$ \\rho\\_{t} = \\rho\\_{\\infty} - 2t\\beta^{t}\\_{2}/\\left(1-\\beta^{t}\\_{2}\\right) $$\r\n\r\n$$\\rho_{\\infty} = \\frac{2}{1-\\beta_2} - 1$$ \r\n\r\nIf the variance is tractable - $\\rho\\_{t} > 4$ then:\r\n\r\n...the adaptive learning rate is computed as:\r\n\r\n$$ l\\_{t} = \\sqrt{\\left(1-\\beta^{t}\\_{2}\\right)/v\\_{t}}$$\r\n\r\n...the variance rectification term is calculated as:\r\n\r\n$$ r\\_{t} = \\sqrt{\\frac{(\\rho\\_{t}-4)(\\rho\\_{t}-2)\\rho\\_{\\infty}}{(\\rho\\_{\\infty}-4)(\\rho\\_{\\infty}-2)\\rho\\_{t}}}$$\r\n\r\n...and we update parameters with adaptive momentum:\r\n\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} - \\alpha\\_{t}r\\_{t}\\hat{m}\\_{t}l\\_{t} $$\r\n\r\nIf the variance isn't tractable we update instead with:\r\n\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} - \\alpha\\_{t}\\hat{m}\\_{t} $$","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1908.03265v4","title":"On the Variance of the Adaptive Learning Rate and Beyond","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/d05c1ec007ae75de5f6df7b05de7cca59751dea5/torch/optim/radam.py#L6","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":65,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"The Hype Index: an NLP-driven Measure of Market News Attention","date":"2025-05-30","arxiv_id":"2506.06329","n_code_links":0,"syntology":null},{"paper":null,"title":"Model Editing with Graph-Based External Memory","date":"2025-05-23","arxiv_id":"2505.18343","n_code_links":0,"syntology":null},{"paper":null,"title":"Exploring the Innovation Opportunities for Pre-trained Models","date":"2025-05-21","arxiv_id":"2505.15790","n_code_links":0,"syntology":null},{"paper":null,"title":"RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs","date":"2025-05-19","arxiv_id":"2505.13697","n_code_links":0,"syntology":null},{"paper":null,"title":"Comparative Analysis of Evolutionary Algorithms for Energy-Aware Production Scheduling","date":"2025-04-22","arxiv_id":"2504.15672","n_code_links":0,"syntology":null},{"paper":null,"title":"Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks","date":"2025-04-05","arxiv_id":"2504.04277","n_code_links":0,"syntology":null},{"paper":null,"title":"What the F*ck Is Artificial General Intelligence?","date":"2025-03-31","arxiv_id":"2503.23923","n_code_links":0,"syntology":null},{"paper":"/paper/bridging-evolutionary-multiobjective","title":"Bridging Evolutionary Multiobjective Optimization and GPU Acceleration via Tensorization","date":"2025-03-26","arxiv_id":"2503.20286","n_code_links":2,"syntology":null},{"paper":"/paper/guidelines-for-the-choice-of-the-baseline-in","title":"Guidelines For The Choice Of The Baseline in XAI Attribution Methods","date":"2025-03-25","arxiv_id":"2503.19813","n_code_links":1,"syntology":null},{"paper":null,"title":"The Imitation Game According To Turing","date":"2025-01-29","arxiv_id":"2501.17629","n_code_links":0,"syntology":null},{"paper":"/paper/why-these-documents-explainable-generative","title":"Why These Documents? Explainable Generative Retrieval with Hierarchical Category Paths","date":"2024-11-08","arxiv_id":"2411.05572","n_code_links":1,"syntology":null},{"paper":null,"title":"Danoliteracy of Generative, Large Language Models","date":"2024-10-30","arxiv_id":"2410.22839","n_code_links":0,"syntology":null},{"paper":null,"title":"Misrepresented Technological Solutions in Imagined Futures: The Origins and Dangers of AI Hype in the Research Community","date":"2024-08-08","arxiv_id":"2408.15244","n_code_links":0,"syntology":null},{"paper":null,"title":"Decoding Knowledge Claims: The Evaluation of Scientific Publication Contributions through Semantic Analysis","date":"2024-07-26","arxiv_id":"2407.18646","n_code_links":0,"syntology":null},{"paper":null,"title":"Continuous fake media detection: adapting deepfake detectors to new generative techniques","date":"2024-06-12","arxiv_id":"2406.08171","n_code_links":0,"syntology":null},{"paper":"/paper/hype-hyperbolic-entailment-filtering-for","title":"HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts","date":"2024-04-26","arxiv_id":"2404.17507","n_code_links":1,"syntology":null},{"paper":null,"title":"AI Safety: Necessary, but insufficient and possibly problematic","date":"2024-03-26","arxiv_id":"2403.17419","n_code_links":0,"syntology":null},{"paper":null,"title":"The Interplay of Learning, Analytics, and Artificial Intelligence in Education: A Vision for Hybrid Intelligence","date":"2024-03-24","arxiv_id":"2403.16081","n_code_links":0,"syntology":null},{"paper":null,"title":"Are you a robot? Detecting Autonomous Vehicles from Behavior Analysis","date":"2024-03-14","arxiv_id":"2403.09571","n_code_links":0,"syntology":null},{"paper":null,"title":"Assessing the Reasoning Abilities of ChatGPT in the Context of Claim Verification","date":"2024-02-16","arxiv_id":"2402.10735","n_code_links":0,"syntology":null},{"paper":null,"title":"Neither hype nor gloom do DNNs justice","date":"2023-12-08","arxiv_id":"2312.05355","n_code_links":0,"syntology":null},{"paper":null,"title":"Hypothesis Network Planned Exploration for Rapid Meta-Reinforcement Learning Adaptation","date":"2023-11-07","arxiv_id":"2311.03701","n_code_links":0,"syntology":null},{"paper":null,"title":"HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding","date":"2023-10-30","arxiv_id":"2310.19676","n_code_links":0,"syntology":null},{"paper":null,"title":"Web3 Meets AI Marketplace: Exploring Opportunities, Analyzing Challenges, and Suggesting Solutions","date":"2023-10-29","arxiv_id":"2310.19099","n_code_links":0,"syntology":null},{"paper":null,"title":"Does Artificial Intelligence benefit UK businesses? An empirical study of the impact of AI on productivity","date":"2023-10-06","arxiv_id":"2310.05985","n_code_links":0,"syntology":null},{"paper":null,"title":"Wisdom of the Crowds or Ignorance of the Masses? A data-driven guide to WSB","date":"2023-08-18","arxiv_id":"2308.09485","n_code_links":0,"syntology":null},{"paper":null,"title":"Proposing a conceptual framework: social media listening for public health behavior","date":"2023-07-30","arxiv_id":"2308.02037","n_code_links":0,"syntology":null},{"paper":null,"title":"Amplifying Limitations, Harms and Risks of Large Language Models","date":"2023-07-06","arxiv_id":"2307.04821","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning to Prompt in the Classroom to Understand AI Limits: A pilot study","date":"2023-07-04","arxiv_id":"2307.01540","n_code_links":0,"syntology":null},{"paper":null,"title":"Wearable-based Fair and Accurate Pain Assessment Using Multi-Attribute Fairness Loss in Convolutional Neural Networks","date":"2023-07-03","arxiv_id":"2307.05333","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":4},{"task":"/task/deep-learning","name":"Deep Learning","papers":3},{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/language-modeling","name":"Language Modeling","papers":3},{"task":"/task/articles","name":"Articles","papers":2},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":2},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":2},{"task":"/task/classification-1","name":"Classification","papers":2},{"task":"/task/classification","name":"General Classification","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/translation","name":"Translation","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/anomaly-detection","name":"Anomaly Detection","papers":1},{"task":"/task/attribute","name":"Attribute","papers":1},{"task":"/task/autonomous-vehicles","name":"Autonomous Vehicles","papers":1},{"task":null,"name":"CPU","papers":1},{"task":"/task/claim-verification","name":"Claim Verification","papers":1},{"task":"/task/continual-learning","name":"Continual Learning","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/deepfake-detection","name":"DeepFake Detection","papers":1}],"tasks_shown":20,"n_tasks":73,"usage_by_year":[{"year":"2019","papers":2},{"year":"2020","papers":6},{"year":"2021","papers":7},{"year":"2022","papers":11},{"year":"2023","papers":19},{"year":"2024","papers":10},{"year":"2025","papers":10}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/radam"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}