{"url":"/method/rmsprop","slug":"rmsprop","name":"RMSProp","full_name":"RMSProp","full_name_withheld":false,"description_markdown":"**RMSProp** is an unpublished adaptive learning rate optimizer [proposed by Geoff Hinton](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf). The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:\r\n\r\n$$E\\left[g^{2}\\right]\\_{t} = \\gamma E\\left[g^{2}\\right]\\_{t-1} + \\left(1 - \\gamma\\right) g^{2}\\_{t}$$\r\n\r\n$$\\theta\\_{t+1} = \\theta\\_{t} - \\frac{\\eta}{\\sqrt{E\\left[g^{2}\\right]\\_{t} + \\epsilon}}g\\_{t}$$\r\n\r\nHinton suggests $\\gamma=0.9$, with a good default for $\\eta$ as $0.001$.\r\n\r\nImage: [Alec Radford](https://twitter.com/alecrad)","description_state":"present","introduced_year":2013,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/fd8e2064e094f301d910b91a757b860aae3e3116/torch/optim/rmsprop.py#L69-L108","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":519,"archive_num_papers":519,"papers_newest_first":[{"paper":null,"title":"Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate","date":"2025-06-22","arxiv_id":"2506.22479","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Learning-Based BMD Estimation from Radiographs with Conformal Uncertainty Quantification","date":"2025-05-28","arxiv_id":"2505.22551","n_code_links":0,"syntology":null},{"paper":null,"title":"Intelligent Incident Hypertension Prediction in Obstructive Sleep Apnea","date":"2025-05-27","arxiv_id":"2505.20615","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Learning for Breast Cancer Detection: Comparative Analysis of ConvNeXT and EfficientNet","date":"2025-05-24","arxiv_id":"2505.18725","n_code_links":0,"syntology":null},{"paper":null,"title":"SuperPure: Efficient Purification of Localized and Distributed Adversarial Patches via Super-Resolution GAN Models","date":"2025-05-22","arxiv_id":"2505.16318","n_code_links":0,"syntology":null},{"paper":"/paper/vulnerability-of-transfer-learned-neural","title":"Vulnerability of Transfer-Learned Neural Networks to Data Reconstruction Attacks in Small-Data Regime","date":"2025-05-20","arxiv_id":"2505.14323","n_code_links":1,"syntology":null},{"paper":null,"title":"Defect Detection in Photolithographic Patterns Using Deep Learning Models Trained on Synthetic Data","date":"2025-05-15","arxiv_id":"2505.10192","n_code_links":0,"syntology":null},{"paper":null,"title":"Real-World fNIRS-Based Brain-Computer Interfaces: Benchmarking Deep Learning and Classical Models in Interactive Gaming","date":"2025-05-15","arxiv_id":"2505.10536","n_code_links":0,"syntology":null},{"paper":null,"title":"Trial and Trust: Addressing Byzantine Attacks with Comprehensive Defense Strategy","date":"2025-05-12","arxiv_id":"2505.07614","n_code_links":0,"syntology":null},{"paper":"/paper/v-efficientnets-vector-valued-efficiently","title":"V-EfficientNets: Vector-Valued Efficiently Scaled Convolutional Neural Network Models","date":"2025-05-08","arxiv_id":"2505.05659","n_code_links":1,"syntology":null},{"paper":"/paper/anomalymatch-discovering-rare-objects-of","title":"AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning","date":"2025-05-06","arxiv_id":"2505.03509","n_code_links":1,"syntology":null},{"paper":null,"title":"CSASN: A Multitask Attention-Based Framework for Heterogeneous Thyroid Carcinoma Classification in Ultrasound Images","date":"2025-05-04","arxiv_id":"2505.02211","n_code_links":0,"syntology":null},{"paper":null,"title":"Conformal Prediction for Indoor Positioning with Correctness Coverage Guarantees","date":"2025-05-03","arxiv_id":"2505.01810","n_code_links":0,"syntology":null},{"paper":null,"title":"Sharp higher order convergence rates for the Adam optimizer","date":"2025-04-28","arxiv_id":"2504.19426","n_code_links":0,"syntology":null},{"paper":null,"title":"Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness","date":"2025-04-21","arxiv_id":"2504.14882","n_code_links":0,"syntology":null},{"paper":null,"title":"Covariant Gradient Descent","date":"2025-04-07","arxiv_id":"2504.05279","n_code_links":0,"syntology":null},{"paper":"/paper/training-frozen-feature-pyramid-dinov2-for","title":"Training Frozen Feature Pyramid DINOv2 for Eyelid Measurements with Infinite Encoding and Orthogonal Regularization","date":"2025-04-01","arxiv_id":"2504.00515","n_code_links":1,"syntology":null},{"paper":null,"title":"Efficient Building Roof Type Classification: A Domain-Specific Self-Supervised Approach","date":"2025-03-28","arxiv_id":"2503.22251","n_code_links":0,"syntology":null},{"paper":"/paper/world-model-agents-with-change-based","title":"World Model Agents with Change-Based Intrinsic Motivation","date":"2025-03-26","arxiv_id":"2503.21047","n_code_links":1,"syntology":null},{"paper":null,"title":"Deep learning-based identification of precipitation clouds from all-sky camera data for observatory safety","date":"2025-03-24","arxiv_id":"2503.18670","n_code_links":0,"syntology":null},{"paper":null,"title":"Weakly Supervised Pixel-Level Annotation with Visual Interpretability","date":"2025-02-25","arxiv_id":"2502.17824","n_code_links":0,"syntology":null},{"paper":null,"title":"Reducing false positives in strong lens detection through effective augmentation and ensemble learning","date":"2025-02-20","arxiv_id":"2502.14936","n_code_links":0,"syntology":null},{"paper":null,"title":"CAE-Net: Generalized Deepfake Image Detection using Convolution and Attention Mechanisms with Spatial and Frequency Domain Features","date":"2025-02-15","arxiv_id":"2502.10682","n_code_links":0,"syntology":null},{"paper":null,"title":"AI-Driven Solutions for Falcon Disease Classification: Concatenated ConvNeXt cum EfficientNet AI Model Approach","date":"2025-02-07","arxiv_id":"2502.04682","n_code_links":0,"syntology":null},{"paper":"/paper/enhancing-brain-tumor-segmentation-using","title":"Enhancing Brain Tumor Segmentation Using Channel Attention and Transfer learning","date":"2025-01-19","arxiv_id":"2501.11196","n_code_links":1,"syntology":null},{"paper":"/paper/iot-based-real-time-medical-related-human","title":"IoT-Based Real-Time Medical-Related Human Activity Recognition Using Skeletons and Multi-Stage Deep Learning for Healthcare","date":"2025-01-13","arxiv_id":"2501.07039","n_code_links":1,"syntology":null},{"paper":"/paper/layerdropback-a-universally-applicable","title":"LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks","date":"2024-12-23","arxiv_id":"2412.18027","n_code_links":1,"syntology":null},{"paper":null,"title":"Comparative Analysis of Deep Learning Approaches for Harmful Brain Activity Detection Using EEG","date":"2024-12-10","arxiv_id":"2412.07878","n_code_links":0,"syntology":null},{"paper":null,"title":"Impact of Privacy Parameters on Deep Learning Models for Image Classification","date":"2024-12-09","arxiv_id":"2412.06689","n_code_links":0,"syntology":null},{"paper":null,"title":"Thermal Image-based Fault Diagnosis in Induction Machines via Self-Organized Operational Neural Networks","date":"2024-12-08","arxiv_id":"2412.05901","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":97},{"task":"/task/image-classification","name":"image-classification","papers":62},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":60},{"task":"/task/classification-1","name":"Classification","papers":39},{"task":"/task/deep-learning","name":"Deep Learning","papers":37},{"task":"/task/classification","name":"General Classification","papers":35},{"task":"/task/object-detection","name":"Object Detection","papers":31},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":31},{"task":"/task/architecture-search","name":"Neural Architecture Search","papers":26},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":25},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":20},{"task":"/task/object-detection-1","name":"object-detection","papers":20},{"task":"/task/diagnostic","name":"Diagnostic","papers":18},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":16},{"task":"/task/segmentation","name":"Segmentation","papers":14},{"task":null,"name":"GPU","papers":12},{"task":"/task/prediction","name":"Prediction","papers":12},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":12},{"task":"/task/quantization","name":"Quantization","papers":10},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":10}],"tasks_shown":20,"n_tasks":353,"usage_by_year":[{"year":"2015","papers":3},{"year":"2016","papers":9},{"year":"2017","papers":20},{"year":"2018","papers":33},{"year":"2019","papers":49},{"year":"2020","papers":83},{"year":"2021","papers":86},{"year":"2022","papers":65},{"year":"2023","papers":66},{"year":"2024","papers":79},{"year":"2025","papers":26}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/rmsprop"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}