{"url":"/method/lars","slug":"lars","name":"LARS","full_name":"LARS","full_name_withheld":false,"description_markdown":"**Layer-wise Adaptive Rate Scaling**, or **LARS**, is a large batch optimization technique.  There are two notable differences between LARS and other adaptive algorithms such as [Adam](https://paperswithcode.com/method/adam) or [RMSProp](https://paperswithcode.com/method/rmsprop): first, LARS uses a separate learning rate for each layer and not for each weight. And second, the magnitude of the update is controlled with respect to the weight norm for better control of training speed.\r\n\r\n$$m\\_{t} = \\beta\\_{1}m\\_{t-1} + \\left(1-\\beta\\_{1}\\right)\\left(g\\_{t} + \\lambda{x\\_{t}}\\right)$$\r\n$$x\\_{t+1}^{\\left(i\\right)} = x\\_{t}^{\\left(i\\right)}  - \\eta\\_{t}\\frac{\\phi\\left(|| x\\_{t}^{\\left(i\\right)} ||\\right)}{|| m\\_{t}^{\\left(i\\right)} || }m\\_{t}^{\\left(i\\right)} $$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Large Batch Training of Convolutional Networks","paper":"/paper/large-batch-training-of-convolutional","first_author":"Yang You","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/large-batch-training-of-convolutional"},"source":{"url":"http://arxiv.org/abs/1708.03888v3","title":"Large Batch Training of Convolutional Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/kakaobrain/torchlars/blob/3b3d7e9c7bd35a31b2c2fa6f213beb0bf6881892/torchlars/lars.py#L11","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Large Batch Optimization","url":"/methods/category/large-batch-optimization","pwc_aliases":[]}],"n_papers_tagged":77,"archive_num_papers":77,"papers_newest_first":[{"paper":null,"title":"Analyzing Breast Cancer Survival Disparities by Race and Demographic Location: A Survival Analysis Approach","date":"2025-06-08","arxiv_id":"2506.07191","n_code_links":0,"syntology":null},{"paper":null,"title":"Enhancing Federated Survival Analysis through Peer-Driven Client Reputation in Healthcare","date":"2025-05-22","arxiv_id":"2505.16190","n_code_links":0,"syntology":null},{"paper":null,"title":"Is Self-Supervised Pre-training on Satellite Imagery Better than ImageNet? A Systematic Study with Sentinel-2","date":"2025-02-15","arxiv_id":"2502.10669","n_code_links":0,"syntology":null},{"paper":"/paper/self-supervised-frameworks-for-speaker","title":"Self-Supervised Frameworks for Speaker Verification via Bootstrapped Positive Sampling","date":"2025-01-29","arxiv_id":"2501.17772","n_code_links":1,"syntology":null},{"paper":"/paper/tackling-small-sample-survival-analysis-via","title":"Tackling Small Sample Survival Analysis via Transfer Learning: A Study of Colorectal Cancer Prognosis","date":"2025-01-21","arxiv_id":"2501.12421","n_code_links":1,"syntology":null},{"paper":"/paper/prediction-of-lung-metastasis-from","title":"Prediction of Lung Metastasis from Hepatocellular Carcinoma using the SEER Database","date":"2025-01-20","arxiv_id":"2501.11720","n_code_links":1,"syntology":null},{"paper":null,"title":"Diagnosis and Severity Assessment of Ulcerative Colitis using Self Supervised Learning","date":"2024-12-09","arxiv_id":"2412.07806","n_code_links":0,"syntology":null},{"paper":null,"title":"SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generation","date":"2024-10-15","arxiv_id":"2410.11315","n_code_links":0,"syntology":null},{"paper":null,"title":"The Informed Elastic Net for Fast Grouped Variable Selection and FDR Control in Genomics Research","date":"2024-10-07","arxiv_id":"2410.05211","n_code_links":0,"syntology":null},{"paper":null,"title":"Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs","date":"2024-06-17","arxiv_id":"2406.11278","n_code_links":0,"syntology":null},{"paper":null,"title":"Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation","date":"2024-05-29","arxiv_id":"2405.18688","n_code_links":0,"syntology":null},{"paper":null,"title":"Self-Supervised Multiple Instance Learning for Acute Myeloid Leukemia Classification","date":"2024-03-08","arxiv_id":"2403.05379","n_code_links":0,"syntology":null},{"paper":"/paper/seer-facilitating-structured-reasoning-and","title":"SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning","date":"2024-01-24","arxiv_id":"2401.13246","n_code_links":1,"syntology":{"ran":13,"of":16,"unverified":3,"pointer_only":1}},{"paper":null,"title":"Survival Analysis of Young Triple-Negative Breast Cancer Patients","date":"2024-01-15","arxiv_id":"2401.08712","n_code_links":0,"syntology":null},{"paper":null,"title":"Enhancing Contrastive Learning with Efficient Combinatorial Positive Pairing","date":"2024-01-11","arxiv_id":"2401.05730","n_code_links":0,"syntology":null},{"paper":null,"title":"Quantum Algorithms for the Pathwise Lasso","date":"2023-12-21","arxiv_id":"2312.14141","n_code_links":0,"syntology":null},{"paper":"/paper/importance-of-feature-extraction-in-the","title":"Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trend","date":"2023-11-22","arxiv_id":"2311.13717","n_code_links":2,"syntology":null},{"paper":"/paper/seer-a-knapsack-approach-to-exemplar","title":"SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQA","date":"2023-10-10","arxiv_id":"2310.06675","n_code_links":1,"syntology":{"ran":6,"of":16,"unverified":10,"pointer_only":0}},{"paper":null,"title":"Revisiting LARS for Large Batch Training Generalization of Neural Networks","date":"2023-09-25","arxiv_id":"2309.14053","n_code_links":0,"syntology":null},{"paper":null,"title":"Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)","date":"2023-09-24","arxiv_id":"2309.13681","n_code_links":0,"syntology":null},{"paper":null,"title":"SEER: Super-Optimization Explorer for HLS using E-graph Rewriting with MLIR","date":"2023-08-15","arxiv_id":"2308.07654","n_code_links":0,"syntology":null},{"paper":"/paper/the-role-of-entropy-and-reconstruction-in","title":"The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning","date":"2023-07-20","arxiv_id":"2307.10907","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":2}},{"paper":"/paper/semi-supervised-learning-made-simple-with-1","title":"Semi-supervised learning made simple with self-supervised clustering","date":"2023-06-13","arxiv_id":"2306.07483","n_code_links":1,"syntology":{"ran":2,"of":5,"unverified":3,"pointer_only":5}},{"paper":"/paper/hiding-in-plain-sight-disguising-data","title":"Hiding in Plain Sight: Disguising Data Stealing Attacks in Federated Learning","date":"2023-06-05","arxiv_id":"2306.03013","n_code_links":2,"syntology":null},{"paper":null,"title":"Using Spatio-Temporal Dual-Stream Network with Self-Supervised Learning for Lung Tumor Classification on Radial Probe Endobronchial Ultrasound Video","date":"2023-05-04","arxiv_id":"2305.02719","n_code_links":0,"syntology":null},{"paper":null,"title":"In-Distribution and Out-of-Distribution Self-supervised ECG Representation Learning for Arrhythmia Detection","date":"2023-04-13","arxiv_id":"2304.06427","n_code_links":0,"syntology":null},{"paper":null,"title":"Saddle-to-Saddle Dynamics in Diagonal Linear Networks","date":"2023-04-02","arxiv_id":"2304.00488","n_code_links":0,"syntology":null},{"paper":null,"title":"Towards domain-invariant Self-Supervised Learning with Batch Styles Standardization","date":"2023-03-10","arxiv_id":"2303.06088","n_code_links":0,"syntology":null},{"paper":"/paper/infobatch-lossless-training-speed-up-by","title":"InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning","date":"2023-03-08","arxiv_id":"2303.04947","n_code_links":1,"syntology":{"ran":0,"of":2,"unverified":2,"pointer_only":2}},{"paper":"/paper/towards-a-unified-theoretical-understanding","title":"Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism","date":"2023-03-04","arxiv_id":"2303.02387","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":0}}],"papers_shown":30,"tasks":[{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":20},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":10},{"task":"/task/image-classification","name":"Image Classification","papers":10},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/clustering","name":"Clustering","papers":6},{"task":"/task/question-answering","name":"Question Answering","papers":5},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":5},{"task":"/task/survival-analysis","name":"Survival Analysis","papers":5},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":5},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":4},{"task":"/task/object-detection","name":"Object Detection","papers":4},{"task":"/task/prognosis","name":"Prognosis","papers":4},{"task":"/task/semi-supervised-image-classification","name":"Semi-Supervised Image Classification","papers":4},{"task":"/task/image-classification","name":"image-classification","papers":4},{"task":"/task/object-detection-1","name":"object-detection","papers":4},{"task":"/task/diversity","name":"Diversity","papers":3},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":3},{"task":"/task/self-supervised-image-classification","name":"Self-Supervised Image Classification","papers":3},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":3},{"task":"/task/16k","name":"16k","papers":2}],"tasks_shown":20,"n_tasks":106,"usage_by_year":[{"year":"2017","papers":2},{"year":"2018","papers":3},{"year":"2019","papers":1},{"year":"2020","papers":6},{"year":"2021","papers":16},{"year":"2022","papers":16},{"year":"2023","papers":18},{"year":"2024","papers":9},{"year":"2025","papers":6}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/lars"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}