Methods › General › Large Batch Optimization › LAMB
LAMB
Introduced by Yang You et al. in Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
LAMB is a a layerwise adaptive large batch optimization technique. It provides a strategy for adapting the learning rate in large batch settings. LAMB uses Adam as the base algorithm and then forms an update as:
rₜ = mₜ/(√(vₜ) + ϵ) xₜ₊₁⁽ⁱ⁾ = xₜ⁽ⁱ⁾ - ηₜ(ϕ(|| xₜ⁽ⁱ⁾ ||))/(|| mₜ⁽ⁱ⁾ ||)(rₜ⁽ⁱ⁾+λxₜ⁽ⁱ⁾)
Unlike LARS, the adaptivity of LAMB is two-fold: (i) per dimension normalization with respect to the square root of the second moment used in Adam and (ii) layerwise normalization obtained due to layerwise adaptivity.
Papers archive 2025-07-28
30 shown of 199, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation 12 Jun 2025 · 0 repositories · arXiv:2506.10524
-
Rapid yet accurate Tile-circuit and device modeling for Analog In-Memory Computing 5 May 2025 · 0 repositories · arXiv:2506.00004
-
Don't Fight Hallucinations, Use Them: Estimating Image Realism using NLI over Atomic Facts 20 Mar 2025 · 1 repository · arXiv:2503.15948
-
Efficient or Powerful? Trade-offs Between Machine Learning and Deep Learning for Mental Illness Detection on Social Media 3 Mar 2025 · 0 repositories · arXiv:2503.01082
-
Robust Bias Detection in MLMs and its Application to Human Trait Ratings 21 Feb 2025 · 1 repository · arXiv:2502.15600
-
Meursault as a Data Point 3 Feb 2025 · 0 repositories · arXiv:2502.01364
-
Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing 10 Jan 2025 · 1 repository · arXiv:2501.06278
-
TradingAgents: Multi-Agents LLM Financial Trading Framework 28 Dec 2024 · 1 repository · arXiv:2412.20138Syntology ran 0 of 14 samples · 14 unverified
-
A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit 23 Nov 2024 · 1 repository · arXiv:2411.15404
-
BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI 21 Nov 2024 · 1 repository · arXiv:2411.14254
-
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm 30 Oct 2024 · 1 repository · arXiv:2410.23182
-
A Bayesian Perspective on the Maximum Score Problem 22 Oct 2024 · 0 repositories · arXiv:2410.17153
-
Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning 27 Sep 2024 · 0 repositories · arXiv:2409.19075
-
Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection 19 Sep 2024 · 1 repository · arXiv:2409.12541
-
BioMNER: A Dataset for Biomedical Method Entity Recognition 28 Jun 2024 · 0 repositories · arXiv:2406.20038
-
Concept Formation and Alignment in Language Models: Bridging Statistical Patterns in Latent Space to Concept Taxonomy 8 Jun 2024 · 0 repositories · arXiv:2406.05315
-
Effect of antibody levels on the spread of disease in multiple infections 31 May 2024 · 0 repositories · arXiv:2405.20702
-
CEEBERT: Cross-Domain Inference in Early Exit BERT 23 May 2024 · 1 repository · arXiv:2405.15039Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
-
A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media 1 May 2024 · 0 repositories · arXiv:2405.00903
-
Exploring Internal Numeracy in Language Models: A Case Study on ALBERT 25 Apr 2024 · 0 repositories · arXiv:2404.16574
-
Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge 20 Apr 2024 · 1 repository · arXiv:2404.13292
-
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? 27 Mar 2024 · 0 repositories · arXiv:2403.18338
-
CRISPR: Ensemble Model 5 Mar 2024 · 0 repositories · arXiv:2403.03018
-
Breaking MLPerf Training: A Case Study on Optimizing BERT 4 Feb 2024 · 0 repositories · arXiv:2402.02447
-
An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks 8 Jan 2024 · 1 repository · arXiv:2401.03653
-
Tensor-Aware Energy Accounting 19 Nov 2023 · 1 repository · arXiv:2311.11424
-
Generative AI for Hate Speech Detection: Evaluation and Findings 16 Nov 2023 · 0 repositories · arXiv:2311.09993
-
Revisiting LARS for Large Batch Training Generalization of Neural Networks 25 Sep 2023 · 0 repositories · arXiv:2309.14053
-
AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer 22 Sep 2023 · 1 repository · arXiv:2309.12689
-
Identification of the Relevance of Comments in Codes Using Bag of Words and Transformer Based Models 11 Aug 2023 · 1 repository · arXiv:2308.06144
Tasks archive 2025-07-28
20 shown of 194 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections