Methods › General › Large Batch Optimization › LAMB
LAMB
Introduced by Yang You et al. in Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
LAMB is a a layerwise adaptive large batch optimization technique. It provides a strategy for adapting the learning rate in large batch settings. LAMB uses Adam as the base algorithm and then forms an update as:
rₜ = mₜ/(√(vₜ) + ϵ) xₜ₊₁⁽ⁱ⁾ = xₜ⁽ⁱ⁾ - ηₜ(ϕ(|| xₜ⁽ⁱ⁾ ||))/(|| mₜ⁽ⁱ⁾ ||)(rₜ⁽ⁱ⁾+λxₜ⁽ⁱ⁾)
Unlike LARS, the adaptivity of LAMB is two-fold: (i) per dimension normalization with respect to the square root of the second moment used in Adam and (ii) layerwise normalization obtained due to layerwise adaptivity.
In Syntology repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). The lookup matched a paper by its title, the paper Syntology's graph keys as 2025.acl-short.96, not a method entry. The archive credits no paper with an arXiv id for this method, so whether this answer (about the paper Syntology's graph keys as 2025.acl-short.96) is about this method is not checked.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 199 · with a code link: 86 · where Syntology ran a sample: 21 (15 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (21 of 199 tagged: 15 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 199, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
ALBERT: Advanced Localization and Bidirectional Encoder Representations from Transformers for Automotive Damage Evaluation 12 Jun 2025 · 0 repositories · arXiv:2506.10524
-
Rapid yet accurate Tile-circuit and device modeling for Analog In-Memory Computing 5 May 2025 · 0 repositories · arXiv:2506.00004
-
Don't Fight Hallucinations, Use Them: Estimating Image Realism using NLI over Atomic Facts 20 Mar 2025 · 1 repository · arXiv:2503.15948
-
Efficient or Powerful? Trade-offs Between Machine Learning and Deep Learning for Mental Illness Detection on Social Media 3 Mar 2025 · 0 repositories · arXiv:2503.01082
-
Robust Bias Detection in MLMs and its Application to Human Trait Ratings 21 Feb 2025 · 1 repository · arXiv:2502.15600
-
Meursault as a Data Point 3 Feb 2025 · 0 repositories · arXiv:2502.01364
-
Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing 10 Jan 2025 · 1 repository · arXiv:2501.06278
-
TradingAgents: Multi-Agents LLM Financial Trading Framework 28 Dec 2024 · 1 repository · arXiv:2412.20138Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 9 unverified (of 14 harvested samples)
-
A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit 23 Nov 2024 · 1 repository · arXiv:2411.15404
-
BERT-Based Approach for Automating Course Articulation Matrix Construction with Explainable AI 21 Nov 2024 · 1 repository · arXiv:2411.14254
-
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm 30 Oct 2024 · 1 repository · arXiv:2410.23182
-
A Bayesian Perspective on the Maximum Score Problem 22 Oct 2024 · 0 repositories · arXiv:2410.17153
-
Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning 27 Sep 2024 · 0 repositories · arXiv:2409.19075
-
Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection 19 Sep 2024 · 1 repository · arXiv:2409.12541
-
BioMNER: A Dataset for Biomedical Method Entity Recognition 28 Jun 2024 · 0 repositories · arXiv:2406.20038
-
Concept Formation and Alignment in Language Models: Bridging Statistical Patterns in Latent Space to Concept Taxonomy 8 Jun 2024 · 0 repositories · arXiv:2406.05315
-
Effect of antibody levels on the spread of disease in multiple infections 31 May 2024 · 0 repositories · arXiv:2405.20702
-
CEEBERT: Cross-Domain Inference in Early Exit BERT 23 May 2024 · 1 repository · arXiv:2405.15039Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
A Named Entity Recognition and Topic Modeling-based Solution for Locating and Better Assessment of Natural Disasters in Social Media 1 May 2024 · 0 repositories · arXiv:2405.00903
-
Exploring Internal Numeracy in Language Models: A Case Study on ALBERT 25 Apr 2024 · 0 repositories · arXiv:2404.16574
-
Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge 20 Apr 2024 · 1 repository · arXiv:2404.13292
-
mALBERT: Is a Compact Multilingual BERT Model Still Worth It? 27 Mar 2024 · 0 repositories · arXiv:2403.18338
-
CRISPR: Ensemble Model 5 Mar 2024 · 0 repositories · arXiv:2403.03018
-
Breaking MLPerf Training: A Case Study on Optimizing BERT 4 Feb 2024 · 0 repositories · arXiv:2402.02447
-
An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks 8 Jan 2024 · 1 repository · arXiv:2401.03653
-
Tensor-Aware Energy Accounting 19 Nov 2023 · 1 repository · arXiv:2311.11424
-
Generative AI for Hate Speech Detection: Evaluation and Findings 16 Nov 2023 · 0 repositories · arXiv:2311.09993
-
Revisiting LARS for Large Batch Training Generalization of Neural Networks 25 Sep 2023 · 0 repositories · arXiv:2309.14053
-
AMPLIFY:Attention-based Mixup for Performance Improvement and Label Smoothing in Transformer 22 Sep 2023 · 1 repository · arXiv:2309.12689
-
Identification of the Relevance of Comments in Codes Using Bag of Words and Transformer Based Models 11 Aug 2023 · 1 repository · arXiv:2308.06144
-
Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection 9 Aug 2023 · 1 repository · arXiv:2308.04950
-
Gradient-Based Word Substitution for Obstinate Adversarial Examples Generation in Language Models 24 Jul 2023 · 0 repositories · arXiv:2307.12507
-
CAME: Confidence-guided Adaptive Memory Efficient Optimization 5 Jul 2023 · 2 repositories · arXiv:2307.02047Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result (of 2 harvested samples) · 1 pointer-only (licence)
-
SparseOptimizer: Sparsify Language Models through Moreau-Yosida Regularization and Accelerate via Compiler Co-design 27 Jun 2023 · 0 repositories · arXiv:2306.15656
-
MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates 2 Jun 2023 · 1 repository · arXiv:2306.01685Syntology official (archive's flag): 9 ran · 9 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks 21 May 2023 · 0 repositories · arXiv:2305.11916
-
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning 29 Mar 2023 · 1 repository · arXiv:2303.16445
-
Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures 25 Mar 2023 · 0 repositories · arXiv:2303.16100
-
Elementwise Language Representation 27 Feb 2023 · 0 repositories · arXiv:2302.13475
-
Clinical BioBERT Hyperparameter Optimization using Genetic Algorithm 8 Feb 2023 · 0 repositories · arXiv:2302.03822
-
A Stability Analysis of Fine-Tuning a Pre-Trained Model 24 Jan 2023 · 0 repositories · arXiv:2301.09820
-
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers 5 Jan 2023 · 7 repositories · arXiv:2301.02111Syntology community repositories only · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 1 pointer-only (licence)
-
Floods Relevancy and Identification of Location from Twitter Posts using NLP Techniques 1 Jan 2023 · 0 repositories · arXiv:2301.00321
-
Fast FullSubNet: Accelerate Full-band and Sub-band Fusion Model for Single-channel Speech Enhancement 18 Dec 2022 · 1 repository · arXiv:2212.09019Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 11 harvested samples)
-
From Xception to NEXcepTion: New Design Decisions and Neural Architecture Search 16 Dec 2022 · 1 repository · arXiv:2212.08448
-
Learning-To-Embed: Adopting Transformer based models for E-commerce Products Representation Learning 7 Dec 2022 · 0 repositories · arXiv:2212.03725
-
Entity-Assisted Language Models for Identifying Check-worthy Sentences 19 Nov 2022 · 0 repositories · arXiv:2211.10678
-
Dark patterns in e-commerce: a dataset and its baseline evaluations 12 Nov 2022 · 1 repository · arXiv:2211.06543
-
The Architectural Bottleneck Principle 11 Nov 2022 · 0 repositories · arXiv:2211.06420
-
Collateral facilitation in humans and language models 9 Nov 2022 · 1 repository · arXiv:2211.05198Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Large-batch Optimization for Dense Visual Predictions 20 Oct 2022 · 1 repository · arXiv:2210.11078
-
Vote'n'Rank: Revision of Benchmarking with Social Choice Theory 11 Oct 2022 · 1 repository · arXiv:2210.05769Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 11 unverified (of 17 harvested samples)
-
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs 6 Oct 2022 · 1 repository · arXiv:2210.03052
-
Monolith: Real Time Recommendation System With Collisionless Embedding Table 16 Sep 2022 · 1 repository · arXiv:2209.07663
-
DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text 12 Sep 2022 · 0 repositories · arXiv:2209.05286
-
Addressing Token Uniformity in Transformers via Singular Value Transformation 24 Aug 2022 · 1 repository · arXiv:2208.11790Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
A Comparative Study on COVID-19 Fake News Detection Using Different Transformer Based Models 2 Aug 2022 · 0 repositories · arXiv:2208.01355
-
KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP 21 Jun 2022 · 0 repositories · arXiv:2206.10265
-
Emoji-based Fine-grained Attention Network for Sentiment Analysis in the Microblog Comments 10 Jun 2022 · 0 repositories · arXiv:2206.12262
-
Life after BERT: What do Other Muppets Understand about Language? 21 May 2022 · 1 repository · arXiv:2205.10696
-
Large Scale Transfer Learning for Differentially Private Image Classification 6 May 2022 · 0 repositories · arXiv:2205.02973
-
ALBETO and DistilBETO: Lightweight Spanish Language Models 19 Apr 2022 · 2 repositories · arXiv:2204.09145
-
Accelerating Attention through Gradient-Based Learned Runtime Pruning 7 Apr 2022 · 0 repositories · arXiv:2204.03227
-
PALBERT: Teaching ALBERT to Ponder 7 Apr 2022 · 1 repository · arXiv:2204.03276Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result (of 2 harvested samples)
-
Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding 6 Apr 2022 · 0 repositories · arXiv:2204.02922
-
ANNA: Enhanced Language Representation for Question Answering 28 Mar 2022 · 0 repositories · arXiv:2203.14507
-
Mono vs Multilingual BERT: A Case Study in Hindi and Marathi Named Entity Recognition 24 Mar 2022 · 0 repositories · arXiv:2203.12907
-
Detecting and Diagnosing Terrestrial Gravitational-Wave Mimics Through Feature Learning 9 Mar 2022 · 0 repositories · arXiv:2203.05086
-
Does Transliteration Help Multilingual Language Modeling? 29 Jan 2022 · 1 repository · arXiv:2201.12501Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
An Application of Pseudo-Log-Likelihoods to Natural Language Scoring 23 Jan 2022 · 0 repositories · arXiv:2201.09377
-
Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework 16 Jan 2022 · 0 repositories
-
DECK: Behavioral Tests to Improve Interpretability and Generalizability of BERT Models Detecting Depression from Text 16 Jan 2022 · 0 repositories
-
Multi-Stage Pre-Training for Math-Understanding: μ²(AL)BERT 16 Jan 2022 · 0 repositories
-
PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting 16 Jan 2022 · 0 repositories
-
A Comparative Study of Transformers on Word Sense Disambiguation 30 Nov 2021 · 0 repositories · arXiv:2111.15417
-
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion 24 Nov 2021 · 1 repository · arXiv:2111.12417Syntology 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 2 pointer-only (licence)
-
How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task 18 Nov 2021 · 1 repository · arXiv:2111.09612
-
ANNA: Enhanced Language Representation for Question Answering 16 Nov 2021 · 0 repositories
-
Life after BERT: What do Other Muppets Understand about Language? 16 Nov 2021 · 0 repositories
-
PALBERT: Teaching ALBERT to Ponder 16 Nov 2021 · 0 repositories
-
Understanding Attention in Machine Reading Comprehension 16 Nov 2021 · 0 repositories
-
Character-level HyperNetworks for Hate Speech Detection 11 Nov 2021 · 1 repository · arXiv:2111.06336
-
The Golden Rule as a Heuristic to Measure the Fairness of Texts Using Machine Learning 29 Oct 2021 · 0 repositories · arXiv:2111.00107
-
Bridge the Gap Between CV and NLP! A Gradient-based Textual Adversarial Attack Framework 28 Oct 2021 · 1 repository · arXiv:2110.15317
-
Ensemble ALBERT on SQuAD 2.0 19 Oct 2021 · 1 repository · arXiv:2110.09665
-
Local and Global Context-Based Pairwise Models for Sentence Ordering 8 Oct 2021 · 1 repository · arXiv:2110.04291
-
Transliteration: A Simple Technique For Improving Multilingual Language Modeling 29 Sep 2021 · 0 repositories
-
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers 17 Sep 2021 · 0 repositories · arXiv:2109.08406
-
Transformer-based Language Models for Factoid Question Answering at BioASQ9b 15 Sep 2021 · 1 repository · arXiv:2109.07185
-
So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements 2 Sep 2021 · 0 repositories · arXiv:2109.01226
-
Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech 1 Sep 2021 · 0 repositories · arXiv:2109.00591
-
Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models 26 Aug 2021 · 1 repository · arXiv:2108.11574
-
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks 1 Aug 2021 · 1 repository
-
DeepBlueAI at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Stacking Diverse Language Model-Based Methods 1 Aug 2021 · 0 repositories
-
Gulu at SemEval-2021 Task 7: Detecting and Rating Humor and Offense 1 Aug 2021 · 0 repositories
-
How Many Layers and Why? An Analysis of the Model Depth in Transformers 1 Aug 2021 · 0 repositories
-
hub at SemEval-2021 Task 7: Fusion of ALBERT and Word Frequency Information Detecting and Rating Humor and Offense 1 Aug 2021 · 0 repositories
-
YNU-HPCC at SemEval-2021 Task 5: Using a Transformer-based Model with Auxiliary Information for Toxic Span Detection 1 Aug 2021 · 1 repository
-
YNU-HPCC at SemEval-2021 Task 6: Combining ALBERT and Text-CNN for Persuasion Detection in Texts and Images 1 Aug 2021 · 0 repositories
-
ZYJ at SemEval-2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense with ALBERT-Based Model 1 Aug 2021 · 0 repositories
Tasks archive 2025-07-28
20 shown of 194 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections