Methods › General › Large Batch Optimization › Adafactor
Adafactor
Introduced by Noam Shazeer et al. in Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of adaptivity. This is achieved through maintaining a factored representation of the squared gradient accumulator across training steps. Specifically, by tracking moving averages of the row and column sums of the squared gradients for matrix-valued variables, we are able to reconstruct a low-rank approximation of the exponentially smoothed accumulator at each training step that is optimal with respect to the generalized Kullback-Leibler divergence. For an n ×m matrix, this reduces the memory requirements from O(n m) to O(n + m).
Instead of defining the optimization algorithm in terms of absolute step sizes {αₜ}ₜ₌₁ᵀ, the authors define the optimization algorithm in terms of relative step sizes {ρₜ}ₜ₌₁ᵀ, which get multiplied by the scale of the parameters. The scale of a parameter vector or matrix is defined as the root-mean-square of its components, lower-bounded by a small constant ϵ₂. The reason for this lower bound is to allow zero-initialized parameters to escape 0.
Proposed hyperparameters are: ϵ₁ = 10⁻³⁰, ϵ₂ = 10⁻³, d=1, pₜ = min(10⁻², 1/(√(t))), β̂_(2ₜ) = 1 - t^(-0.8).
In Syntology harvested code stored, below machine verification (unverified_code; harvested), served level V1; retrieve with get_harvested_code_for_paper(arxiv_id='1804.04235'). Repositories: 5 listed of 5 in the graph, strongest link: mentioned on GitHub. The lookup matched a paper by its title, arXiv:1804.04235, not a method entry. This answer is about arXiv:1804.04235, the paper the archive credits with this method.
Origin paper of this answer, arXiv:1804.04235 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 733 · with a code link: 363 · where Syntology ran a sample: 103 (89 with a run with no instrument failure, 14 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (103 of 733 tagged: 89 with a run with no instrument failure, 14 where every run was a failure of Syntology's instrument)
Page 1 of 8: papers 1 to 100 of 733, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems 8 Jul 2025 · 0 repositories · arXiv:2507.05940
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
-
Fretting-Transformer: Encoder-Decoder Model for MIDI to Tablature Transcription 17 Jun 2025 · 0 repositories · arXiv:2506.14223
-
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation 9 Jun 2025 · 0 repositories · arXiv:2506.08210
-
A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair 5 Jun 2025 · 0 repositories · arXiv:2506.04987
-
Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking 29 May 2025 · 0 repositories · arXiv:2505.23117
-
ShIOEnv: A CLI Behavior-Capturing Environment Enabling Grammar-Guided Command Synthesis for Dataset Curation 23 May 2025 · 1 repository · arXiv:2505.18374
-
LogiCase: Effective Test Case Generation from Logical Description in Competitive Programming 21 May 2025 · 0 repositories · arXiv:2505.15039Syntology 11 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 8 unverified (of 19 harvested samples) · 19 pointer-only (licence)
-
EEG-to-Text Translation: A Model for Deciphering Human Brain Activity 20 May 2025 · 1 repository · arXiv:2505.13936
-
Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation 16 May 2025 · 1 repository · arXiv:2505.11754
-
Multilingual Machine Translation with Quantum Encoder Decoder Attention-based Convolutional Variational Circuits 14 May 2025 · 0 repositories · arXiv:2505.09407
-
Cardioformer: Advancing AI in ECG Analysis with Multi-Granularity Patching and ResNet 8 May 2025 · 1 repository · arXiv:2505.05538
-
Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization 8 May 2025 · 0 repositories · arXiv:2505.05070
-
GASCADE: Grouped Summarization of Adverse Drug Event for Enhanced Cancer Pharmacovigilance 7 May 2025 · 1 repository · arXiv:2505.04284
-
A review of DNA restriction-free overlapping sequence cloning techniques for synthetic biology 6 May 2025 · 0 repositories · arXiv:2505.03681
-
JaccDiv: A Metric and Benchmark for Quantifying Diversity of Generated Marketing Text in the Music Industry 29 Apr 2025 · 0 repositories · arXiv:2504.20849
-
Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks 28 Apr 2025 · 0 repositories · arXiv:2504.19444
-
Fast-Powerformer: A Memory-Efficient Transformer for Accurate Mid-Term Wind Power Forecasting 15 Apr 2025 · 0 repositories · arXiv:2504.10923
-
Sigma: A dataset for text-to-code semantic parsing with statistical analysis 5 Apr 2025 · 1 repository · arXiv:2504.04301
-
Advancing Sentiment Analysis in Tamil-English Code-Mixed Texts: Challenges and Transformer-Based Solutions 30 Mar 2025 · 0 repositories · arXiv:2503.23295
-
Enhancing Knowledge Graph Completion with Entity Neighborhood and Relation Context 29 Mar 2025 · 0 repositories · arXiv:2503.23205
-
Scaling Down Text Encoders of Text-to-Image Diffusion Models 25 Mar 2025 · 1 repository · arXiv:2503.19897
-
Exploring Training and Inference Scaling Laws in Generative Retrieval 24 Mar 2025 · 1 repository · arXiv:2503.18941
-
Predicting the Road Ahead: A Knowledge Graph based Foundation Model for Scene Understanding in Autonomous Driving 24 Mar 2025 · 0 repositories · arXiv:2503.18730
-
Enhancing Code LLM Training with Programmer Attention 19 Mar 2025 · 0 repositories · arXiv:2503.14936
-
DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Models 17 Mar 2025 · 0 repositories · arXiv:2503.12885
-
ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documents 13 Mar 2025 · 0 repositories · arXiv:2503.10233
-
A LongFormer-Based Framework for Accurate and Efficient Medical Text Summarization 10 Mar 2025 · 0 repositories · arXiv:2503.06888
-
Roamify: Designing and Evaluating an LLM Based Google Chrome Extension for Personalised Itinerary Planning 10 Mar 2025 · 1 repository · arXiv:2504.10489
-
Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform 9 Mar 2025 · 0 repositories · arXiv:2503.06676
-
MoEMoE: Question Guided Dense and Scalable Sparse Mixture-of-Expert for Multi-source Multi-modal Answering 8 Mar 2025 · 0 repositories · arXiv:2503.06296
-
A Transformer Model for Predicting Chemical Reaction Products from Generic Templates 4 Mar 2025 · 0 repositories · arXiv:2503.05810
-
Multimodal Representation Alignment for Image Generation: Text-Image Interleaved Control Is Easier Than You Think 27 Feb 2025 · 1 repository · arXiv:2502.20172
-
LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation 25 Feb 2025 · 0 repositories · arXiv:2502.18302
-
A Fine-Tuning Approach for T5 Using Knowledge Graphs to Address Complex Tasks 23 Feb 2025 · 0 repositories · arXiv:2502.16484
-
Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correction 21 Feb 2025 · 0 repositories · arXiv:2502.15261
-
Empowering LLMs with Logical Reasoning: A Comprehensive Survey 21 Feb 2025 · 0 repositories · arXiv:2502.15652
-
Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognition 20 Feb 2025 · 1 repository · arXiv:2502.14195
-
Order-agnostic Identifier for Large Language Model-based Generative Recommendation 15 Feb 2025 · 0 repositories · arXiv:2502.10833
-
A Large-Scale Benchmark for Vietnamese Sentence Paraphrases 11 Feb 2025 · 1 repository · arXiv:2502.07188
-
Improving Adaptive Moment Optimization via Preconditioner Diagonalization 11 Feb 2025 · 0 repositories · arXiv:2502.07488
-
Robust and Secure Code Watermarking for Large Language Models via ML/Crypto Codesign 4 Feb 2025 · 0 repositories · arXiv:2502.02068
-
General Embedding vs. Task-Specific Embedding: A Comparative Approach to Enhancing NLP Performance 30 Jan 2025 · 0 repositories
-
Visualizing Uncertainty in Translation Tasks: An Evaluation of LLM Performance and Confidence Metrics 26 Jan 2025 · 1 repository · arXiv:2501.17187
-
Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning 24 Jan 2025 · 0 repositories · arXiv:2501.14680
-
Confidence Estimation for Error Detection in Text-to-SQL Systems 16 Jan 2025 · 1 repository · arXiv:2501.09527
-
Exploring Robustness of Multilingual LLMs on Real-World Noisy Data 14 Jan 2025 · 1 repository · arXiv:2501.08322
-
MathReader : Text-to-Speech for Mathematical Documents 13 Jan 2025 · 1 repository · arXiv:2501.07088
-
Bridging Dialects: Translating Standard Bangla to Regional Variants Using Neural Models 10 Jan 2025 · 0 repositories · arXiv:2501.05749
-
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions 8 Jan 2025 · 0 repositories · arXiv:2501.04437
-
How to Select Pre-Trained Code Models for Reuse? A Learning Perspective 7 Jan 2025 · 1 repository · arXiv:2501.03783
-
Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models 5 Jan 2025 · 0 repositories · arXiv:2501.02599
-
BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion 2 Jan 2025 · 1 repository · arXiv:2501.01069
-
ReFormer: Generating Radio Fakes for Data Augmentation 31 Dec 2024 · 0 repositories · arXiv:2501.00282
-
YAD: Leveraging T5 for Improved Automatic Diacritization of Yorùbá Text 28 Dec 2024 · 1 repository · arXiv:2412.20218
-
HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis 25 Dec 2024 · 0 repositories · arXiv:2412.18981
-
Optimizing Large Language Models with an Enhanced LoRA Fine-Tuning Algorithm for Efficiency and Robustness in NLP Tasks 25 Dec 2024 · 0 repositories · arXiv:2412.18729
-
Advancing Explainability in Neural Machine Translation: Analytical Metrics for Attention and Alignment Consistency 24 Dec 2024 · 0 repositories · arXiv:2412.18669
-
HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation 24 Dec 2024 · 0 repositories · arXiv:2412.18524
-
Identifying Cyberbullying Roles in Social Media 21 Dec 2024 · 0 repositories · arXiv:2412.16417
-
Adversarial Robustness through Dynamic Ensemble Learning 20 Dec 2024 · 0 repositories · arXiv:2412.16254
-
Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation 20 Dec 2024 · 0 repositories · arXiv:2412.16135
-
The First Multilingual Model For The Detection of Suicide Texts 20 Dec 2024 · 0 repositories · arXiv:2412.15498
-
LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning 17 Dec 2024 · 0 repositories · arXiv:2412.12541
-
Graph-Guided Textual Explanation Generation Framework 16 Dec 2024 · 0 repositories · arXiv:2412.12318
-
Neural Text Normalization for Luxembourgish using Real-Life Variation Data 12 Dec 2024 · 0 repositories · arXiv:2412.09383
-
SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization 12 Dec 2024 · 1 repository · arXiv:2412.08894
-
Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy 12 Dec 2024 · 0 repositories · arXiv:2412.09415
-
Are Clinical T5 Models Better for Clinical Text? 8 Dec 2024 · 1 repository · arXiv:2412.05845
-
Leveraging Large Language Models for Comparative Literature Summarization with Reflective Incremental Mechanisms 3 Dec 2024 · 0 repositories · arXiv:2412.02149
-
SiTSE: Sinhala Text Simplification Dataset and Evaluation 2 Dec 2024 · 1 repository · arXiv:2412.01293
-
A Comprehensive Guide to Explainable AI: From Classical Models to LLMs 1 Dec 2024 · 1 repository · arXiv:2412.00800
-
Towards Santali Linguistic Inclusion: Building the First Santali-to-English Translation Model using mT5 Transformer and Data Augmentation 29 Nov 2024 · 0 repositories · arXiv:2411.19726
-
An Extensive Evaluation of Factual Consistency in Large Language Models for Data-to-Text Generation 28 Nov 2024 · 0 repositories · arXiv:2411.19203
-
A survey on cutting-edge relation extraction techniques based on language models 27 Nov 2024 · 0 repositories · arXiv:2411.18157
-
Generative Fuzzy System for Sequence Generation 21 Nov 2024 · 0 repositories · arXiv:2411.13867
-
A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation 19 Nov 2024 · 0 repositories · arXiv:2411.12157
-
Leveraging Multimodal Models for Enhanced Neuroimaging Diagnostics in Alzheimer's Disease 12 Nov 2024 · 0 repositories · arXiv:2411.07871
-
Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models 4 Nov 2024 · 1 repository · arXiv:2411.02083Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
Training-free Regional Prompting for Diffusion Transformers 4 Nov 2024 · 1 repository · arXiv:2411.02395Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm 30 Oct 2024 · 1 repository · arXiv:2410.23182
-
Building A Coding Assistant via the Retrieval-Augmented Language Model 21 Oct 2024 · 1 repository · arXiv:2410.16229
-
Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerland 17 Oct 2024 · 0 repositories · arXiv:2410.13456
-
Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5 15 Oct 2024 · 0 repositories · arXiv:2410.11627
-
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers 14 Oct 2024 · 2 repositories · arXiv:2410.10629
-
Single Ground Truth Is Not Enough: Add Linguistic Variability to Aspect-based Sentiment Analysis Evaluation 13 Oct 2024 · 0 repositories · arXiv:2410.09807
-
Enhancing SPARQL Generation by Triplet-order-sensitive Pre-training 8 Oct 2024 · 1 repository · arXiv:2410.05731
-
Leveraging free energy in pretraining model selection for improved fine-tuning 8 Oct 2024 · 0 repositories · arXiv:2410.05612
-
Deep Transfer Learning Based Peer Review Aggregation and Meta-review Generation for Scientific Articles 5 Oct 2024 · 0 repositories · arXiv:2410.04202
-
GPT-4o as the Gold Standard: A Scalable and General Purpose Approach to Filter Language Model Pretraining Data 3 Oct 2024 · 0 repositories · arXiv:2410.02755
-
Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement 2 Oct 2024 · 0 repositories · arXiv:2410.01869
-
GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction 1 Oct 2024 · 0 repositories · arXiv:2410.00380
-
Abstractive Summarization of Low resourced Nepali language using Multilingual Transformers 29 Sep 2024 · 0 repositories · arXiv:2409.19566
-
Enhancing Automatic Keyphrase Labelling with Text-to-Text Transfer Transformer (T5) Architecture: A Framework for Keyphrase Generation and Filtering 25 Sep 2024 · 0 repositories · arXiv:2409.16760
-
Probing Omissions and Distortions in Transformer-based RDF-to-Text Models 25 Sep 2024 · 0 repositories · arXiv:2409.16707
-
Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs 24 Sep 2024 · 0 repositories · arXiv:2409.16371
-
Improving Academic Skills Assessment with NLP and Ensemble Learning 23 Sep 2024 · 0 repositories · arXiv:2409.19013
-
EMMeTT: Efficient Multimodal Machine Translation Training 20 Sep 2024 · 0 repositories · arXiv:2409.13523
-
Exploring Large Language Models for Product Attribute Value Identification 19 Sep 2024 · 0 repositories · arXiv:2409.12695
-
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions 19 Sep 2024 · 1 repository · arXiv:2409.12958
Tasks archive 2025-07-28
20 shown of 492 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections