Methods › Natural Language Processing › Language Models › ULMFiT
Universal Language Model Fine-tuning
ULMFiT
Introduced by Jeremy Howard et al. in Universal Language Model Fine-tuning for Text Classification
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Universal Language Model Fine-tuning, or ULMFiT, is an architecture and transfer learning method that can be applied to NLP tasks. It involves a 3-layer AWD-LSTM architecture for its representations. The training consists of three steps: 1) general language model pre-training on a Wikipedia-based text, 2) fine-tuning the language model on a target task, and 3) fine-tuning the classifier on the target task.
As different layers capture different types of information, they are fine-tuned to different extents using discriminative fine-tuning. Training is performed using Slanted triangular learning rates (STLR), a learning rate scheduling strategy that first linearly increases the learning rate and then linearly decays it.
Fine-tuning the target classifier is achieved in ULMFiT using gradual unfreezing. Rather than fine-tuning all layers at once, which risks catastrophic forgetting, ULMFiT gradually unfreezes the model starting from the last layer (i.e., closest to the output) as this contains the least general knowledge. First the last layer is unfrozen and all unfrozen layers are fine-tuned for one epoch. Then the next group of frozen layers is unfrozen and fine-tuned and repeat, until all layers are fine-tuned until convergence at the last iteration.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 40 · with a code link: 17 · where Syntology ran a sample: 1 (0 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (1 of 40 tagged: 0 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 40 of 40, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Advanced Deep Learning Techniques for Analyzing Earnings Call Transcripts: Methodologies and Applications 27 Feb 2025 · 0 repositories · arXiv:2503.01886
-
No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts 24 Oct 2024 · 0 repositories · arXiv:2410.19184
-
RICo: Reddit ideological communities 5 Jun 2024 · 1 repository
-
Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study 15 Dec 2023 · 0 repositories
-
Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information 8 Dec 2023 · 0 repositories · arXiv:2312.04944
-
Explainable and High-Performance Hate and Offensive Speech Detection 26 Jun 2022 · 0 repositories · arXiv:2206.12983
-
IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages 15 Nov 2021 · 1 repository · arXiv:2111.07906
-
Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling 27 Aug 2021 · 1 repository · arXiv:2108.12177
-
Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts 24 Aug 2021 · 1 repository · arXiv:2108.10939
-
Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System 20 Jul 2021 · 0 repositories · arXiv:2107.09625
-
WHOSe Heritage: Classification of UNESCO World Heritage "Outstanding Universal Value" Documents with Soft Labels 12 Apr 2021 · 1 repository · arXiv:2104.05547
-
L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset 21 Mar 2021 · 1 repository · arXiv:2103.11408
-
Experimental Evaluation of Deep Learning models for Marathi Text Classification 13 Jan 2021 · 0 repositories · arXiv:2101.04899
-
LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT 13 Jan 2021 · 1 repository · arXiv:2101.04965
-
HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection 1 Dec 2020 · 2 repositories
-
Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating 1 Dec 2020 · 0 repositories
-
Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification 18 Nov 2020 · 0 repositories · arXiv:2011.09448
-
Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language 13 Oct 2020 · 2 repositories · arXiv:2010.06447
-
Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection 5 Oct 2020 · 0 repositories · arXiv:2010.02094
-
FarsTail: A Persian Natural Language Inference Dataset 18 Sep 2020 · 1 repository · arXiv:2009.08820
-
Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication 12 Sep 2020 · 0 repositories · arXiv:2009.05836
-
HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection 22 Aug 2020 · 2 repositories · arXiv:2008.09820
-
Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings 1 May 2020 · 0 repositories
-
Offensive language detection in Arabic using ULMFiT 1 May 2020 · 1 repository
-
Text Categorization for Conflict Event Annotation 1 May 2020 · 0 repositories
-
Inferring the source of official texts: can SVM beat ULMFiT? 2 Mar 2020 · 1 repository
-
Localized Flood DetectionWith Minimal Labeled Social Media Data Using Transfer Learning 10 Feb 2020 · 0 repositories · arXiv:2003.04973
-
Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks 28 Dec 2019 · 0 repositories · arXiv:1912.12397
-
A Comparative Study of Pretrained Language Models on Thai Social Text Categorization 3 Dec 2019 · 0 repositories · arXiv:1912.01580
-
Evolution of transfer learning in natural language processing 16 Oct 2019 · 0 repositories · arXiv:1910.07370
-
The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews 2 Oct 2019 · 2 repositories · arXiv:1910.00896
-
Analyzing Customer Feedback for Product Fit Prediction 28 Aug 2019 · 0 repositories · arXiv:1908.10896
-
Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches 17 Jul 2019 · 0 repositories · arXiv:1907.07543
-
Evaluating Language Model Finetuning Techniques for Low-resource Languages 30 Jun 2019 · 2 repositories · arXiv:1907.00409
-
Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish 17 Jun 2019 · 0 repositories · arXiv:1906.09325
-
Figure Eight at SemEval-2019 Task 3: Ensemble of Transfer Learning Methods for Contextual Emotion Detection 1 Jun 2019 · 0 repositories
-
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment 1 Jun 2019 · 0 repositories
-
An Empirical Evaluation of Text Representation Schemes on Multilingual Social Web to Filter the Textual Aggression 16 Apr 2019 · 0 repositories · arXiv:1904.08770
-
Low Resource Text Classification with ULMFit and Backtranslation 21 Mar 2019 · 1 repository · arXiv:1903.09244
-
Universal Language Model Fine-tuning for Text Classification 18 Jan 2018 · 66 repositories · arXiv:1801.06146Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 5 harvested samples) · 3 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 51 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections