{"url":"/method/ulmfit","slug":"ulmfit","name":"ULMFiT","full_name":"Universal Language Model Fine-tuning","full_name_withheld":false,"description_markdown":"**Universal Language Model Fine-tuning**, or **ULMFiT**, is an architecture and transfer learning method that can be applied to NLP tasks. It involves a 3-layer [AWD-LSTM](https://paperswithcode.com/method/awd-lstm) architecture for its representations. The training consists of three steps: 1) general language model pre-training on a Wikipedia-based text, 2) fine-tuning the language model on a target task, and 3) fine-tuning the classifier on the target task.\r\n\r\nAs different layers capture different types of information, they are fine-tuned to different extents using [discriminative fine-tuning](https://paperswithcode.com/method/discriminative-fine-tuning). Training is performed using [Slanted triangular learning rates](https://paperswithcode.com/method/slanted-triangular-learning-rates) (STLR), a learning rate scheduling strategy that first linearly increases the learning rate and then linearly decays it.\r\n\r\nFine-tuning the target classifier is achieved in ULMFiT using gradual unfreezing. Rather than fine-tuning all layers at once, which risks catastrophic forgetting, ULMFiT gradually unfreezes the model starting from the last layer (i.e., closest to the output) as this contains the least general knowledge. First the last layer is unfrozen and all unfrozen layers are fine-tuned for one epoch. Then the next group of frozen layers is unfrozen and fine-tuned and repeat, until all layers are fine-tuned until convergence at the last iteration.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Universal Language Model Fine-tuning for Text Classification","paper":"/paper/universal-language-model-fine-tuning-for-text","first_author":"Jeremy Howard","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/universal-language-model-fine-tuning-for-text"},"source":{"url":"http://arxiv.org/abs/1801.06146v5","title":"Universal Language Model Fine-tuning for Text Classification","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Language Models","url":"/methods/category/language-models","pwc_aliases":[]}],"n_papers_tagged":40,"archive_num_papers":40,"papers_newest_first":[{"paper":null,"title":"Advanced Deep Learning Techniques for Analyzing Earnings Call Transcripts: Methodologies and Applications","date":"2025-02-27","arxiv_id":"2503.01886","n_code_links":0,"syntology":null},{"paper":null,"title":"No Argument Left Behind: Overlapping Chunks for Faster Processing of Arbitrarily Long Legal Texts","date":"2024-10-24","arxiv_id":"2410.19184","n_code_links":0,"syntology":null},{"paper":"/paper/rico-reddit-ideological-communities","title":"RICo: Reddit ideological communities","date":"2024-06-05","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/exploring-multi-level-threats-in-telegram","title":"Exploring Multi-Level Threats in Telegram Data with AI-Human Annotation: A Preliminary Study","date":"2023-12-15","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Illicit Darkweb Classification via Natural-language Processing: Classifying Illicit Content of Webpages based on Textual Information","date":"2023-12-08","arxiv_id":"2312.04944","n_code_links":0,"syntology":null},{"paper":null,"title":"Explainable and High-Performance Hate and Offensive Speech Detection","date":"2022-06-26","arxiv_id":"2206.12983","n_code_links":0,"syntology":null},{"paper":"/paper/iiitt-dravidian-codemix-fire2021","title":"IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages","date":"2021-11-15","arxiv_id":"2111.07906","n_code_links":1,"syntology":null},{"paper":"/paper/offensive-language-identification-in-low","title":"Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling","date":"2021-08-27","arxiv_id":"2108.12177","n_code_links":1,"syntology":null},{"paper":"/paper/sn-computer-science-towards-offensive","title":"Towards Offensive Language Identification for Tamil Code-Mixed YouTube Comments and Posts","date":"2021-08-24","arxiv_id":"2108.10939","n_code_links":1,"syntology":null},{"paper":null,"title":"Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System","date":"2021-07-20","arxiv_id":"2107.09625","n_code_links":0,"syntology":null},{"paper":"/paper/whose-heritage-classification-of-unesco-world","title":"WHOSe Heritage: Classification of UNESCO World Heritage \"Outstanding Universal Value\" Documents with Soft Labels","date":"2021-04-12","arxiv_id":"2104.05547","n_code_links":1,"syntology":null},{"paper":"/paper/l3cubemahasent-a-marathi-tweet-based","title":"L3CubeMahaSent: A Marathi Tweet-based Sentiment Analysis Dataset","date":"2021-03-21","arxiv_id":"2103.11408","n_code_links":1,"syntology":null},{"paper":null,"title":"Experimental Evaluation of Deep Learning models for Marathi Text Classification","date":"2021-01-13","arxiv_id":"2101.04899","n_code_links":0,"syntology":null},{"paper":"/paper/ladiff-ulmfit-a-layer-differentiated-training","title":"LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT","date":"2021-01-13","arxiv_id":"2101.04965","n_code_links":1,"syntology":null},{"paper":"/paper/hinglishnlp-at-semeval-2020-task-9-fine-tuned","title":"HinglishNLP at SemEval-2020 Task 9: Fine-tuned Language Models for Hinglish Sentiment Detection","date":"2020-12-01","arxiv_id":null,"n_code_links":2,"syntology":null},{"paper":null,"title":"Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating","date":"2020-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Palomino-Ochoa at SemEval-2020 Task 9: Robust System based on Transformer for Code-Mixed Sentiment Classification","date":"2020-11-18","arxiv_id":"2011.09448","n_code_links":0,"syntology":null},{"paper":"/paper/pagsusuri-ng-rnn-based-transfer-learning","title":"Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language","date":"2020-10-13","arxiv_id":"2010.06447","n_code_links":2,"syntology":null},{"paper":null,"title":"Gauravarora@HASOC-Dravidian-CodeMix-FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection","date":"2020-10-05","arxiv_id":"2010.02094","n_code_links":0,"syntology":null},{"paper":"/paper/farstail-a-persian-natural-language-inference","title":"FarsTail: A Persian Natural Language Inference Dataset","date":"2020-09-18","arxiv_id":"2009.08820","n_code_links":1,"syntology":null},{"paper":null,"title":"Fine-tuning Pre-trained Contextual Embeddings for Citation Content Analysis in Scholarly Publication","date":"2020-09-12","arxiv_id":"2009.05836","n_code_links":0,"syntology":null},{"paper":"/paper/hinglishnlp-fine-tuned-language-models-for","title":"HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection","date":"2020-08-22","arxiv_id":"2008.09820","n_code_links":2,"syntology":null},{"paper":null,"title":"Evaluation Metrics for Headline Generation Using Deep Pre-Trained Embeddings","date":"2020-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/offensive-language-detection-in-arabic-using","title":"Offensive language detection in Arabic using ULMFiT","date":"2020-05-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Text Categorization for Conflict Event Annotation","date":"2020-05-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/inferring-the-source-of-official-texts-can","title":"Inferring the source of official texts: can SVM beat ULMFiT?","date":"2020-03-02","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Localized Flood DetectionWith Minimal Labeled Social Media Data Using Transfer Learning","date":"2020-02-10","arxiv_id":"2003.04973","n_code_links":0,"syntology":null},{"paper":null,"title":"Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks","date":"2019-12-28","arxiv_id":"1912.12397","n_code_links":0,"syntology":null},{"paper":null,"title":"A Comparative Study of Pretrained Language Models on Thai Social Text Categorization","date":"2019-12-03","arxiv_id":"1912.01580","n_code_links":0,"syntology":null},{"paper":null,"title":"Evolution of transfer learning in natural language processing","date":"2019-10-16","arxiv_id":"1910.07370","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/transfer-learning","name":"Transfer Learning","papers":14},{"task":"/task/classification","name":"General Classification","papers":12},{"task":"/task/language-modelling","name":"Language Modelling","papers":12},{"task":"/task/text-classification","name":"Text Classification","papers":12},{"task":"/task/language-modeling","name":"Language Modeling","papers":11},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":9},{"task":"/task/text-classification-1","name":"text-classification","papers":8},{"task":"/task/classification-1","name":"Classification","papers":7},{"task":"/task/decision-making","name":"Decision Making","papers":3},{"task":"/task/hate-speech-detection","name":"Hate Speech Detection","papers":3},{"task":"/task/sentiment-classification","name":"Sentiment Classification","papers":3},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":2},{"task":"/task/language-identification","name":"Language Identification","papers":2},{"task":"/task/management","name":"Management","papers":2},{"task":"/task/marketing","name":"Marketing","papers":2},{"task":"/task/sentence","name":"Sentence","papers":2},{"task":"/task/text-categorization","name":"Text Categorization","papers":2},{"task":"/task/transliteration","name":"Transliteration","papers":2},{"task":"/task/word-embeddings","name":"Word Embeddings","papers":2},{"task":"/task/regression-1","name":"regression","papers":2}],"tasks_shown":20,"n_tasks":51,"usage_by_year":[{"year":"2018","papers":1},{"year":"2019","papers":12},{"year":"2020","papers":13},{"year":"2021","papers":8},{"year":"2022","papers":1},{"year":"2023","papers":2},{"year":"2024","papers":2},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ulmfit"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}