{"url":"/method/discriminative-fine-tuning","slug":"discriminative-fine-tuning","name":"Discriminative Fine-Tuning","full_name":"Discriminative Fine-Tuning","full_name_withheld":false,"description_markdown":"**Discriminative Fine-Tuning** is a fine-tuning strategy that is used for [ULMFiT](https://paperswithcode.com/method/ulmfit) type models. Instead of using the same learning rate for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates. For context, the regular stochastic gradient descent ([SGD](https://paperswithcode.com/method/sgd)) update of a model’s parameters $\\theta$ at time step $t$ looks like the following (Ruder, 2016):\r\n\r\n$$ \\theta\\_{t} = \\theta\\_{t-1} − \\eta\\cdot\\nabla\\_{\\theta}J\\left(\\theta\\right)$$\r\n\r\nwhere $\\eta$ is the learning rate and $\\nabla\\_{\\theta}J\\left(\\theta\\right)$ is the gradient with regard to the model’s objective function. For discriminative fine-tuning, we split the parameters $\\theta$ into {$\\theta\\_{1}, \\ldots, \\theta\\_{L}$} where $\\theta\\_{l}$ contains the parameters of the model at the $l$-th layer and $L$ is the number of layers of the model. Similarly, we obtain {$\\eta\\_{1}, \\ldots, \\eta\\_{L}$} where $\\theta\\_{l}$ where $\\eta\\_{l}$ is the learning rate of the $l$-th layer. The SGD update with discriminative finetuning is then:\r\n\r\n$$ \\theta\\_{t}^{l} = \\theta\\_{t-1}^{l} - \\eta^{l}\\cdot\\nabla\\_{\\theta^{l}}J\\left(\\theta\\right) $$\r\n\r\nThe authors find that empirically it worked well to first choose the learning rate $\\eta^{L}$ of the last layer by fine-tuning only the last layer and using $\\eta^{l-1}=\\eta^{l}/2.6$ as the learning rate for lower layers.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Universal Language Model Fine-tuning for Text Classification","paper":"/paper/universal-language-model-fine-tuning-for-text","first_author":"Jeremy Howard","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/universal-language-model-fine-tuning-for-text"},"source":{"url":"http://arxiv.org/abs/1801.06146v5","title":"Universal Language Model Fine-tuning for Text Classification","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/fastai/fastai/blob/43001e17ba469308e9688dfe99a891018bcf7ad4/courses/dl2/imdb_scripts/finetune_lm.py#L132","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Fine-Tuning","url":"/methods/category/fine-tuning","pwc_aliases":[]}],"n_papers_tagged":1990,"archive_num_papers":1990,"papers_newest_first":[{"paper":"/paper/making-language-model-a-hierarchical","title":"Making Language Model a Hierarchical Classifier and Generator","date":"2025-07-17","arxiv_id":"2507.12930","n_code_links":1,"syntology":null},{"paper":null,"title":"Generative Click-through Rate Prediction with Applications to Search Advertising","date":"2025-07-15","arxiv_id":"2507.11246","n_code_links":0,"syntology":null},{"paper":null,"title":"Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery","date":"2025-07-04","arxiv_id":"2507.03605","n_code_links":0,"syntology":null},{"paper":null,"title":"Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models","date":"2025-06-28","arxiv_id":"2506.22957","n_code_links":0,"syntology":null},{"paper":null,"title":"Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems?","date":"2025-06-26","arxiv_id":"2506.21274","n_code_links":0,"syntology":null},{"paper":null,"title":"Large Language Models Acing Chartered Accountancy","date":"2025-06-26","arxiv_id":"2506.21031","n_code_links":0,"syntology":null},{"paper":null,"title":"Large Language Model-Driven Code Compliance Checking in Building Information Modeling","date":"2025-06-25","arxiv_id":"2506.20551","n_code_links":0,"syntology":null},{"paper":null,"title":"InsertRank: LLMs can reason over BM25 scores to Improve Listwise Reranking","date":"2025-06-17","arxiv_id":"2506.14086","n_code_links":0,"syntology":null},{"paper":null,"title":"M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models","date":"2025-06-17","arxiv_id":"2506.14532","n_code_links":0,"syntology":null},{"paper":null,"title":"Toward a Graph Foundation Model: Pre-Training Transformers With Random Walks","date":"2025-06-17","arxiv_id":"2506.14098","n_code_links":0,"syntology":null},{"paper":"/paper/decomposing-mlp-activations-into","title":"Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization","date":"2025-06-12","arxiv_id":"2506.10920","n_code_links":1,"syntology":null},{"paper":"/paper/neuralnexus-at-bea-2025-shared-task-retrieval","title":"NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors","date":"2025-06-12","arxiv_id":"2506.10627","n_code_links":1,"syntology":null},{"paper":null,"title":"A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning","date":"2025-06-11","arxiv_id":"2506.09429","n_code_links":0,"syntology":null},{"paper":null,"title":"Latent Multi-Head Attention for Small Language Models","date":"2025-06-11","arxiv_id":"2506.09342","n_code_links":0,"syntology":null},{"paper":null,"title":"AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP","date":"2025-06-10","arxiv_id":"2506.08768","n_code_links":0,"syntology":null},{"paper":"/paper/evaluating-llms-across-multi-cognitive-levels","title":"Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving","date":"2025-06-10","arxiv_id":"2506.08349","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":3}},{"paper":null,"title":"Generative Voice Bursts during Phone Call","date":"2025-06-09","arxiv_id":"2506.07526","n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization","date":"2025-06-09","arxiv_id":"2506.07570","n_code_links":0,"syntology":null},{"paper":null,"title":"Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models","date":"2025-06-08","arxiv_id":"2506.07121","n_code_links":0,"syntology":null},{"paper":null,"title":"RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation","date":"2025-06-07","arxiv_id":"2506.06677","n_code_links":0,"syntology":null},{"paper":null,"title":"Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey","date":"2025-06-06","arxiv_id":"2506.11102","n_code_links":0,"syntology":null},{"paper":null,"title":"The Lock-in Hypothesis: Stagnation by Algorithm","date":"2025-06-06","arxiv_id":"2506.06166","n_code_links":0,"syntology":null},{"paper":null,"title":"Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective","date":"2025-06-05","arxiv_id":"2506.05166","n_code_links":0,"syntology":null},{"paper":"/paper/mathematical-reasoning-for-unmanned-aerial","title":"Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning","date":"2025-06-05","arxiv_id":"2506.04998","n_code_links":1,"syntology":null},{"paper":null,"title":"Privacy and Security Threat for OpenAI GPTs","date":"2025-06-04","arxiv_id":"2506.04036","n_code_links":0,"syntology":null},{"paper":null,"title":"An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models","date":"2025-06-03","arxiv_id":"2506.02730","n_code_links":0,"syntology":null},{"paper":null,"title":"Rethinking the effects of data contamination in Code Intelligence","date":"2025-06-03","arxiv_id":"2506.02791","n_code_links":0,"syntology":null},{"paper":"/paper/how-neural-networks-organize-concepts","title":"How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability","date":"2025-06-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Power-of-Two (PoT) Weights in Large Language Models (LLMs)","date":"2025-05-31","arxiv_id":"2506.00315","n_code_links":0,"syntology":null},{"paper":"/paper/agent-x-evaluating-deep-multimodal-reasoning","title":"Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks","date":"2025-05-30","arxiv_id":"2505.24876","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":431},{"task":"/task/language-modeling","name":"Language Modeling","papers":323},{"task":"/task/text-generation","name":"Text Generation","papers":192},{"task":"/task/large-language-model","name":"Large Language Model","papers":123},{"task":"/task/question-answering","name":"Question Answering","papers":106},{"task":"/task/sentence","name":"Sentence","papers":99},{"task":"/task/decoder","name":"Decoder","papers":81},{"task":"/task/retrieval","name":"Retrieval","papers":78},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":65},{"task":"/task/text-classification","name":"Text Classification","papers":59},{"task":"/task/prompt-engineering","name":"Prompt Engineering","papers":57},{"task":"/task/text-classification-1","name":"text-classification","papers":54},{"task":"/task/decision-making","name":"Decision Making","papers":53},{"task":"/task/in-context-learning","name":"In-Context Learning","papers":53},{"task":"/task/translation","name":"Translation","papers":53},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":52},{"task":null,"name":"GPU","papers":49},{"task":"/task/natural-language-understanding","name":"Natural Language Understanding","papers":40},{"task":"/task/few-shot-learning","name":"Few-Shot Learning","papers":39},{"task":"/task/machine-translation","name":"Machine Translation","papers":39}],"tasks_shown":20,"n_tasks":725,"usage_by_year":[{"year":"2018","papers":2},{"year":"2019","papers":68},{"year":"2020","papers":141},{"year":"2021","papers":173},{"year":"2022","papers":164},{"year":"2023","papers":491},{"year":"2024","papers":701},{"year":"2025","papers":250}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/discriminative-fine-tuning"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}