</> Code with Papers by Syntology

Methods › General › Fine-Tuning › Discriminative Fine-Tuning

Discriminative Fine-Tuning

1,990 papers tagged archive 2025-07-28

Introduced by Jeremy Howard et al. in Universal Language Model Fine-tuning for Text Classification

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate for all layers of the model, discriminative fine-tuning allows us to tune each layer with different learning rates. For context, the regular stochastic gradient descent (SGD) update of a model’s parameters θ at time step t looks like the following (Ruder, 2016):

θₜ = θₜ₋₁ − η·∇_θJ(θ)

where η is the learning rate and ∇_θJ(θ) is the gradient with regard to the model’s objective function. For discriminative fine-tuning, we split the parameters θ into {θ₁, …, θ_L} where θₗ contains the parameters of the model at the l-th layer and L is the number of layers of the model. Similarly, we obtain {η₁, …, η_L} where θₗ where ηₗ is the learning rate of the l-th layer. The SGD update with discriminative finetuning is then:

θₜˡ = θₜ₋₁ˡ - ηˡ·∇_(θˡ)J(θ)

The authors find that empirically it worked well to first choose the learning rate ηᴸ of the last layer by fine-tuning only the last layer and using ηˡ⁻¹=ηˡ/2.6 as the learning rate for lower layers.

PaperSourceSee Code · fastai/fastai

Papers archive 2025-07-28

30 shown of 1,990, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

Tasks archive 2025-07-28

20 shown of 725 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

TaskPapers
Language Modelling431
Language Modeling323
Text Generation192
Large Language Model123
Question Answering106
Sentence99
Decoder81
Retrieval78
Transfer Learning65
Text Classification59
Prompt Engineering57
text-classification54
Decision Making53
In-Context Learning53
Translation53
Sentiment Analysis52
GPU49
Natural Language Understanding40
Few-Shot Learning39
Machine Translation39

Usage over time archive 2025-07-28

Papers per year tagged with Discriminative Fine-Tuning: 2018 to 2025, peak 701 701 0 2018: 2 papers 2018 2019: 68 papers 2019 2020: 141 papers 2020 2021: 173 papers 2021 2022: 164 papers 2022 2023: 491 papers 2023 2024: 701 papers 2024 2025: 250 papers 2025
Papers per year the archive tags with this method, by the paper's archive date (1,990 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Fine-Tuning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections