Papers › Prune Once for All: Sparse Pre-Trained Language Models

Prune Once for All: Sparse Pre-Trained Language Models

10 Nov 2021arXiv:2111.05754archive 2025-07-28

Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, Moshe Wasserblat

Transformer-based language models are applied to a wide range of applications in natural language processing. However, they are inefficient and difficult to deploy. In recent years, many compression algorithms have been proposed to increase the implementation efficiency of large Transformer-based models on target hardware. In this work we present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We demonstrate our method with three known architectures to create sparse pre-trained BERT-Base, BERT-Large and DistilBERT. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss. Moreover, we show how to further compress the sparse models' weights to 8bit precision using quantization-aware training. For example, with our sparse pre-trained BERT-Large fine-tuned on SQuADv1.1 and quantized to 8bit we achieve a compression ratio of 40X for the encoder with less than 1% accuracy loss. To the best of our knowledge, our results show the best compression-to-accuracy ratio for BERT-Base, BERT-Large, and DistilBERT.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

intellabs/model-compression-research-package officialmentioned in papermentioned on GitHubpytorchnot reachable when probed 2026-09-17 — repositories for recent papers often appear after camera-readySyntology: not harvested report
intel/intel-extension-for-transformers mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Efficient Streaming Language Models with Attention Sinks (arXiv:2309.17453; that paper's own run record: 8 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified) and Effective Quantization for Diffusion Models on CPUs (arXiv:2311.16133; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

AllNatural Language InferenceQuantizationQuestion AnsweringSentiment AnalysisTransfer Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Natural Language Inference MultiNLI Dev BERT-Large-uncased-PruneOFA (90% unstruct sparse) Matched 83.74 #2 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Large-uncased-PruneOFA (90% unstruct sparse) Mismatched 84.2 #2 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) Matched 83.47 #3 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) Mismatched 84.08 #3 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (85% unstruct sparse) Matched 82.71 #4 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (85% unstruct sparse) Mismatched 83.67 #4 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (90% unstruct sparse) Matched 81.45 #5 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (90% unstruct sparse) Mismatched 82.43 #5 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) Matched 81.4 #6 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) Mismatched 82.51 #6 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (85% unstruct sparse) Matched 81.35 #7 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (85% unstruct sparse) Mismatched 82.03 #7 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (90% unstruct sparse) Matched 80.68 #8 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (90% unstruct sparse) Mismatched 81.47 #8 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) Matched 80.66 #9 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) Mismatched 81.14 #9 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) Matched 78.8 #10 of 10 Archive leaderboard report
Natural Language Inference MultiNLI Dev DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) Mismatched 80.4 #10 of 10 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Large-uncased-PruneOFA (90% unstruct sparse) EM 83.35 #10 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Large-uncased-PruneOFA (90% unstruct sparse) F1 90.2 #10 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) EM 83.22 #11 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8) F1 90.02 #11 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (85% unstruct sparse) EM 81.1 #12 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (85% unstruct sparse) F1 88.42 #12 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) EM 80.84 #13 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8) F1 88.24 #13 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (90% unstruct sparse) EM 79.83 #14 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev BERT-Base-uncased-PruneOFA (90% unstruct sparse) F1 87.25 #14 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (85% unstruct sparse) EM 78.1 #19 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (85% unstruct sparse) F1 85.82 #19 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) EM 77.03 #21 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8) F1 85.13 #21 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (90% unstruct sparse) EM 76.91 #22 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (90% unstruct sparse) F1 84.82 #22 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) EM 75.62 #25 of 55 Archive leaderboard report
Question Answering SQuAD1.1 dev DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8) F1 83.87 #25 of 55 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionAttention DropoutBERTBPEDense ConnectionsDistilBERTDropoutLabel SmoothingLayer NormalizationLinear LayerLinear Warmup With Linear DecayMulti-Head AttentionPosition-Wise Feed-Forward LayerPruningResidual ConnectionSoftmaxTransformerWeight DecayWordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections