Papers › Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, Kurt Keutzer
Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Linguistic Acceptability | CoLA | Q-BERT (Shen et al., 2020) | Accuracy | 65.1 | #23 of 43 | Archive leaderboard | report |
| Natural Language Inference | MultiNLI | Q-BERT (Shen et al., 2020) | Matched | 87.8 | #20 of 67 | Archive leaderboard | report |
| Natural Language Inference | QNLI | Q-BERT (Shen et al., 2020) | Accuracy | 93.0 | #22 of 43 | Archive leaderboard | report |
| Natural Language Inference | RTE | Q-BERT (Shen et al., 2020) | Accuracy | 84.7 | #28 of 90 | Archive leaderboard | report |
| Semantic Textual Similarity | MRPC | Q-BERT (Shen et al., 2020) | Accuracy | 88.2 | #22 of 45 | Archive leaderboard | report |
| Semantic Textual Similarity | STS Benchmark | Q-BERT (Shen et al., 2020) | Pearson Correlation | 0.911 | #13 of 66 | Archive leaderboard | report |
| Sentiment Analysis | SST-2 Binary classification | Q-BERT (Shen et al., 2020) | Accuracy | 94.8 | #31 of 87 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections