Model Compression

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

3 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 440 papers with code (1,356 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
  24 Feb 2016 59 repositories listed Syntology ran 4 of 4 samples · 0 unverified · 2 pointer-only (licence)
  (2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car.
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
  23 Aug 2019 40 repositories listed Syntology ran 6 of 32 samples · 26 unverified
  Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  31 Oct 2022 17 repositories listed Syntology ran 5 of 15 samples · 10 unverified · 1 pointer-only (licence)
  In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient.
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
  10 Feb 2018 12 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 1 pointer-only (licence)
  Model compression is a critical technique to efficiently deploy neural network models on mobile devices which have limited computation resources and tight power budgets.
• Learned Step Size Quantization
  21 Feb 2019 9 repositories listed Syntology ran 7 of 23 samples · 16 unverified · 6 pointer-only (licence)
  Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases.
• The State of Sparsity in Deep Neural Networks
  25 Feb 2019 6 repositories listed
  We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on…
• Patient Knowledge Distillation for BERT Model Compression
  25 Aug 2019 5 repositories listed Syntology ran 9 of 27 samples · 18 unverified · 27 pointer-only (licence)
  Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks.
• Model compression via distillation and quantization
  15 Feb 2018 5 repositories listed Syntology ran 2 of 4 samples · 2 unverified
  Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning.
• Ternary Weight Networks
  16 May 2016 5 repositories listed
  We present a memory and computation efficient ternary weight networks (TWNs) - with weights constrained to +1, 0 and -1.
• Sharpness-aware Quantization for Deep Neural Networks
  24 Nov 2021 4 repositories listed Syntology ran 0 of 11 samples · 11 unverified
  However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance.
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
  8 Feb 2021 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Text to speech (TTS) has been broadly used to synthesize natural and intelligible speech in different scenarios.
• Training with Quantization Noise for Extreme Model Compression
  15 Apr 2020 4 repositories listed
  A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator.
• Contrastive Representation Distillation
  23 Oct 2019 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  We demonstrate that this objective ignores important structural knowledge of the teacher network.
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
  27 Sep 2019 4 repositories listed Syntology ran 1 of 5 samples · 4 unverified
  Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front-end devices.
• To prune, or not to prune: exploring the efficacy of pruning for model compression
  5 Oct 2017 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model.
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  14 Jan 2022 3 repositories listed
  As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to…
• ZeroQ: A Novel Zero Shot Quantization Framework
  1 Jan 2020 3 repositories listed Syntology ran 4 of 19 samples · 15 unverified
  Importantly, ZeroQ has a very low computational overhead, and it can finish the entire quantization process in less than 30s (0.
• Data-Free Adversarial Distillation
  23 Dec 2019 3 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 2 pointer-only (licence)
  Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer.
• MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face Images
  19 Nov 2017 3 repositories listed
  On the other hand, KD is proved to be useful for model compression for the FER problem, and we discovered that its effects gets more and more significant with the decreasing model size.
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
  19 Nov 2015 3 repositories listed
  The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent.
• Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic
  24 Aug 2024 2 repositories listed Syntology ran 7 of 7 samples · 0 unverified
  Our algorithm works in two steps: i) Localization: identify tiny (1% of the total parameters) localized regions in the finetuned models containing essential skills for the downstream tasks, and ii) Stitching:…
• Adversarial Fine-tuning of Compressed Neural Networks for Joint Improvement of Robustness and Efficiency
  14 Mar 2024 2 repositories listed
  We present experiments on two benchmark datasets showing that adversarial fine-tuning of compressed models can achieve robustness performance comparable to adversarially trained models, while also improving…
• LLM Inference Unveiled: Survey and Roofline Model Insights
  26 Feb 2024 2 repositories listed Syntology ran 10 of 10 samples · 0 unverified
  Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference…
• Safety and Performance, Why Not Both? Bi-Objective Optimized Model Compression against Heterogeneous Attacks Toward AI Software Deployment
  2 Jan 2024 2 repositories listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  To mitigate this issue, AI software compression plays a crucial role, which aims to compress model size while keeping high performance.
• Privacy and Accuracy Implications of Model Complexity and Integration in Heterogeneous Federated Learning
  29 Nov 2023 2 repositories listed Syntology ran 4 of 5 samples · 1 unverified · 5 pointer-only (licence)
  Federated Learning (FL) has been proposed as a privacy-preserving solution for distributed machine learning, particularly in heterogeneous FL settings where clients have varying computational capabilities and thus train…
• LXMERT Model Compression for Visual Question Answering
  23 Oct 2023 2 repositories listed
  Large-scale pretrained models such as LXMERT are becoming popular for learning cross-modal representations on text-image pairs for vision-language tasks.
• Uncovering the Hidden Cost of Model Compression
  29 Aug 2023 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  This empirical investigation underscores the need for a nuanced understanding beyond mere accuracy in sparse and quantized settings, thereby paving the way for further exploration in Visual Prompting techniques tailored…
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
  25 Aug 2023 2 repositories listed Syntology ran 9 of 16 samples · 7 unverified
  LWC modulates the extreme values of weights by optimizing the clipping threshold.
• Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision
  1 Jul 2023 2 repositories listed
  We explore the best practices to alleviate the variation's influence during low-bit transformer QAT and propose a variation-aware quantization scheme for both vision and language transformers.
• Performance-aware Approximation of Global Channel Pruning for Multitask CNNs
  21 Mar 2023 2 repositories listed
  Global channel pruning (GCP) aims to remove a subset of channels (filters) across different layers from a deep model without hurting the performance.

Syntology lines on 20 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.