{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/r-2-range-regularization-for-model","title":"R2 Loss: Range Restriction Loss for Model Compression and Quantization","arxiv_id":"2303.08253","date":"2023-03-14","proceeding":null,"authors":["Arnav Kundu","Chungkuk Yoo","Srijan Mishra","Minsik Cho","Saurabh Adya"],"abstract":"Model quantization and compression is widely used techniques to reduce usage of computing resource at inference time. While state-of-the-art works have been achieved reasonable accuracy with higher bit such as 4bit or 8bit, but still it is challenging to quantize/compress a model further, e.g., 1bit or 2bit. To overcome the challenge, we focus on outliers in weights of a pre-trained model which disrupt effective lower bit quantization and compression. In this work, we propose Range Restriction Loss (R2-Loss) for building lower bit quantization and compression friendly models by removing outliers from weights during pre-training. By effectively restricting range of weights, we mold the overall distribution into a tight shape to ensure high quantization bit resolution, therefore allowing model compression and quantization techniques can to utilize their limited numeric representation powers better. We introduce three different, L-inf R2-Loss, its extension Margin R2-Loss and a new Soft-Min-MaxR2-Loss to be used as an auxiliary loss during full-precision model training. These R2-Loss can be used in different cases such as L-inf and Margin R2-Loss would be effective for symmetric quantization, while Soft-Min-Max R2-Loss shows better performance for model compression. In our experiment, R2-Loss improves lower bit quantization accuracy with state-of-the-art post-training quantization (PTQ), quantization-aware training (QAT), and model compression techniques. With R2-Loss, MobileNet-V2 2bit weight and 8bit activation PTQ, MobileNet-V1 2bit weight and activation QAT, ResNet18 1bit weight compression are improved to 59.49% from 50.66%, 59.05% from 55.96%, and 52.58% from 45.54%, respectively.","url_abs":"https://arxiv.org/abs/2303.08253v2","url_pdf":"https://arxiv.org/pdf/2303.08253v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"mobilenetv1","method_name":"MobileNetV1"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 4bit-1dim model compression using DKM","rank_in_archive_order":3,"of":12,"metrics":{"Top-1":"70.52"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 4bit-1dim model compression using DKM","rank_in_archive_order":4,"of":12,"metrics":{"Top-1":"69.63"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 2bit-1dim model compression using DKM","rank_in_archive_order":5,"of":12,"metrics":{"Top-1":"68.63"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 2bit-1dim model compression using DKM","rank_in_archive_order":6,"of":12,"metrics":{"Top-1":"67.62"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 4bit-4dim model compression using DKM","rank_in_archive_order":7,"of":12,"metrics":{"Top-1":"66.1"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 2bit-2dim model compression using DKM","rank_in_archive_order":8,"of":12,"metrics":{"Top-1":"64.7"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 4bit-4dim model compression using DKM","rank_in_archive_order":9,"of":12,"metrics":{"Top-1":"61.4"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"ResNet-18 + 1bit-1dim model compression using DKM","rank_in_archive_order":10,"of":12,"metrics":{"Top-1":"59.7"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 2bit-2dim model compression using DKM","rank_in_archive_order":11,"of":12,"metrics":{"Top-1":"53.99"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-imagenet","task":"Model Compression","dataset":"ImageNet","model":"MobileNet-v1 + 1bit-1dim model compression using DKM","rank_in_archive_order":12,"of":12,"metrics":{"Top-1":"52.58"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-qnli","task":"Model Compression","dataset":"QNLI","model":"MobileBERT + 2bit-1dim model compression using DKM","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"82.13"},"uses_additional_data":false},{"leaderboard":"/sota/model-compression-on-qnli","task":"Model Compression","dataset":"QNLI","model":"MobileBERT + 1bit-1dim model compression using DKM","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"63.17"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"MobileNet-v1 + EWGS + R2Loss","rank_in_archive_order":25,"of":27,"metrics":{"Top-1 Accuracy (%)":"69.79","Weight bits":"4"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"MobileNet-v1 + LSQ + R2Loss","rank_in_archive_order":26,"of":27,"metrics":{"Top-1 Accuracy (%)":"69.64"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"ResNet-18 + PACT + R2Loss","rank_in_archive_order":27,"of":27,"metrics":{"Activation bits":"4","Top-1 Accuracy (%)":"68.45","Weight bits":"2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.08253","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}