{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sharpness-aware-quantization-for-deep-neural","title":"Sharpness-aware Quantization for Deep Neural Networks","arxiv_id":"2111.12273","date":"2021-11-24","proceeding":null,"authors":["Jing Liu","Jianfei Cai","Bohan Zhuang"],"abstract":"Network quantization is a dominant paradigm of model compression. However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance. Recently, Sharpness-Aware Minimization (SAM) has been proposed to smooth the loss landscape and improve the generalization performance of the models. Nevertheless, directly applying SAM to the quantized models can lead to perturbation mismatch or diminishment issues, resulting in suboptimal performance. In this paper, we propose a novel method, dubbed Sharpness-Aware Quantization (SAQ), to explore the effect of SAM in model compression, particularly quantization for the first time. Specifically, we first provide a unified view of quantization and SAM by treating them as introducing quantization noises and adversarial perturbations to the model weights, respectively. According to whether the noise and perturbation terms depend on each other, SAQ can be formulated into three cases, which are analyzed and compared comprehensively. Furthermore, by introducing an efficient training strategy, SAQ only incurs a little additional training overhead compared with the default optimizer (e.g., SGD or AdamW). Extensive experiments on both convolutional neural networks and Transformers across various datasets (i.e., ImageNet, CIFAR-10/100, Oxford Flowers-102, Oxford-IIIT Pets) show that SAQ improves the generalization performance of the quantized models, yielding the SOTA results in uniform quantization. For example, on ImageNet, SAQ outperforms AdamW by 1.2% on the Top-1 accuracy for 4-bit ViT-B/16. Our 4-bit ResNet-50 surpasses the previous SOTA method by 0.9% on the Top-1 accuracy.","url_abs":"https://arxiv.org/abs/2111.12273v5","url_pdf":"https://arxiv.org/pdf/2111.12273v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sharpness-aware-quantization-for-deep-neural","repo_url":"https://github.com/zhuang-group/saq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"sharpness-aware-quantization-for-deep-neural","repo_url":"https://github.com/zip-group/saq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"sharpness-aware-quantization-for-deep-neural","repo_url":"https://github.com/ziplab/saq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"sharpness-aware-quantization-for-deep-neural","repo_url":"https://github.com/yangyucheng000/Paper-3/tree/main/SharpDRO-ms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"sharpness-aware-minimization","method_name":"Sharpness-Aware Minimization"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"CNN37","rank_in_archive_order":207,"of":211,"metrics":{"Percentage correct":"35.05"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.12273","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12273"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zip-group/saq","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziplab/saq","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuang-group/saq","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangyucheng000/Paper-3/tree/main/SharpDRO-ms","reach":null}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3da0ed128a47216d","entry":"data_parallel","repo":"zhuang-group/saq","repo_kind":"official","path":"core/model_transform.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/model_transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3da0ed128a47216d"}},{"code_sha256_prefix":"c40f16f8c552876e","entry":"get_cifar_dataloader","repo":"zhuang-group/saq","repo_kind":"official","path":"core/dataloader.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c40f16f8c552876e"}},{"code_sha256_prefix":"b02977776bf3caf6","entry":"get_cifar_train_val_test_dataloader","repo":"zhuang-group/saq","repo_kind":"official","path":"core/dataloader.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b02977776bf3caf6"}},{"code_sha256_prefix":"961a136f0e5151f9","entry":"get_imagenet_dataloader","repo":"zhuang-group/saq","repo_kind":"official","path":"core/dataloader.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"961a136f0e5151f9"}},{"code_sha256_prefix":"5d93362f9df2f82e","entry":"get_logger","repo":"zhuang-group/saq","repo_kind":"official","path":"core/logger.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d93362f9df2f82e"}},{"code_sha256_prefix":"601ccb52d01d6f2c","entry":"get_lr","repo":"zhuang-group/saq","repo_kind":"official","path":"core/engine.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"601ccb52d01d6f2c"}},{"code_sha256_prefix":"cb0169bdacbbefc8","entry":"get_optimizer","repo":"zhuang-group/saq","repo_kind":"official","path":"core/optim.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/optim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb0169bdacbbefc8"}},{"code_sha256_prefix":"e8b97da0d2b443fb","entry":"list2sequential","repo":"zhuang-group/saq","repo_kind":"official","path":"core/model_transform.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/model_transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8b97da0d2b443fb"}},{"code_sha256_prefix":"46bf64171d2a9bf5","entry":"model2list","repo":"zhuang-group/saq","repo_kind":"official","path":"core/model_transform.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/model_transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46bf64171d2a9bf5"}},{"code_sha256_prefix":"f924bc6970c6a9da","entry":"train","repo":"zhuang-group/saq","repo_kind":"official","path":"core/engine.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f924bc6970c6a9da"}},{"code_sha256_prefix":"d9191590db201cdf","entry":"val","repo":"zhuang-group/saq","repo_kind":"official","path":"core/engine.py","file_url":"https://github.com/zhuang-group/saq/blob/HEAD/core/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9191590db201cdf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}