{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lsq-improving-low-bit-quantization-through","title":"LSQ+: Improving low-bit quantization through learnable offsets and better initialization","arxiv_id":"2004.09576","date":"2020-04-20","proceeding":null,"authors":["Yash Bhalgat","Jinwon Lee","Markus Nagel","Tijmen Blankevoort","Nojun Kwak"],"abstract":"Unlike ReLU, newer activation functions (like Swish, H-swish, Mish) that are frequently employed in popular efficient architectures can also result in negative activation values, with skewed positive and negative ranges. Typical learnable quantization schemes [PACT, LSQ] assume unsigned quantization for activations and quantize all negative activations to zero which leads to significant loss in performance. Naively using signed quantization to accommodate these negative values requires an extra sign bit which is expensive for low-bit (2-, 3-, 4-bit) quantization. To solve this problem, we propose LSQ+, a natural extension of LSQ, wherein we introduce a general asymmetric quantization scheme with trainable scale and offset parameters that can learn to accommodate the negative activations. Gradient-based learnable quantization schemes also commonly suffer from high instability or variance in the final training performance, hence requiring a great deal of hyper-parameter tuning to reach a satisfactory performance. LSQ+ alleviates this problem by using an MSE-based initialization scheme for the quantization parameters. We show that this initialization leads to significantly lower variance in final performance across multiple training runs. Overall, LSQ+ shows state-of-the-art results for EfficientNet and MixNet and also significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations (e.g.: 1.8% gain with W4A4 quantization and upto 5.6% gain with W2A2 quantization of EfficientNet-B0 on ImageNet dataset). To the best of our knowledge, ours is the first work to quantize such architectures to extremely low bit-widths.","url_abs":"https://arxiv.org/abs/2004.09576v1","url_pdf":"https://arxiv.org/pdf/2004.09576v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lsq-improving-low-bit-quantization-through","repo_url":"https://github.com/DeadAt0m/LSQ-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"lsq-improving-low-bit-quantization-through","repo_url":"https://github.com/DeadAt0m/LSQFakeQuantize-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"lsq-improving-low-bit-quantization-through","repo_url":"https://github.com/ZouJiu1/LSQplus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"lsq-improving-low-bit-quantization-through","repo_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"efficientnet","method_name":"EfficientNet"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"inverted-residual-block","method_name":"Inverted Residual Block"},{"method_slug":"mixconv","method_name":"MixConv"},{"method_slug":"mixnet","method_name":"MixNet"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"squeeze-and-excitation-block","method_name":"Squeeze-and-Excitation Block"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"EfficientNet-W4A4","rank_in_archive_order":18,"of":27,"metrics":{"Activation bits":"4","Top-1 Accuracy (%)":"73.8","Weight bits":"4"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"MixNet-W4A4","rank_in_archive_order":20,"of":27,"metrics":{"Activation bits":"4","Top-1 Accuracy (%)":"71.7","Weight bits":"4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.09576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.09576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZouJiu1/LSQplus","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DeadAt0m/LSQ-PyTorch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DeadAt0m/LSQFakeQuantize-PyTorch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_honours":2,"unverified":7},"by_repo_kind":{"listed":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ad708b63b59159a6","entry":"flops_to_string","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/flops_counter.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/flops_counter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ad708b63b59159a6"}},{"code_sha256_prefix":"a488bae52f1bfd5d","entry":"grad_scale","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a488bae52f1bfd5d"}},{"code_sha256_prefix":"ed1a69e092272dba","entry":"round_pass","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ed1a69e092272dba"}},{"code_sha256_prefix":"40695dc7514bcb11","entry":"Conv1x1Bn","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/utils/mixnet.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/utils/mixnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"40695dc7514bcb11"}},{"code_sha256_prefix":"86d29f3a94353011","entry":"Conv3x3Bn","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/utils/mixnet.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/utils/mixnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"86d29f3a94353011"}},{"code_sha256_prefix":"572064beb59d24e5","entry":"add_lsqmodule","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/utils/add_lsqmodule.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/utils/add_lsqmodule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"572064beb59d24e5"}},{"code_sha256_prefix":"50cb4735b09169be","entry":"efficientnet_b0","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/utils/effnet.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/utils/effnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"50cb4735b09169be"}},{"code_sha256_prefix":"e8fc1d1906b17dfb","entry":"mixnet_s","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/utils/mixnet.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/utils/mixnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e8fc1d1906b17dfb"}},{"code_sha256_prefix":"4c52df27c25660c5","entry":"params_to_string","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/flops_counter.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/flops_counter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"4c52df27c25660c5"}},{"code_sha256_prefix":"e52c4d73ce21cdfa","entry":"resume_checkpoint","repo":"yashbhalgat/QualcommAI-MicroNet-submission-MixNet","repo_kind":"listed","path":"lsq_quantizer/helpers.py","file_url":"https://github.com/yashbhalgat/QualcommAI-MicroNet-submission-MixNet/blob/HEAD/lsq_quantizer/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e52c4d73ce21cdfa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}